A feature extraction method of an image, a semantic segmentation method and a related device thereof

Abstract

The application discloses a feature extraction method of an image, a semantic segmentation method and a related device thereof. The method comprises the following steps: acquiring a spectral image, wherein the spectral image is collected by using multiple waveband lights; extracting a first spatial feature of the spectral image by using a semantic feature extraction network; and extracting a target image feature of the spectral image by using the semantic feature extraction network based on the first spatial feature, wherein the input feature of each level of feature extraction comprises an input feature of a current level obtained by performing attention processing on the first spatial feature and a second spatial feature obtained by each previous level of feature extraction. In the foregoing manner, the application can enhance the extraction capability of spatial information in the spectral image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image feature extraction method, a semantic segmentation method, and related apparatus. Background Technology

[0002] Multispectral and hyperspectral images, due to their large spatial scale and high spectral resolution, have immense application value in land cover and resource surveys, as well as environmental monitoring. With the integration of multispectral and hyperspectral cameras into drones, drone-borne multispectral and hyperspectral images are also widely used in water quality monitoring, plant growth monitoring, and pest and disease monitoring. Because multispectral and hyperspectral images contain rich spatial and spectral information, they are crucial for accurate classification of ground features. Compared to traditional visible light images, multispectral and hyperspectral images, with their refined spectral features, can achieve precise species classification that visible light images cannot identify, such as tree species classification and crop classification (winter wheat and rice). However, due to the large number of bands in hyperspectral images and the strong correlation between adjacent bands, there is a certain degree of data redundancy. Furthermore, different satellite sensors and drone-borne sensors differ, resulting in variations in the scale and number of bands in the acquired data. Therefore, extracting image features from spectral images with different bands and from different sources is particularly important.

[0003] Existing technologies mention using ViT (Vision Transformer) networks to extract image features from spectral images. However, during the training process, ViT networks gradually deepen the extraction of semantic information while ignoring spatial information, resulting in the extracted spectral image features containing less spatial information, which in turn leads to inaccurate semantic segmentation results. Summary of the Invention

[0004] The main technical problem addressed in this application is to provide an image feature extraction method, a semantic segmentation method, and related apparatus, which can enhance the ability to extract spatial information from spectral images.

[0005] To address the aforementioned technical problems, the first aspect of this application provides an image feature extraction method, which includes: acquiring a spectral image, wherein the spectral image is obtained by collecting environmental data using multiple wavelength bands of light; extracting first spatial features of the spectral image using a semantic feature extraction network; and performing several levels of feature extraction based on the first spatial features using the semantic feature extraction network to obtain target image features of the spectral image, wherein the input features of each level of feature extraction include the current level's input features obtained by attention processing using the first spatial features and the second spatial features obtained from each previous level of feature extraction.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a semantic segmentation method for an image. The method includes: extracting features from a spectral image using the method described in the first aspect to obtain target image features of the spectral image, wherein the spectral image is acquired by collecting environmental data using multiple wavelengths of light; and performing segmentation prediction on the spectral image using the target image features to obtain a semantic segmentation result of the spectral image.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor executing the program instructions stored in the memory to implement the method described in the first or second aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium for storing program instructions that can be executed to implement the methods described in the first or second aspect.

[0009] The beneficial effects of this application are as follows: Unlike existing technologies, this application acquires spectral images of the environment using multiple wavelengths of light, and then uses a semantic feature extraction network to extract the first spatial features of the spectral images. Based on the first spatial features, the semantic feature extraction network performs several levels of feature extraction to obtain the target image features of the spectral images. The input features for each level of feature extraction include the current level's input features obtained by attention processing using the first spatial features and the second spatial features obtained from each previous level of feature extraction. Since each level of feature extraction utilizes the first spatial features for attention processing, the extracted target image features can contain a large amount of spatial information, enhancing the ability to extract spatial information from spectral images. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the first embodiment of the image feature extraction method provided in this application;

[0011] Figure 2 This is a flowchart illustrating the second embodiment of the image feature extraction method provided in this application;

[0012] Figure 3 This is a flowchart illustrating the third embodiment of the image feature extraction method provided in this application;

[0013] Figure 4 This is a schematic diagram of the framework of one embodiment of the semantic feature extraction network provided in this application;

[0014] Figure 5 This is a flowchart illustrating one embodiment of the semantic segmentation method for images provided in this application;

[0015] Figure 6 This is a schematic diagram of one embodiment of the spectral image obtained in this application;

[0016] Figure 7 This is a schematic diagram of an embodiment of the semantic segmentation result of the spectral image obtained in this application;

[0017] Figure 8 This is a schematic diagram of the framework of one embodiment of the electronic device provided in this application;

[0018] Figure 9 This is a schematic diagram of a framework of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that the embodiments of this application contain descriptions involving "first," "second," etc., which are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image feature extraction method provided in this application. The method includes:

[0022] S11: Acquire spectral image.

[0023] In one embodiment, the spectral image is acquired by capturing the environment using light across multiple wavelengths. The spectral image can be a multispectral image or a hyperspectral image. It can be captured by a satellite or a drone. Different spectral images may contain different numbers of channels, ranging from a few to dozens or even hundreds.

[0024] In other embodiments, multispectral images may be acquired, and the multispectral images may be processed as follows to obtain the target features of the multispectral images.

[0025] S12: Extract the first spatial features of the spectral image using a semantic feature extraction network.

[0026] In one embodiment, the spectral image can be first subjected to dimensionality reduction processing, that is, images containing different numbers of channels can be processed so that the processed images contain the same number of channels. Specifically, feature extraction can be performed on the spectral image to obtain multiple initial spatial features at different scales. These multiple initial spatial features at different scales can be transformed to obtain spectral features of a first number of dimensions. The first spatial feature can then be obtained using the spectral features of the first number of dimensions. The first number is set by the user, for example, 5, 6, 7, etc.

[0027] In another embodiment, the spectral image can be directly input into the spatial feature extraction module, which then extracts the first spatial features of the spectral image.

[0028] S13: Utilize a semantic feature extraction network to perform several levels of feature extraction based on the first spatial features to obtain the target image features of the spectral image.

[0029] In one embodiment, the input features for each level of feature extraction include the current level's input features obtained by performing attention processing using the first spatial features and the second spatial features obtained from each previous level of feature extraction. Specifically, for each level of feature extraction, multi-head interactive attention processing is performed using the first spatial features and the corresponding second spatial features from each previous level of feature extraction to obtain the current level's input features. For example, for the first level of feature extraction, the semantic feature extraction network can perform multi-head interactive attention processing using the first spatial features and the initial encoded features to obtain the first level's input features; for the second level of feature extraction, multi-head interactive attention processing can be performed using the first spatial features and the second spatial features obtained from the first level to obtain the second level's input features; for the third level of feature extraction, multi-head interactive attention processing can be performed using the first spatial features, the second spatial features obtained from the first level, and the second spatial features obtained from the second level to obtain the third level's input features. Further, the current level's input features are encoded to obtain the current level's output features, and the second spatial features corresponding to the current level's feature extraction are extracted from the current level's output features; the target image features are obtained using the second spatial features corresponding to the last level's feature extraction.

[0030] The process involves multi-head interactive attention processing using the first spatial features and the corresponding second spatial features extracted from each previous level to obtain the input features of the current level. This includes: using the first spatial features as key elements, the corresponding second spatial features extracted from each previous level as value elements, and the reference features of the current level as query elements, performing multi-head interactive attention processing to obtain the input features of the current level, and then extracting features from the input features of the current level to obtain the output features of the current level. The reference features of the first level are the initial coding features of the spectral image, and the reference features of non-first levels are the output features of the previous level. In one specific embodiment, interactive attention processing can be performed using the first spatial features and the reference features of the current level to obtain the first attention features; and interactive attention processing can be performed using the reference features of the current level and the corresponding second spatial features extracted from each previous level to obtain the second attention features corresponding to each previous level; the reference features of the current level, the first attention features, and the second attention features corresponding to each previous level are then fused to obtain the input features of the current level. For example, for the first level, interactive attention processing is performed using the first spatial features and the initial encoded features of the spectral image to obtain the first attention feature. Since there are no corresponding second spatial features extracted from other levels before the first level, the initial encoded features of the spectral image and the first attention feature can be directly fused to obtain the input feature of the first level. For the second level, interactive attention processing is performed using the first spatial features and the output features of the first level to obtain the first attention feature; then, interactive attention processing is performed using the output features of the first level and the second spatial features extracted from the first level to obtain the corresponding second attention feature of the first level; the output features of the first level, the first attention feature, and the corresponding second attention feature of the first level are fused to obtain the input feature of the second level. The acquisition methods for the input features of other levels follow the same logic and will not be detailed here. The fusion method can be to add the product of the first attention feature and the attention parameter, the reference feature of the current level, and the second attention feature corresponding to each previous level to obtain the input feature of the current level. The attention parameter is a learnable parameter used to balance the order-of-magnitude difference between the attention feature and the output feature of the current level. The attention parameter is initially 0 and can be changed during training.

[0031] In another specific implementation, the first spatial feature, the corresponding second spatial feature extracted from each previous level, and the reference feature of the current level can be standardized first to obtain the initial standardized feature, the spatial standardized feature corresponding to each previous level, and the reference standardized feature. Then, interactive attention processing is performed on the initial standardized feature and the reference standardized feature to obtain the first attention feature; and interactive attention processing is performed on the reference standardized feature and the spatial standardized feature corresponding to each previous level to obtain the second attention feature corresponding to each previous level. Specifically, the input features from the second level to the Nth level can be obtained with reference to Formula 1 below.

[0032]

[0033] in, This represents the input feature at level i+1. This represents the output feature of the i-th level. Indicates the first spatial feature; This represents the second spatial feature obtained from the nth level feature extraction; norm indicates the standardization process performed on the feature.

[0034] For the first level, since there are no corresponding spatially normalized features extracted from other levels before it, the first attention feature and the reference normalized feature can be fused to obtain the input features for the first level. The first attention feature is obtained by performing interactive attention processing on the initial normalized feature and the initial normalized encoded feature of the spectral image.

[0035] The extraction of the second spatial features corresponding to the current level's feature extraction from the current level's output features includes: performing interactive attention processing using the current level's output features as key elements and value elements, and the current level's attention features as query elements, respectively, to obtain the second spatial features corresponding to the current level's feature extraction; the current level's attention features are obtained by performing interactive attention processing using the current level's reference features and the second spatial features corresponding to the previous level's feature extraction. In one specific implementation, interactive attention processing is performed using the current level's output features and current level's attention features to obtain a third attention feature; the third attention feature and the current level's attention feature are then added together to obtain a fused feature. A feedforward neural network layer is used to process the fused feature to obtain the second spatial features corresponding to the current level's feature extraction.

[0036] In another specific implementation, the output features and attention features of the current level can be standardized first to obtain standardized output features and standardized attention features of the current level. Then, the standardized output features and standardized attention features of the current level are combined to obtain a third attention feature. The attention features of the current level and the third attention feature are added together to obtain a fused feature. The fused feature is then standardized to obtain a standardized result. A feedforward neural network layer is used to process the standardized result of the fused feature to obtain the second spatial feature corresponding to the current level feature extraction. Specifically, the second spatial feature corresponding to the current level feature extraction can be obtained by referring to Formula 2 below. The processing of the standardized result of the fused feature using a feedforward neural network layer can be achieved by adding a CFFN (convolutional feed-forward network) module after the attention module to enhance the continuity of the spatial features using its zero-padding operation.

[0037]

[0038] in, This indicates that the second spatial feature corresponding to the (i+1)th level feature extraction is extracted; represents the attention feature at level i+1; norm represents the standardization of the feature; CFFN represents the use of a feedforward neural network layer to process the feature.

[0039] In the above embodiments, when performing several levels of feature extraction using the first spatial features, the number of levels can be set by the user, for example, set to perform four levels of feature extraction.

[0040] In one embodiment, steps S12 and S13 can be executed by a semantic feature extraction network. The semantic feature extraction network includes a first spatial feature extraction module, several sequentially connected feature encoding layers, several spatial feature injection modules, and several second spatial feature extraction modules. Each feature encoding layer is connected to the corresponding spatial feature injection module and the second spatial feature extraction module. The first spatial feature extraction module extracts first spatial features from the spectral image. The several spatial feature injection modules perform a step of multi-head interactive attention processing using the first spatial features and the second spatial features corresponding to each previous feature extraction layer. For example, the first-level spatial feature injection module performs interactive attention processing using the first spatial features and the first encoded features of the spectral image; the second-level spatial feature injection module performs multi-head interactive attention processing using the first spatial features and the second spatial features corresponding to the first-level feature extraction layer to obtain the second-level input features; and the third-level spatial feature injection module performs multi-head interactive attention processing using the first spatial features, the second spatial features corresponding to the first-level feature extraction layer, and the second spatial features corresponding to the second-level feature extraction layer to obtain the third-level input features. A series of sequentially connected feature encoding layers encode the input features of the current corresponding level to obtain the output features of the current level. For example, the first-level feature encoding layer encodes the input features of the first level to obtain the output features of the first level; the second-level feature encoding layer encodes the input features of the second level to obtain the output features of the second level. Several levels of second-space feature extraction modules extract the corresponding second-space features from the output features of the current corresponding level. For example, the first-level second-space feature extraction module extracts the corresponding second-space features from the output features of the first level; the second-level second-space feature extraction module extracts the corresponding second-space features from the output features of the second level.

[0041] The above method involves acquiring a spectral image of the environment using multiple wavelengths of light; then, using a semantic feature extraction network to extract the first spatial features of the spectral image; finally, using the semantic feature extraction network to perform several levels of feature extraction based on the first spatial features to obtain the target image features of the spectral image. The input features for each level of feature extraction include the current level's input features obtained by attention processing using the first spatial features and the second spatial features obtained from each previous level of feature extraction. Since each level of feature extraction utilizes the first spatial features for attention processing, the extracted target image features can contain a large number of spatial features.

[0042] Please see Figure 2 , Figure 2This is a flowchart illustrating a second embodiment of the image feature extraction method provided in this application. The method includes:

[0043] S21: Acquire spectral image.

[0044] Spectral images are obtained by collecting images of the environment using light in multiple wavelengths.

[0045] S22: Extract the first spatial features of the spectral image using a semantic feature extraction network.

[0046] S23: Perform image block encoding on the spectral features of the spectral image to obtain the feature blocks corresponding to different image blocks in the spectral image, and perform position encoding on the spectral features of the spectral image to obtain the position encoding information corresponding to different image blocks in the spectral image.

[0047] In one embodiment, the spectral features of the spectral image are the spectral features after dimensionality reduction of the spectral image; that is, the dimensionality-reduced spectral features are image block encoded to obtain feature blocks corresponding to different image blocks in the spectral image; at the same time, the spectral features of the spectral image are position encoded to obtain position encoding information corresponding to different image blocks in the spectral image.

[0048] In another embodiment, the spectral features of the spectral image are obtained by directly extracting features from the spectral image.

[0049] S24: Using the feature blocks and position coding information corresponding to different image blocks, the initial coding features of the spectral image are obtained.

[0050] S25: For each level of feature extraction, multi-head interactive attention processing is performed using the first spatial features and the second spatial features corresponding to each previous level of feature extraction to obtain the input features of the current level. The input features of the current level are encoded to obtain the output features of the current level. The second spatial features corresponding to the current level feature extraction are extracted from the output features of the current level.

[0051] S26: Use the last-level feature extraction to extract the corresponding second-space feature separation to obtain multiple target features at different scales.

[0052] In one specific implementation, four levels of feature extraction are performed on the spectral image. For the first level of feature extraction, interactive attention processing is performed using the first spatial features and the initial encoded features of the spectral image to obtain the first attention features. The first attention features and the initial encoded features are fused to obtain the first level of input features. The first level of input features are encoded to obtain the first level of output features. The second spatial features corresponding to the first level of feature extraction are extracted from the first level of output features.

[0053] For the second-level feature extraction, interactive attention processing is performed using the first spatial features and the output features of the first level to obtain the first attention features; interactive attention processing is performed using the output features of the first level and the corresponding second spatial features extracted from the first-level features to obtain the second attention features; the output features of the first level, the first attention features, and the second attention features are fused to obtain the input features of the second level; the input features of the second level are encoded to obtain the output features of the second level, and the corresponding second spatial features of the second-level feature extraction are extracted from the output features of the second level.

[0054] For the third-level feature extraction, interactive attention processing is performed using the first spatial features and the output features of the second level to obtain the first attention features; interactive attention processing is performed using the output features of the second level and the corresponding second spatial features extracted from the first level to obtain the second attention features corresponding to the first level; interactive attention processing is performed using the output features of the second level and the corresponding second spatial features extracted from the second level to obtain the second attention features corresponding to the second level; the output features of the second level, the first attention features, the corresponding second attention features of the first level, and the corresponding second attention features of the second level are fused to obtain the input features of the third level; the input features of the third level are encoded to obtain the output features of the third level, and the corresponding second spatial features of the third-level feature extraction are extracted from the output features of the third level.

[0055] For the fourth-level feature extraction, interactive attention processing is performed using the first spatial features and the output features of the third level to obtain the first attention features; interactive attention processing is performed using the output features of the third level and the corresponding second spatial features extracted from the first level to obtain the second attention features corresponding to the first level; interactive attention processing is performed using the output features of the third level and the corresponding second spatial features extracted from the second level to obtain the second attention features corresponding to the second level; interactive attention processing is performed using the output features of the third level and the corresponding second spatial features extracted from the third level to obtain the second attention features corresponding to the third level; the output features of the third level, the first attention features, the corresponding second attention features of the first level, the corresponding second attention features of the second level, and the corresponding second attention features of the third level are fused to obtain the input features of the fourth level; the input features of the fourth level are encoded to obtain the output features of the fourth level, and the corresponding second spatial features of the fourth-level feature extraction are extracted from the output features of the fourth level.

[0056] By using the fourth-level feature extraction, the corresponding second-space feature separation is obtained to obtain multiple target features at different scales.

[0057] S27: Upsample the target features at the largest scale to obtain new target features.

[0058] In one implementation, a 2x2 deconvolution layer can be used to upsample the target features at the largest scale. For example, a 1 / 8 target feature layer can be upsampled to a 1 / 4 target feature layer to obtain new target features.

[0059] S28: Use multiple target features and new target features as target image features.

[0060] The new target features obtained after upsampling and the multiple target features at different scales obtained by separating the corresponding second spatial features using the fourth-level feature extraction are used as the target image features. By segmenting and predicting the target image features, the spatial and semantic information of the image can be obtained.

[0061] In this embodiment, the specific implementation methods of steps S21, S22 and S25 are described in the relevant description of the first embodiment of the image feature extraction method, and will not be repeated here.

[0062] Please see Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the image feature extraction method provided in this application. Figure 4 This is a schematic diagram of the framework of an implementation of the semantic feature extraction network provided in this application; the method includes:

[0063] S31: Acquire spectral image.

[0064] Spectral images are obtained by collecting images of the environment using light in multiple wavelengths.

[0065] S32: Perform dimensionality reduction feature extraction on the spectral image to obtain spectral features of the first order of magnitude, which is smaller than the dimension of the spectral image.

[0066] Spectral images contain anywhere from a few to dozens or even hundreds of channels. Spectral images with varying channel counts, after being processed by a patch embedding module, generate 16 * (the number of channels) feature channels. Without data preprocessing, it is difficult to train a semantic feature extraction network capable of handling spectral images from different sources. Therefore, this implementation first performs dimensionality reduction on the spectral image and then extracts features from the dimensionality-reduced image to obtain spectral features of a first-order dimension. The patch embedding module is used to convert a multi-dimensional image into a preset dimension, which can be one-dimensional.

[0067] In one embodiment, this step is performed by a spectral feature extraction module, which may or may not belong to a semantic feature extraction network. The spectral feature extraction module may consist of two 3x3 convolutional layers and one 1x1 convolutional layer. The input dimension of the first convolutional layer is C, and the output dimension of the last convolutional layer is a first number, which can be determined by the semantic feature extraction network. In a specific embodiment, the semantic feature extraction network used in this application can be a ViT network, in which case the first number can be 3. The number of channels C can support a large number of multispectral and some hyperspectral images. Images with fewer than C channels are padded with 0s, and spectral images with more than C channels can be input after dimensionality reduction using principal component analysis. The number of channels C can be adjusted according to the actual number of spectral bands in the image; no specific limitation is made here.

[0068] S33: Multiple initial spatial features at different scales are obtained by extracting spectral features from spectral images.

[0069] In one embodiment, the scale of each initial spatial feature is smaller than the scale of the spectral image.

[0070] S34: After converting multiple initial spatial features into the second quantity dimension, they are concatenated to obtain the first spatial feature.

[0071] S35: Perform several levels of feature extraction using the first spatial features to obtain the target image features of the spectral image.

[0072] In one embodiment, steps S33-S35 are performed by the semantic feature extraction network. The spectral feature extraction module sends the spectral features of the first dimension to the first spatial feature extraction module of the semantic feature extraction network, so that the first spatial feature extraction module uses the spectral features of the spectral image to extract multiple initial spatial features of different scales. The first spatial feature extraction module uses the Stem layer (i.e., downsampling layer) and BottleNeck layer of ResNet (residual network) for the first layer feature extraction. The Stem layer consists of 3 convolutional layers and 1 max pooling layer. The BottleNeck layer uses the native BottleNeck layer of the ResNet network with 2 encoding layers. Then, the second and third layers of feature extraction are performed using two 3*3 convolutional layers with a stride of 2, respectively. During this process, the feature channels are deepened by a factor of 2 and the feature size is reduced by a factor of 1 / 2, resulting in three initial spatial features of different scales. The dimensions of the three initial spatial features are transformed to a preset dimension using three 1*1 convolutional layers, which are set by the user, for example, 64 dimensions. At this point, image features with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the original image are obtained. Finally, flattening the three initial spatial features transforms them into second-order spatial features. Concatenating the three second-order spatial features yields the first spatial feature. The semantic feature extraction network then sequentially connects several levels of feature encoding layers, spatial feature injection modules, and second-order spatial feature extraction modules to perform several levels of feature extraction using the first spatial feature, resulting in the target image features of the spectral image.

[0073] In one specific embodiment, the semantic feature extraction network of this application is an improvement upon the existing ViT (VisionTransformer) network, which is used to perform image classification tasks. The ViT network can include a position embedding module, a patch embedding module, and four feature encoding layers (Transformer Blocks). Figure 4As shown, the semantic feature extraction network provided in this application adds a first spatial feature extraction module, a four-level spatial feature injection module, and a four-level second spatial feature extraction module to the ViT network. Each level of feature encoding layer is connected to the corresponding level spatial feature injection module and the second spatial feature extraction module. The first spatial feature extraction module is used to extract the first spatial features of the spectral image. The corresponding level spatial feature injection module is used to perform multi-head interactive attention processing using the first spatial features and the corresponding second spatial features extracted from each previous level. Each level spatial feature injection module contains a corresponding number of spatial feature injection modules; for example, the first level spatial feature injection module contains one spatial feature injection module, and the second level spatial feature injection module contains two spatial feature injection modules. The corresponding level feature encoding layer is used to encode the input features of the current level to obtain the output features of the current level. The corresponding level second spatial feature extraction module is used to extract the second spatial features corresponding to the current level feature extraction from the output features of the current level.

[0074] like Figure 4 For the first-level feature extraction, the spatial injection module 10 uses the first spatial features output by the first spatial feature extraction module and the initial coding features of the spectral image output by the block coding module to perform interactive attention processing to obtain the first attention features; the first-level spatial feature injection module fuses the first attention features and the initial coding features to obtain the first-level input features; the feature coding layer 1 encodes the first-level input features to obtain the first-level output features; and the second spatial extraction module 1 extracts the second spatial features corresponding to the first-level feature extraction from the first-level output features.

[0075] For the second-level feature extraction, the spatial injection module 21 performs interactive attention processing using the first spatial features and the output features of the first level to obtain the first attention features; the spatial injection module 20 performs interactive attention processing using the output features of the first level and the corresponding second spatial features extracted from the first-level features to obtain the second attention features; the second-level spatial injection module fuses the output features of the first level, the first attention features, and the second attention features to obtain the input features of the second level; the feature encoding layer 2 encodes the input features of the second level to obtain the output features of the second level; and the second spatial extraction module 2 extracts the corresponding second spatial features from the output features of the second level.

[0076] For the third-level feature extraction, the spatial injection module 32 performs interactive attention processing using the first spatial features and the output features of the second level to obtain the first attention features; the spatial injection module 31 performs interactive attention processing using the output features of the second level and the corresponding second spatial features extracted from the first level to obtain the second attention features corresponding to the first level; the spatial injection module 30 performs interactive attention processing using the output features of the second level and the corresponding second spatial features extracted from the second level to obtain the second attention features corresponding to the second level; the third-level spatial injection module fuses the output features of the second level, the first attention features, the corresponding second attention features of the first level, and the corresponding second attention features of the second level to obtain the input features of the third level; the feature encoding layer 3 encodes the input features of the third level to obtain the output features of the third level; and the second spatial extraction module 3 extracts the corresponding second spatial features from the output features of the third level.

[0077] For the fourth-level feature extraction, spatial injection module 43 performs interactive attention processing using the first spatial features and the output features of the third level to obtain the first attention features; spatial injection module 42 performs interactive attention processing using the output features of the third level and the corresponding second spatial features extracted from the first level to obtain the second attention features corresponding to the first level; spatial injection module 41 performs interactive attention processing using the output features of the third level and the corresponding second spatial features extracted from the second level to obtain the second attention features corresponding to the second level; spatial injection module 40 performs interactive attention processing using the output features of the third level and the corresponding second spatial features extracted from the third level to obtain the second attention features corresponding to the third level; the fourth-level spatial injection module fuses the output features of the third level, the first attention features, the corresponding second attention features of the first level, the corresponding second attention features of the second level, and the corresponding second attention features of the third level to obtain the input features of the fourth level; feature encoding layer 4 encodes the input features of the fourth level to obtain the output features of the fourth level; and the second spatial extraction module 4 extracts the second spatial features corresponding to the fourth-level feature extraction from the output features of the fourth level. Multiple target features at different scales are separated using the second spatial features corresponding to the fourth-level feature extraction.

[0078] The semantic feature extraction network provided in this application uses the native ViT network for feature extraction without requiring any modifications to the ViT network structure. The native ViT network can be trained using multimodal data, such as images, videos, and text, significantly improving its semantic information extraction capabilities compared to networks trained only on image data. Furthermore, the native ViT network is smaller and trains faster than dense prediction networks. Therefore, the semantic feature extraction network obtained by improving the native ViT network in this application can obtain rich semantic and spatial information when extracting image features.

[0079] Please see Figure 5 , Figure 5 This is a flowchart illustrating one embodiment of the semantic segmentation method for images provided in this application. Figure 6 This is a schematic diagram of one embodiment of the spectral image obtained in this application. Figure 7 This is a schematic diagram of an embodiment of the semantic segmentation result of the spectral image obtained in this application; the method includes:

[0080] S51: Extract features from the spectral image to obtain the target image features of the spectral image.

[0081] like Figure 6 As shown, the spectral image can be obtained by collecting environmental data using light across multiple wavelengths. Feature extraction of the spectral image can be performed using the methods provided in the feature extraction method embodiments of any of the above images. For specific feature extraction steps, please refer to the feature extraction method embodiments of the above images, which will not be repeated here.

[0082] S52: Use the features of the target image to perform segmentation prediction on the spectral image to obtain the semantic segmentation result of the spectral image.

[0083] In one embodiment, segmenting and predicting a spectral image can involve classifying the targets contained in the spectral image; for example, displaying the same type of target using the same color and different targets using different colors. Figure 7 As shown, Figure 7 That is to Figure 6 The semantic segmentation result is obtained by segmenting and predicting the target image features obtained through feature extraction.

[0084] Please see Figure 8 , Figure 8 This is a schematic diagram of one embodiment of the electronic device provided in this application.

[0085] The electronic device 80 includes a memory 81 and a processor 82 coupled to each other. The memory 81 stores program instructions, and the processor 82 executes the program instructions to implement the steps in any of the above method embodiments. Specifically, the electronic device 80 may include, but is not limited to, desktop computers, laptops, servers, mobile phones, tablets, etc., and is not limited thereto.

[0086] Specifically, processor 82 controls itself and memory 81 to implement the steps in any of the above method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0087] Please see Figure 9 , Figure 9 This is a schematic diagram of a computer-readable storage medium according to one embodiment of the present application. The computer-readable storage medium 90 stores program instructions 91, which, when executed by a processor, are used to implement the steps in any of the above method embodiments.

[0088] The computer-readable storage medium 90 can specifically be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium that can store computer programs. Alternatively, it can be a server that stores the computer program, which can send the stored computer program to other devices for execution or can also execute the stored computer program itself.

[0089] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0090] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for feature extraction from an image, characterized in that, include: Acquire spectral images, which are obtained by collecting environmental data using light across multiple wavelengths; The first spatial features of the spectral image are extracted using a semantic feature extraction network; as well as, The semantic feature extraction network is used to perform several levels of feature extraction based on the first spatial features to obtain the target image features of the spectral image. Each level of feature extraction includes: performing multi-head interactive attention processing using the first spatial features and the second spatial features corresponding to each previous level of feature extraction to obtain the input features of the current level; encoding the input features of the current level to obtain the output features of the current level; extracting the second spatial features corresponding to the current level of feature extraction from the output features of the current level; and using the second spatial features corresponding to the last level of feature extraction to obtain the target image features. The step of performing multi-head interactive attention processing using the first spatial feature and the corresponding second spatial feature extracted from each previous level to obtain the input feature of the current level includes: performing multi-head interactive attention processing with the first spatial feature as the key element, the corresponding second spatial feature extracted from each previous level as the value element, and the reference feature of the current level as the query element to obtain the input feature of the current level; the reference feature of the first level is the initial encoding feature of the spectral image, and the reference feature of the non-first level is the output feature of the previous level feature extraction layer; and / or, the step of extracting the second spatial feature corresponding to the current level feature extraction from the output feature of the current level includes: performing interactive attention processing with the output feature of the current level as the key element and the value element, and the attention feature of the current level as the query element to obtain the second spatial feature corresponding to the current level feature extraction; the attention feature of the current level is obtained by performing interactive attention processing using the reference feature of the current level and the second spatial feature corresponding to the previous level feature extraction.

2. The method according to claim 1, characterized in that, The process of performing multi-head interactive attention processing, using the first spatial feature as the key element, the corresponding second spatial feature extracted from each previous level as the value element, and the reference feature of the current level as the query element, to obtain the input feature of the current level includes: The first attention feature is obtained by performing interactive attention processing with the reference feature of the current level using the first spatial feature and the reference feature of the current level; and the second attention feature corresponding to the previous level is obtained by performing interactive attention processing with the reference feature of the current level and the corresponding second spatial feature extracted from each previous level. The reference features of the current level, the first attention features, and the second attention features corresponding to each previous level are fused to obtain the input features of the current level.

3. The method according to claim 2, characterized in that, The step of performing interactive attention processing using the first spatial feature and the reference feature of the current level to obtain the first attention feature; and performing interactive attention processing using the reference feature of the current level and the corresponding second spatial feature extracted from each previous level to obtain the second attention feature corresponding to each previous level, includes: The first spatial feature, the second spatial feature corresponding to each previous level of feature, and the reference feature of the current level are respectively standardized to obtain the initial standardized feature, the spatial standardized feature corresponding to each previous level, and the reference standardized feature. Interactive attention processing is performed on the initial normalized feature and the reference normalized feature to obtain the first attention feature; and interactive attention processing is performed on the reference normalized feature and the spatial normalized feature corresponding to each previous level to obtain the second attention feature corresponding to each previous level. And / or, the fusion of the reference features of the current level, the first attention feature, and the second attention features corresponding to each previous level to obtain the input features of the current level includes: The input features of the current level are obtained by adding the product of the first attention feature and the attention parameter, the reference feature of the current level, and the second attention feature corresponding to each previous level.

4. The method according to claim 1, characterized in that, The step of performing interactive attention processing, using the output features of the current level as key elements and value elements, and the attention features of the current level as query elements, to obtain the second spatial features corresponding to the current level feature extraction, includes: The third attention feature is obtained by interactive attention processing using the output features of the current level and the attention features of the current level. By fusing the third attention feature and the current level attention feature, the second spatial feature corresponding to the current level feature extraction is obtained.

5. The method according to claim 4, characterized in that, The step of performing interactive attention processing using the output features of the current level and the attention features of the current level to obtain the third attention feature includes: The output features and attention features of the current level are standardized respectively to obtain the standardized output features and the standardized attention features of the current level. The output normalized features are subjected to interactive attention processing with the normalized attention features of the current level to obtain the third attention feature; And / or, the step of obtaining the second spatial feature corresponding to the current level feature extraction by fusing the third attention feature and the current level attention feature includes: The current level attention feature and the third attention feature are added together to obtain the fused feature; The fused features or the standardized results of the fused features are processed using a feedforward neural network layer to obtain the second spatial features corresponding to the current level feature extraction.

6. The method according to claim 1, characterized in that, Before performing several levels of feature extraction based on the first spatial features to obtain the target image features of the spectral image, the method further includes: The spectral features of the spectral image are image block encoded to obtain feature blocks corresponding to different image blocks in the spectral image, and the spectral features of the spectral image are position encoded to obtain position encoding information corresponding to different image blocks in the spectral image. The initial coded image features of the spectral image are obtained by using the feature blocks and positional coding information corresponding to the different image blocks; And / or, both the second spatial feature and the target image feature are multi-scale features, and the step of extracting the corresponding second spatial feature using the last-level feature to obtain the target image feature includes: The second spatial features corresponding to the last-level feature extraction are used to separate and obtain multiple target features of different scales; The target features at the largest scale are upsampled to obtain new target features; The multiple target features and the new target features are used as the target image features.

7. The method according to claim 1, characterized in that, The semantic feature extraction network includes a first spatial feature extraction module, several levels of feature coding layers connected in sequence, several levels of spatial feature injection modules, and several levels of second spatial feature extraction modules. Each level of the feature coding layer is connected to the corresponding level of the spatial feature injection module and the second spatial feature extraction module. The step of extracting the first spatial feature of the spectral image is performed by the first spatial feature extraction module. The step of using the first spatial feature and the features of each previous level to extract the corresponding second spatial feature for multi-head interaction attention processing is executed by the spatial feature injection module of the corresponding level; The step of encoding the input features of the current level to obtain the output features of the current level is performed by the feature encoding layer of the corresponding level; The step of extracting the second spatial feature corresponding to the current level feature extraction from the output features of the current level is executed by the second spatial feature extraction module of the corresponding level.

8. The method according to claim 1, characterized in that, Prior to extracting the first spatial features of the spectral image, the method further includes: The spectral image is subjected to dimensionality reduction feature extraction to obtain a second first-order dimension of spectral features, wherein the second first-order dimension is smaller than the dimension of the spectral image; And / or, the extraction of the first spatial features of the spectral image includes: Multiple initial spatial features at different scales are extracted using the spectral features of the spectral image, wherein the scale of each initial spatial feature is smaller than the scale of the spectral image. The multiple initial spatial features are converted into first and second quantitative dimensions respectively and then concatenated to obtain the first spatial feature.

9. A semantic segmentation method for images, characterized in that, include: The method described in any one of claims 1 to 8 is used to extract features from a spectral image to obtain target image features of the spectral image, wherein the spectral image is obtained by collecting environmental data using multiple wavelength bands of light. The spectral image is segmented and predicted using the features of the target image to obtain the semantic segmentation result of the spectral image.

10. An electronic device, characterized in that, The device includes a memory and a processor that are coupled to each other. The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the image feature extraction method according to any one of claims 1-8 or the image semantic segmentation method according to claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the image feature extraction method as described in any one of claims 1-8 or the image semantic segmentation method as described in claim 9.

Images