Feature extraction method of neural network based on dual-domain hybrid interaction

Abstract

The application discloses a feature extraction method of a neural network based on double-domain mixed interaction, and applies the method to a multispectral image target recognition field.The method firstly constructs a double-domain two-dimensional multi-scale convolution embedding module to realize feature representation in different scales and enhance the richness of the extracted features;secondly, a mixed-domain sharing module is designed to realize information interaction, promote information interaction between the spectral domain and the spatial domain of the image, reduce the heterogeneity of the spatial and spectral domains, and improve the network feature extraction capability;finally, a pyramid scale fusion block is introduced to directly integrate the features of the interaction between the multi-channel, multi-scale, local and global spectral spaces, so that the advanced semantic features are extracted.The application interactively interprets the double-domain information of the image in the network extraction process, so that the extracted image feature information is more separable, and better robustness and effectiveness are exhibited in the image recognition field.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of network computing and representation learning, and particularly relates to a feature extraction method of a neural network based on double-domain mixed interaction, which can be used to solve the target recognition problem of multispectral image data. BACKGROUND

[0002] The effect of image processing depends largely on the effective representation of image features. The high-dimensional image processing method based on features mainly includes feature dimension reduction and information extraction. Dimension reduction is to select features or spectral bands with discriminative characteristics, and to transmit these features or spectral bands to a classifier for optimized training, which can effectively alleviate the Hughes phenomenon caused by high-dimensional data. Information extraction is to map high-dimensional data to a low-dimensional space, in which the image features have better properties. The traditional feature processing method is easy to extend and simple, but it often depends on the design of a specific task and requires the design of extraction rules based on human experience, and can only obtain the description of hand-crafted features or shallow features, and is difficult to complete the description of deep semantic features, and has limitations in achieving the optimal balance between discriminability and robustness.

[0003] Deep learning (DL) technology, as an efficient feature representation method of feature extraction, has promoted the rapid development of the field of image processing, and a large number of DL-based image processing models have been proposed. They include auto-encoder, recurrent neural network, long short-term memory network, convolutional neural network (CNN), and generative adversarial network. Among them, CNN is the most widely used, which can be divided into 1D CNN, 2D CNN and 3D CNN. The image processing method based on CNN or its variants can realize the exploration of spatial, spectral and spatial-spectral information features, and has achieved satisfactory classification results, but they still face some urgent challenges. Due to the limitation of the receptive field, the CNN-based image processing method deeply excavates local information and cannot capture global features. However, due to the increased network depth, feature redundancy and limited labeled samples, the CNN-based image processing effect cannot achieve the best.

[0004] In recent years, the Transformer, originating from natural language processing, has gained widespread popularity. To advance various visual tasks, a variant of the Transformer, called the Visual Transformer (ViT), has been proposed. This variant segments images into many small patches and treats them as continuous data. Due to the impressive results achieved by ViT, an increasing number of researchers are applying it to image processing. Feature extraction network methods based on Transformer and its variants place greater emphasis on global features and have achieved impressive image processing results. However, these methods still face some challenges. First, they use fixed-scale traditional patch embedding operations to divide images into many small patches, ignoring the multi-scale information of objects in the original image data. Second, they focus on extracting spectral information, largely failing to effectively utilize spatial information. Furthermore, their repeated connection of the ViT encoder leads to gradient vanishing and information loss phenomena. Summary of the Invention

[0005] This invention addresses the shortcomings of existing deep learning-based feature extraction methods by providing a feature extraction method based on a dual-domain hybrid interaction neural network. This method fuses semantic features extracted by the dual-dimension multiscale convolutional embedding (D2MSCE) module and the mixed-domain forward (MDFormer) shared module through a scale fusion block. This leverages the complementary advantages of CNN and Transformer, and interprets rich multi-channel and multi-scale spectral spatial information from a local-global perspective, thereby achieving better multi-spectral target recognition results.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The feature extraction method based on a dual-domain hybrid interaction neural network includes the following steps:

[0008] (1) First, set the relevant parameters, including the learning rate. The batch size is 16, and the number of model training iterations is [number missing].

[0009] (2) Input image data Principal component analysis (PCA) was used to perform dimensionality reduction preprocessing on the original multi-spectral images to obtain the dimensionality-reduced image data. This reduces the computational complexity of subsequent image processing;

[0010] (3) For the reduced-dimensional image data, pixel segmentation is performed with each pixel of the image data as the center to obtain a three-dimensional image cube composed of the pixel and its neighboring pixels.

[0011] (4) Preprocessing the 3D image block data, extracting multi-channel spectral spatial features in the preprocessing module. The preprocessing module includes three branches for parallel processing of the 3D image block data;

[0012] (5) For each branch, use dual-domain two-dimensional multi-scale convolutional embedding to perform multi-scale block embedding operation on the image block after the preprocessing module to obtain the spectral dimension and spatial dimension representation at different scales and enhance the multi-scale characteristics of the target.

[0013] (6) For each branch, after the multi-scale block embedding operation, the hybrid domain sharing module realizes the interactive interpretation of the spectral and spatial domains of the multi-scale spatial-spectral image features, reduces the heterogeneity of the domain space and spectral domain, and thus obtains more discriminative and representative dual-domain image feature information.

[0014] (7) For each branch, a pyramid-shaped scale fusion block is applied to the obtained dual-domain image feature information to directly fuse the spatial-spectral interaction information extracted from different channels, scales and branches, so as to obtain advanced semantic information for subsequent image processing.

[0015] (8) Connect the high-level semantic information extracted from the three branches and use a global average pooling layer, two fully connected layers and a softmax layer in sequence to identify the image target.

[0016] The beneficial effects of this invention are as follows:

[0017] This invention can reduce the bias limitations of multi-scale feature extraction methods. By performing multi-branch dual-domain two-dimensional multi-scale convolution embedding operations on preprocessed image blocks, it can achieve feature extraction of spectral and spatial dimensions at different scales, making the interpreted image information richer.

[0018] When using the Transformer network for feature extraction, this invention simultaneously performs interactive interpretation of multi-scale local features in the spectral and spatial domains, enhancing the global information association capability of local features, realizing the extraction and sharing of multi-angle feature information, and thus obtaining more discriminative and representative dual-domain image feature information.

[0019] This invention applies a pyramid-scale fusion mechanism to integrate spatial-spectral interaction information extracted from different channels, scales, and branches, enhances the local correlation of global features, reduces the perceptual differences between different features, realizes semantic information sharing and interactive interpretation, and provides more robust and separable features for subsequent image processing. Attached Figure Description

[0020] Figure 1 This is a flowchart of the feature extraction method of the present invention based on a dual-domain hybrid interaction neural network;

[0021] Figure 2 This is a schematic diagram of the dual-domain two-dimensional multi-scale convolution embedding module of the present invention;

[0022] Figure 3 This is a schematic diagram of the hybrid domain sharing module of the present invention;

[0023] Figure 4 A schematic diagram of a pyramid-shaped scale fusion block;

[0024] Figure 5 The images show a comparison of the visual results of multispectral target recognition using the present invention and existing feature extraction methods. (a) is the ground truth image, (b) is the SVM method, (c) is the 2DCNN method, (d) is the 3DCNN method, (e) is the ViT method, and (f) is the method of the present invention. Detailed Implementation

[0025] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0026] This invention discloses a feature extraction method based on a dual-domain hybrid interaction neural network, the specific flowchart of which is shown below. Figure 1 As shown, the method includes:

[0027] Step 1: Set the relevant parameters;

[0028] The relevant parameters to be set include the learning rate. Batch size, the number of training iterations for the model.

[0029] Step 2: Image dimensionality reduction preprocessing;

[0030] Principal component analysis was used to analyze the original input image data. PCA dimensionality reduction is performed to obtain the dimensionality-reduced image data. This allows for the extraction of a small number of spectral bands containing important information, preserving the 3D characteristics of image data and reducing subsequent computational complexity.

[0031] Step 3: Construct network input;

[0032] Pixel segmentation is performed with each pixel as the center, resulting in a 3D image cube composed of the pixel and its neighboring pixels, which serves as the input to the network. Specifically, this includes:

[0033] Assuming the original image The size is After dimensionality reduction, the resulting image data matrix is ​​as follows: Image matrix Extract the image data matrix obtained after dimensionality reduction. 3D image blocks ,in Representing an image In the middle of the pixel Centered on, the neighborhood size is A three-dimensional image block composed of all its pixels.

[0034] Step 4: Image extraction and preprocessing.

[0035] The 3D image block data is input into a preprocessing module to extract multi-channel spectral spatial features. This preprocessing module comprises three branches, each including a composite convolution module. The operation involves inputting image data into the composite convolution module of each branch. The composite convolution module uses two 2D regular convolutional layers and one 2D deep convolutional layer to initially extract shallow image features. These features are then input into a batch normalization layer and a linear correction unit to enhance generalization ability and avoid gradient vanishing.

[0036] Step 5: Two-domain two-dimensional multi-scale convolution embedding operation;

[0037] like Figure 2 As shown, the dual-domain two-dimensional multi-scale convolutional embedding module includes a spectral multi-scale convolutional embedding branch and a spatial multi-scale convolutional embedding branch. The former is used to obtain spectral feature representations at different scales, and the latter is used to obtain spatial feature representations at different dimensions. Specifically, the dual-domain two-dimensional multi-scale convolutional embedding module is applied to each preprocessing branch to perform multi-scale block embedding operations to obtain spectral and spatial dimension representations at different scales. The spectral multi-scale convolutional embedding branch and the spatial multi-scale convolutional embedding branch specifically include:

[0038] The spectral multi-scale convolutional embedding branch includes three parallel units and skip connections. Each unit is used to extract spectral information at small, medium, and large scales, respectively. Each unit includes a convolutional kernel with a size of [missing information]. The system consists of a 3D convolutional layer, a batch normalization (BN) layer, and a ReLU layer. The spectral multi-scale convolutional embedding branch introduces an element-wise stacking operation to integrate features at three scales to obtain multi-scale spectral features. The spectral multi-scale convolutional embedding branch can be represented as:

[0039] (1)

[0040] in, and These are the input and output data of the spectral multi-scale convolutional embedding branch, respectively. This represents a 3D convolution operation. This indicates BN layer operations. Represents ReLU layer operations For addition operations, This indicates a batch embedding operation.

[0041] The spatial multi-scale convolutional embedding branch includes three parallel units and skip connections. Each unit extracts spatial information at small, medium, and large scales, respectively. Each unit includes a convolutional kernel with a size of [missing information]. It consists of a 3D convolutional layer, a BN layer, and a ReLU layer. The spatial multi-scale convolutional embedding branch can be represented as follows:

[0042] (2)

[0043] in, The input data represents the spatial multi-scale convolutional embedding branch. This represents the output data of the spatial multi-scale convolution embedding branch.

[0044] Step 6: Mixed Domain Sharing Operation;

[0045] like Figure 3 As shown, to enhance feature extraction capabilities and information interaction between the spectral and spatial domains, a hybrid domain sharing module is used to achieve interactive interpretation of multi-scale spatial-spectral image features in both the spectral and spatial domains, thereby obtaining more discriminative and representative dual-domain image feature information. This includes:

[0046] First, the spectral dimension representations at different scales are mapped to three weight matrices, namely... The spatial dimensionality representations at different scales are mapped to three weight matrices. .

[0047] Secondly, to enhance high-quality information interaction while reducing low-quality information interaction, a hybrid domain attention mechanism is proposed to obtain more discriminative mutual features from the spectral and spatial domains. Matrix addition is applied to achieve efficient aggregation of the spectral and spatial domains to generate a cross-domain weight matrix. The formula is expressed as follows:

[0048] (3)

[0049] (4)

[0050] (5)

[0051] in, This represents the learning parameter matrix in the spectral domain. This represents the learning parameter matrix in the spatial domain. By reweighting the spectral dimension representations of the input at different scales through self-attention and cross-attention, high-quality spectral information and spectral spatial relationships are obtained.

[0052] (6)

[0053] (7)

[0054] in, It is the softmax function. It is a cross-interest in the spectral domain. yes The channel dimension, It is the output characteristic of the spectral domain.

[0055] By reweighting the spatial dimension representations of the input at different scales through self-attention and cross-attention, the relationship between spatial information and spectral space is obtained:

[0056] (8)

[0057] (9)

[0058] in, It is a cross-domain concern. It is the output feature of the spatial domain.

[0059] Skip connections are employed in both the spectral and spatial domains to integrate spectral features, spatial features, and spectral-spatial adaptive cross-features, thereby achieving more complex and comprehensive feature representations.

[0060] (10)

[0061] in, Indicates cascading operation, This indicates the output characteristics of the hybrid domain shared module.

[0062] Step 7: Pyramid-scale fusion block fusion features;

[0063] like Figure 4 As shown, to enhance feature extraction capabilities and information interaction between the spectral and spatial domains, a pyramid-shaped scale fusion block is applied to directly fuse spatial-spectral interaction information extracted from different channels, scales, and branches, obtaining high-level semantic information, including:

[0064] The output characteristics of the three hybrid domain shared modules are denoted as follows: ,in, These represent the channel dimensions. For the three output features... Perform downsampling separately to obtain uniform features , and ,in The channel dimension is [value]. Using efficient units, a convolution is performed once on each of the three input features, with a kernel size of [value]. 2D convolutional layer operations yield three efficient features and And in and Use step size 2D average pooling is used for spatial dimensionality reduction to alleviate computational burden. The spatially reduced features are then flattened into... and Calculate the fusion similarity matrix:

[0065] (11)

[0066] in, Representing the fusion similarity matrix, applying the fusion similarity matrix Reweighting And execute the merge function to restore the channel size to .

[0067] Output features of pyramid-scale fusion blocks Represented as:

[0068] (12)

[0069] in, It is a fusion operation, including a reshape operation and a convolution kernel size of [missing information]. 2D convolutional layer operations.

[0070] Step 8: Connect the image features extracted from the three branches, and use a global average pooling layer, two fully connected layers and a softmax layer to identify the image target.

[0071] A cascaded operation is used to aggregate the output features of three pyramid-shaped scale fusion blocks, generating more robust and advanced semantic features for multispectral image processing.

[0072] To verify the algorithm's performance, experiments were conducted on an NVIDIA GeForce RTX 2060 SUPER GPU system with 16.0 GB of memory, using the Tensorflow 2.3 programming platform. The experiments used the Indian Pines dataset, which is 83×86 pixels in size, with 204 spectral bands and 6 target categories.

[0073] Figure 5 This paper compares the visual results of the present invention with existing feature extraction methods for multispectral target recognition algorithms. Figure 5 The right side shows illustrations of different target categories. The proposed method is compared with existing feature extraction algorithms on the Salinas-A dataset, which mainly include SVM, 2DCNN, 3DCNN, and ViT. Figure 5 In the diagram, (a) is the ground truth image, (b)-(e) are the recognition results using SVM, 2DCNN, 3DCNN, and ViT methods, respectively, and (f) is the result of this invention. Experimental results show that, with the ground truth image as a reference, the method of this invention achieves the best target recognition performance, exhibits the best smoothness in target classification across different categories, and has less noise. Comparison of the experimental results reveals that the traditional SVM method has a large number of misidentified pixels in each category, while the feature extraction network can better identify these categories. Compared to other feature extraction network methods, this invention can effectively reduce the probability of misidentification for both the target Lettcue_romaine_7wk (with a large dataset) and the target Lettcue_romaine_5wk (with a small dataset), demonstrating strong separable feature extraction capabilities and superior recognition performance. The method of this invention introduces the idea of ​​dual-domain hybrid interactive features into network feature extraction methods, and designs a feature extraction method based on spectral and spatial domains, and local and global shared collaboration. This method can effectively improve the discriminative feature extraction capability of the network, achieve effective representation of the high-level semantic features of the target, and ultimately obtain better image recognition performance.

[0074] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A feature extraction method based on a dual-domain hybrid interaction neural network, characterized in that, Includes the following steps: Step 1: Set the relevant parameters of the neural network, including learning rate, batch size, and number of model iterations; Step 2: Use principal component analysis to reduce the dimensionality of the original multi-spectral image to obtain the dimensionality-reduced image data; Step 3: For the dimension-reduced image data, perform pixel segmentation with each pixel as the center to obtain a three-dimensional image cube composed of the pixel and its neighboring pixels. Step 4: Input the 3D image block data into the preprocessing module to extract multi-channel spectral spatial features. The preprocessing module includes three branches to process the 3D image block data in parallel. Step 5: For each branch, use the dual-domain two-dimensional multi-scale convolutional embedding module to perform multi-scale block embedding operation on the image block after the preprocessing module to obtain the representation of spectral and spatial dimensions at different scales. Step 6: For each branch, after the multi-scale block embedding operation, the hybrid domain sharing module is used to realize the interactive interpretation of the spectral and spatial domains of the multi-scale spatial spectrum image features to obtain the dual-domain feature information of the image. The interactive interpretation of multi-scale spatial-spectral image features in the spectral and spatial domains using the hybrid domain sharing module includes: First, the spectral dimension representations at different scales are mapped to three weight matrices through linear mapping, namely... The spatial dimensionality representations at different scales are mapped to three weight matrices, namely... ; Matrix addition is used to achieve efficient aggregation of the spectral and spatial domains to generate a cross-domain weight matrix. : (3) (4) (5) in, This represents the learning parameter matrix in the spectral domain. The learning parameter matrix in the spatial domain is used to reweight the spectral dimension representations at different scales of the input through self-attention and cross-attention, thereby obtaining the relationship between spectral information and spectral space. (6) (7) in, It is the softmax function. It is a cross-interest in the spectral domain. yes The channel dimension, It is the output characteristic in the spectral domain; By reweighting the spatial dimension representations of the input at different scales through self-attention and cross-attention, the relationship between spatial information and spectral space is obtained: (8) (9) in, It is a cross-domain concern. It is the output feature of the spatial domain; Skip connections are employed in the spectral and spatial domains to integrate spectral features, spatial features, and spectral-spatial adaptive cross-features. (10) in, Indicates a cascading operation. This indicates the output characteristics of the hybrid domain shared module; Step 7: For each branch, the obtained image dual-domain feature information is fused directly using a pyramid-style scale fusion block to fuse the spatial-spectral interaction information extracted from different channels, scales and branches, thereby obtaining high-level semantic information. Step 8: Connect the high-level semantic information extracted from the three branches, and use a series of interconnected global average pooling layers, two fully connected layers and a softmax layer to identify the target in the image.

2. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1, characterized in that, Step 3 includes: Assuming the original image The size is After dimensionality reduction, the resulting image data matrix is: , The channel dimension is the value after dimensionality reduction. The resulting image data matrix is ​​extracted after dimensionality reduction. 3D image blocks ,in This indicates that the image data matrix is ​​obtained after dimensionality reduction. In the middle of the pixel Centered on, the neighborhood size is A three-dimensional image block composed of all its pixels.

3. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1, characterized in that, Step 4 includes inputting the 3D image block data into a preprocessing module. The preprocessing module includes three branches, each of which includes a composite convolution module. The composite convolution module uses two 2D conventional convolutional layers and one 2D depth convolutional layer to perform preliminary extraction of shallow image features from the 3D image block data before inputting it into a batch normalization layer and a correction linear unit.

4. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1, characterized in that, The dual-domain two-dimensional multi-scale convolutional embedding module in step 5 includes a spectral multi-scale convolutional embedding branch and a spatial multi-scale convolutional embedding branch. The former is used to obtain spectral feature representations at different scales, and the latter is used to obtain spatial feature representations at different dimensions. The spectral multi-scale convolutional embedding branch includes three parallel units, used to extract spectral information at small, medium, and large scales, respectively. Each unit includes a concatenated convolutional kernel of size [size missing]. A 3D convolutional layer, a BN layer, and a ReLU layer: (1) in, This represents the input image data. The output data of the spectral multi-scale convolution embedding branch. This represents a 3D convolution operation. Indicates BN layer operation, Indicates ReLU layer operation, For addition operations, Indicates batch embedding operation; The spatial multi-scale convolutional embedding branch includes three parallel units, used to extract small-scale, medium-scale, and large-scale spatial information, respectively. Each unit includes a concatenated convolutional kernel of size [size missing]. A 3D convolutional layer, a BN layer, and a ReLU layer: (2) in, The input data represents the spatial multi-scale convolutional embedding branch. This represents the output data of the spatial multi-scale convolution embedding branch.

5. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1, characterized in that, In step 7, a pyramid-shaped scale fusion block is used to directly fuse the spatial-spectral interaction information extracted from different channels, scales, and branches to obtain high-level semantic information, including: The output characteristics of the branches containing the three hybrid domain shared modules are denoted as follows: ,in, These represent the channel dimensions; For the three output features Perform downsampling separately to obtain uniform features , and ; Using efficient units, perform one convolution on each of the three unified features, with a kernel size of [missing value]. 2D convolutional layer operations yield three efficient features and ,exist and Use step size The 2D average pooling operation is used for spatial dimensionality reduction, and the spatially reduced features are flattened into... and : (11) in, Representing the fusion similarity matrix, applying the fusion similarity matrix Reweighting And execute the merge function to restore the channel size to ; Output features of pyramid-scale fusion blocks Represented as: (12) in, It is a fusion operation, including a reshape operation and a convolution kernel size of [missing information]. 2D convolutional layer operations.

Images