Feature extraction method of neural network based on double-domain hybrid interaction

Abstract

The invention discloses a feature extraction method of a neural network based on double-domain hybrid interaction, and the method is applied to the field of multi-spectral image target recognition. The method comprises the following steps: firstly, constructing a two-domain-based two-dimensional multi-scale convolution embedding module to realize feature representation on different scales and enhance the richness of extracted features; secondly, designing a mixed domain sharing module to realize information interaction, promoting information interaction between an image spectral domain and a spatial domain, reducing heterogeneity of the spatial domain and the spectral domain, and realizing improvement of network feature extraction capability; and finally, introducing a pyramid scale fusion block to directly integrate features of interaction between multi-channel, multi-scale, local and global spectral spaces, and realizing extraction of advanced semantic features. According to the method, the image double-domain information is interactively interpreted in the network extraction process, so that the extracted image feature information has better separability, and better robustness and effectiveness are shown in the field of image recognition.

Description

Technical Field

[0001] The present invention belongs to the field of network computing and representation learning technology, and specifically relates to a feature extraction method of a neural network based on dual-domain hybrid interaction, which can be used to solve the problem of target recognition of multi-spectral image data. Background Art

[0002] The effect of image processing depends largely on the effective representation of image features. Feature-based high-dimensional image processing methods mainly include feature dimensionality reduction and information extraction. Dimensionality reduction is to select features or spectral segments with discriminative characteristics and transfer these features or spectral segments to classifiers for optimization training, which can effectively alleviate the Hughes phenomenon caused by high-dimensional data. Information extraction is to map high-dimensional data to low-dimensional space. In this low-dimensional feature space, image features have better properties. Traditional feature processing methods are easy to expand and simple and clear, but they often rely on specific task design and require extraction rules to be designed based on human experience. They can only obtain manual features or shallow feature descriptions, and it is difficult to complete the description of deep semantic features. They have limitations in achieving an optimal balance between discriminability and robustness.

[0003] As an efficient feature representation method for feature extraction, deep learning (DL) technology has promoted the rapid development of image processing. A large number of DL-based image processing models have been proposed, including autoencoders, recursive neural networks, long short-term memory networks, convolutional neural networks (CNNs), generative adversarial networks, etc. Among them, CNN is the most widely used and can be divided into 1D CNN, 2D CNN, and 3D CNN. Image processing methods based on CNN or its variants can achieve feature exploration of spatial, spectral, and spatial-spectral information, and have achieved satisfactory classification results, but they still face some pressing challenges. Due to the limitation of the receptive field, CNN-based image processing methods deeply mine local information and cannot capture global features. However, due to the increased network depth, feature redundancy, and limited labeled samples, the image processing effect based on CNN cannot be optimized.

[0004] In recent years, the Transformer, which originated from natural language processing, has gained popularity. In order to promote the development and process of various visual tasks, a Transformer variant, called Visual Transformer (ViT), was proposed, which divides the image into many small patches and treats them as continuous data. Due to the impressive results achieved by ViT, more and more researchers have also applied ViT to the field of image processing. Feature extraction network methods based on Transformer and its variants pay more attention to the characteristics of global features and have achieved impressive image processing effects in the field of image processing. However, these methods still face some problems. First, they use traditional patch embedding operations with fixed scales to divide the image into many small patches, ignoring the multi-scale information of the target in the original data of the image. Second, they focus on extracting spectral information and are largely unable to effectively use spatial information. They repeatedly connect the encoder of ViT, resulting in gradient vanishing and information loss phenomena. Summary of the invention

[0005] In view of the shortcomings of the existing feature extraction methods based on deep learning, the present invention provides a feature extraction method based on a dual-domain hybrid interaction neural network. The method fuses the semantic features extracted by the dual-domain two-dimensional multiscale convolutional embedding module (D2MSCE) and the mixed domain (MixedDomain Former, MDFormer) sharing module through the scale fusion block, giving full play to the complementary advantages of CNN and Transformer, and interpreting rich multi-channel and multi-scale spectral spatial information from a local-global perspective, thereby obtaining better multi-spectral target recognition results.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] The feature extraction method of the neural network based on dual-domain hybrid interaction includes the following steps:

[0008] (1) First, set the relevant parameters, including the learning rate , batch size is 16, model iteration training times;

[0009] (2) Input image data , use the principal component analysis method to perform PCA dimension reduction preprocessing on the original multi-spectral image to obtain the image data after dimension reduction , reducing the computational complexity of subsequent image processing;

[0010] (3) for the image data after dimensionality reduction, pixel segmentation is performed with each pixel of the image data as the center to obtain a three-dimensional image cube consisting of the pixel and its neighboring pixels;

[0011] (4) Preprocessing the 3D image block data to extract multi-channel spectral spatial features in a preprocessing module. The preprocessing module includes three branches to perform parallel processing on the 3D image block data;

[0012] (5) For each branch, dual-domain two-dimensional multi-scale convolution embedding is used to perform multi-scale block embedding operations on the image blocks after the preprocessing module to obtain the representation of spectral dimensions and spatial dimensions at different scales and enhance the multi-scale characteristics of the target;

[0013] (6) For each branch, after the multi-scale block embedding operation, the hybrid domain sharing module realizes the interactive interpretation of the multi-scale spatial-spectral image features in the spectral domain and the spatial domain, reduces the heterogeneity of the spatial and spectral domains, and thus obtains more discriminative and representative image dual-domain feature information;

[0014] (7) For each branch, the obtained image dual-domain feature information is subjected to a pyramid scale fusion block to directly fuse the spatial-spectral interaction information extracted from different channels, scales, and branches to obtain high-level semantic information for subsequent image processing;

[0015] (8) The high-level semantic information extracted by the three branches is connected, and the image objects are recognized using a global average pooling layer, two fully connected layers, and a softmax layer connected in series.

[0016] The beneficial effects of the present invention are:

[0017] The present invention can reduce the preference limitations of multi-scale feature extraction methods. By performing a multi-branch dual-domain two-dimensional multi-scale convolution embedding operation on the preprocessed image blocks, feature extraction of spectral dimensions and spatial dimensions of different scales can be achieved, making the interpreted image information richer.

[0018] When applying the Transformer network for feature extraction, the present invention simultaneously performs interactive interpretation of multi-scale local features in the spectral domain and the spatial domain, thereby enhancing the global information association capability of local features, realizing the extraction and sharing of multi-angle feature information, and thereby obtaining more discriminative and representative image dual-domain feature information.

[0019] The present invention applies a pyramid scale fusion mechanism to integrate the spatial-spectral interaction information extracted between different channels, scales and branches, enhances the local correlation of global features, reduces the perceptual differences of different features, realizes semantic information sharing and interactive interpretation, and provides more robust and separable features for subsequent image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the feature extraction method of the neural network based on dual-domain hybrid interaction of the present invention;

[0021] Figure 2 Schematic diagram of a dual-domain two-dimensional multi-scale convolutional embedding module of the present invention;

[0022] Figure 3 It is a schematic diagram of a hybrid domain sharing module of the present invention;

[0023] Figure 4 Schematic diagram of pyramid scale fusion block;

[0024] Figure 5 Comparison diagram of visual results of multi-spectral target recognition between the present invention and the existing feature extraction method, (a) is the truth map, (b) is the SVM method, (c) is the 2DCNN method, (d) is the 3DCNN method, (e) is the ViT method, and (f) is the method of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and implementation examples.

[0026] The present invention discloses a feature extraction method of a neural network based on dual-domain hybrid interaction, and the specific flow chart is as follows: Figure 1 As shown, the method includes:

[0027] Step 1. Set relevant parameters;

[0028] The relevant parameters set include the learning rate , batch size, number of model iterations for training.

[0029] Step 2: Image dimension reduction preprocessing;

[0030] The original input image data is analyzed using principal component analysis Perform PCA dimensionality reduction processing to obtain the image data after dimensionality reduction , to achieve the extraction of a small number of spectral segments containing important information, retain the 3D characteristics of the image data, and reduce the subsequent calculation complexity.

[0031] Step 3: Construct network input;

[0032] Pixel segmentation is performed with each pixel as the center to obtain a three-dimensional image cube consisting of the pixel and its neighboring pixels as the input of the network, which specifically includes:

[0033] Assume that the original image The size is , after dimensionality reduction, the image data matrix is , the image matrix , extract the image data matrix obtained after dimensionality reduction 3D image blocks ,in Representing images Pixels The center is and the neighborhood size is A three-dimensional image block composed of all pixels.

[0034] Step 4: image extraction preprocessing operation;

[0035] The three-dimensional image block data is input into the preprocessing module to extract multi-channel spectral spatial features. The preprocessing module includes three branches, each of which includes a composite convolution module. The operation is that the image data is input into the composite convolution module of each branch. The composite convolution module uses two 2D conventional convolution layers and one 2D deep convolution layer to perform preliminary extraction of shallow features of the image data, and then inputs the batch normalization layer and the rectified linear unit to enhance the generalization ability and avoid gradient disappearance.

[0036] Step 5: Dual-domain two-dimensional multi-scale convolution embedding operation;

[0037] like Figure 2 As shown, the dual-domain two-dimensional multi-scale convolution embedding module includes a spectral multi-scale convolution embedding branch and a spatial multi-scale convolution embedding branch. The former is used to obtain spectral feature representations of different scales, and the latter is used to obtain spatial feature representations of different dimensions. The specific operation is to apply the dual-domain two-dimensional multi-scale convolution embedding module to perform multi-scale block embedding operations in each preprocessing branch to obtain representations of spectral dimensions and spatial dimensions at different scales. The spectral multi-scale convolution embedding branch and the spatial multi-scale convolution embedding branch specifically include:

[0038] The spectral multi-scale convolution embedding branch includes three parallel units and skip connections, each unit is used to extract small-scale, medium-scale and large-scale spectral information, and each unit includes a convolution kernel size of The spectral multi-scale convolutional embedding branch introduces an element-wise superposition operation to integrate three scale features to obtain multi-scale spectral features. The spectral multi-scale convolutional embedding branch can be expressed as:

[0039] (1)

[0040] in, and are the input data and output data of the spectral multi-scale convolution embedding branch, respectively. represents a 3D convolution operation, Indicates the BN layer operation, ReLU layer operation For the addition operation, Represents a batch embedding operation.

[0041] The spatial multi-scale convolution embedding branch includes three parallel units and skip connections, each of which extracts small-scale, medium-scale and large-scale spatial information respectively. Each unit includes a convolution kernel size of 3D convolutional layer, a BN layer and a ReLU layer. The spatial multi-scale convolutional embedding branch can be expressed as follows:

[0042] (2)

[0043] in, represents the input data of the spatial multi-scale convolutional embedding branch, Represents the output data of the spatial multi-scale convolutional embedding branch.

[0044] Step 6: Hybrid domain sharing operation;

[0045] like Figure 3 As shown in the figure, in order to enhance the feature extraction capability and the information interaction capability between the spectral domain and the spatial domain, the hybrid domain sharing module is used to realize the interactive interpretation of the spectral domain and the spatial domain of the multi-scale spatial spectrum image features, thereby obtaining more discriminative and representative image dual-domain feature information. Including:

[0046] First, by mapping the spectral dimension representations at different scales into three weight matrices, , mapping spatial dimension representations at different scales to three weight matrices .

[0047] Secondly, in order to enhance high-quality information interaction while reducing low-quality information interaction, a hybrid domain attention mechanism is proposed to obtain more discriminative mutual features from the spectral domain and the spatial domain. Matrix addition operation is applied to achieve effective aggregation of the spectral domain and the spatial domain to generate a cross-domain weight matrix The formula is as follows:

[0048] (3)

[0049] (4)

[0050] (5)

[0051] in, represents the learning parameter matrix in the spectral domain, is the learning parameter matrix of the spatial domain. The spectral dimension representations at different scales of the input are reweighted through self-attention and cross-attention to obtain high-quality spectral information and spectral spatial relationships:

[0052] (6)

[0053] (7)

[0054] in, is the softmax function, is the cross-focus of the spectrum domain, yes The channel dimension of is the output feature in the spectral domain.

[0055] The spatial dimension representations at different scales of the input are re-weighted through self-attention and cross-attention to obtain spatial information and spectral spatial relationships:

[0056] (8)

[0057] (9)

[0058] in, is the cross-focus of the spatial domain, is the output feature in the spatial domain.

[0059] Jump connections are used in the spectral and spatial domains to integrate spectral features, spatial features, and spectral-spatial adaptive mutual features to achieve more complex and comprehensive feature representation:

[0060] (10)

[0061] in, Indicates cascade operation, Represents the output features of the mixed-domain shared module.

[0062] Step 7: pyramid scale fusion block fusion features;

[0063] like Figure 4 As shown in the figure, in order to enhance the feature extraction capability and the information interaction capability between the spectral domain and the spatial domain, the pyramid scale fusion block is used to directly fuse the spatial-spectral interaction information extracted from different channels, scales and branches to obtain high-level semantic information, including:

[0064] The output features of the three hybrid domain shared modules are recorded as: ,in, Respectively represent the channel size. For the three output features Downsample separately to obtain unified features , and ,in is the channel dimension. The efficient unit is used to perform a convolution on each of the three input features with a kernel size of The 2D convolution layer operation obtains three efficient features and , and in and The step size used is The 2D average pooling operation is used to reduce the spatial dimension to reduce the computational burden. The features after spatial dimension reduction are flattened into and , calculate the fusion similarity matrix:

[0065] (11)

[0066] in, Represents the fusion similarity matrix, and applies the fusion similarity matrix Reweighting And execute the merge function to restore the channel size to .

[0067] Output features of pyramid scale fusion block It is expressed as:

[0068] (12)

[0069] in, is a fusion operation, including the reshape operation and the convolution kernel size 2D convolutional layer operation.

[0070] Step 8: Connect the image features extracted by the three branches and use a global average pooling layer, two fully connected layers and a softmax layer to identify image targets.

[0071] A cascade operation is adopted to aggregate the output features of three pyramid-scale fusion blocks to generate more robust and higher-level semantic features for multispectral image processing.

[0072] In order to verify the performance of the algorithm, the experimental verification was carried out on the NVIDIA GeForce RTX 2060 SUPER GPU system with 16.0 GB memory and Tensorflow2.3 programming platform. The experiment uses the Indian Pines dataset with a size of 83×86, 204 spectral bands and 6 target categories.

[0073] Figure 5 The visual results of the multi-spectral target recognition algorithm of the present invention are compared with those of the existing feature extraction method. Figure 5 The right side shows the legend of different target categories. The proposed method is compared with the existing feature extraction algorithms on the Salinas-A dataset. The algorithms mainly include: SVM, 2DCNN, 3DCNN, and ViT. Figure 5 In the figure, (a) is the truth map, (b)-(e) are the recognition result maps of SVM, 2DCNN, 3DCNN and ViT methods respectively, and (f) is the result map of the present invention. The experimental results show that: with the truth map as a reference, the recognition effect of the target using the method of the present invention is the best, the classification smoothness of targets of different categories is the best, and there are fewer noise points. Through the comparison of the experimental result maps, it is found that the traditional method SVM has more pixels with recognition errors in each category, while the feature extraction network can better identify these categories. Compared with other feature extraction network methods, the present invention can effectively reduce the probability of recognition errors in both the target Lettcue_romaine_7wk with a large amount of data in the data set and the target Lettcue_romaine_5wk with a small amount of data, showing a strong separable feature extraction ability and having a better recognition effect. The method of the present invention introduces the idea of ​​dual-domain hybrid interactive features into the network feature extraction method, and designs a feature extraction method based on spectral domain and spatial domain, local and global shared collaboration, which can effectively improve the network's discriminative feature extraction ability, realize the effective representation of the target's high-level semantic features, and ultimately obtain better image recognition performance.

[0074] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A feature extraction method based on a neural network with dual-domain hybrid interaction, characterized in that: The steps include: Step 1: Set the relevant parameters of the neural network, including learning rate, batch size, and number of model iteration training; Step 2: Use the principal component analysis method to reduce the dimension of the original multi-spectral image to obtain the image data after dimension reduction; Step 3: for the image data after dimensionality reduction, pixel segmentation is performed with each pixel point as the center to obtain a three-dimensional image cube consisting of the pixel point and its neighboring pixel points; Step 4: input the 3D image block data into a preprocessing module to extract multi-channel spectrum space features, wherein the preprocessing module includes three branches to perform parallel processing on the 3D image block data; Step 5: For each branch, a dual-domain two-dimensional multi-scale convolution embedding module is used to perform a multi-scale block embedding operation on the image blocks after the preprocessing module to obtain representations of spectral dimensions and spatial dimensions at different scales; Step 6: For each branch, after the multi-scale block embedding operation, the hybrid domain sharing module is used to realize the interactive interpretation of the spectral domain and spatial domain of the multi-scale spatial spectrum image features to obtain the image dual-domain feature information; Step 7: For each branch, the obtained image dual-domain feature information is directly fused using a pyramid scale fusion block to fuse the spatial-spectral interaction information extracted from different channels, scales, and branches to obtain high-level semantic information; Step 8: Connect the high-level semantic information extracted by the three branches and use the global average pooling layer, two fully connected layers and a softmax layer connected in series to identify image targets.

2. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1 is characterized in that: The step 3 comprises: Assume that the original image The size is , after dimensionality reduction, the image data matrix is , is the channel dimension after dimensionality reduction, and the image data matrix is ​​obtained after dimensionality reduction. 3D image blocks ,in Indicates the image data matrix obtained after dimensionality reduction Pixels The center is and the neighborhood size is A three-dimensional image block composed of all pixels.

3. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1 is characterized in that: The step 4 includes inputting the three-dimensional image block data into a preprocessing module, the preprocessing module includes three branches, each branch includes a composite convolution module, the composite convolution module uses two 2D conventional convolution layers and one 2D deep convolution layer to perform preliminary extraction of shallow features of the three-dimensional image block data, and then inputs the data into a batch normalization layer and a rectified linear unit.

4. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1 is characterized in that: The dual-domain two-dimensional multi-scale convolution embedding module in step 5 includes a spectral multi-scale convolution embedding branch and a spatial multi-scale convolution embedding branch, the former is used to obtain spectral feature representations of different scales, and the latter is used to obtain spatial feature representations of different dimensions, wherein: The spectral multi-scale convolution embedding branch includes three parallel units, which are used to extract small-scale, medium-scale and large-scale spectral information respectively. Each unit includes a convolution kernel of size 1 connected in series. A 3D convolutional layer, a BN layer and a ReLU layer: (1) in, represents the input image data, is the output data of the spectral multi-scale convolution embedding branch, represents a 3D convolution operation, Indicates the BN layer operation, represents the ReLU layer operation, For the addition operation, Represents a batch embedding operation; The spatial multi-scale convolution embedding branch includes three parallel units for extracting small-scale, medium-scale and large-scale spatial information, respectively. Each unit includes a convolution kernel of size 1 connected in series. A 3D convolutional layer, a BN layer and a ReLU layer: (2) in, Represents the input data of the spatial multi-scale convolutional embedding branch Represents the output data of the spatial multi-scale convolutional embedding branch.

5. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1 is characterized in that: In step 6, the interactive interpretation of the spectral domain and the spatial domain of the multi-scale spatial spectrum image features by using the hybrid domain sharing module includes: First, the spectral dimension representations at different scales are mapped to three weight matrices through linear mapping, which are , mapping the spatial dimension representations at different scales to three weight matrices, respectively ; Apply matrix addition operations to achieve efficient aggregation of spectral and spatial domains to generate a cross-domain weight matrix : (3) (4) (5) in, represents the learning parameter matrix in the spectral domain, is the learning parameter matrix in the spatial domain. The spectral dimension representations at different scales of the input are reweighted through self-attention and cross-attention to obtain the spectral information and spectral spatial relationship: (6) (7) in, is the softmax function, is the cross-focus of the spectrum domain, yes The channel dimension of is the output feature in the spectral domain; The spatial dimension representations at different scales of the input are re-weighted through self-attention and cross-attention to obtain spatial information and spectral spatial relationships: (8) (9) in, is the cross-focus of the spatial domain, is the output feature in the spatial domain. Jump connections are used in the spectral and spatial domains to integrate spectral features, spatial features, and spectral-spatial adaptive mutual features: (10) in, Indicates cascade operation, Represents the output features of the mixed-domain shared module.

6. The feature extraction method based on a dual-domain hybrid interaction neural network according to claim 1 is characterized in that: In step 7, the pyramid scale fusion block is used to directly fuse the spatial-spectral interaction information extracted from different channels, scales and branches to obtain high-level semantic information, including: The output features of the branches where the three hybrid domain shared modules are located are recorded as: ,in, Respectively represent the channel size; For the three output features Downsample separately to obtain unified features , and ; Use efficient units to perform a convolution on each of the three unified features with a kernel size of The 2D convolution layer operation obtains three efficient features and ,exist and The step size used is The 2D average pooling operation is used to reduce the spatial dimension and flatten the features after spatial dimension reduction into and : (11) in, Represents the fusion similarity matrix, and applies the fusion similarity matrix Reweighting And execute the merge function to restore the channel size to ; Output features of pyramid scale fusion block It is expressed as: (12) in, is a fusion operation, including the reshape operation and the convolution kernel size 2D convolutional layer operation.

Images