Feature extraction method based on multi-view interactive neural network

Abstract

The invention discloses a feature extraction method based on a multi-view interactive neural network, and the method comprises the steps: constructing a multi-scale convolutional neural network based on multiple modules, achieving the extraction of multi-scale feature information through a parallel multi-core, and designing the cross-layer connection of a plurality of network modules to achieve the feature interaction among the multiple modules. Information sharing is completed; constructing an interactive Transform network based on local and global features, applying a lightweight residual block in the network to realize local association of the global features, and applying a lightweight convolutional layer to replace a traditional convolutional block to reduce time consumption cost; features extracted by a multi-module multi-scale convolutional neural network and a feature interactive Transform network are fused, and the method is applied to multi-spectral target recognition. According to the method, global and local information is interactively interpreted in the network optimization process, so that feature information is richer, and finally, target recognition performance with relatively high precision is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of network computing and representation learning, and specifically relates to a feature extraction method based on a multi-view interactive neural network, which can be used to solve the problem of target recognition of multi-spectral remote sensing image data. Background Art

[0002] As an efficient feature representation method, feature extraction methods map the original spatial data to a new feature space, where the features have better properties such as separability. Traditional methods use prior knowledge of spatial spectral features to artificially construct information extraction models to represent the spectrum, texture, and geometric properties of images. However, in the face of high-dimensional nonlinear data, these feature extraction methods set based on artificial experience are difficult to achieve an optimal balance between discriminability and robustness and to approximate complex functions.

[0003] As a layer-by-layer feature learning method with a deep structure, deep learning has achieved better results than traditional feature extraction methods in applications based on feature extraction methods. Convolutional Neural Network (CNN) has the excellent characteristics of global parameter sharing and local connection. Therefore, the feature extraction method based on convolutional neural network is one of the most popular deep learning methods in the field of image processing. Among them, remote sensing image classification based on convolutional neural network is one of the important applications of feature extraction network.

[0004] The principle of multi-spectral remote sensing image classification is to consider the spatial characteristics of image pixels on the basis of traditional conventional remote sensing image classification. After years of research by many scientific and technological workers, multi-spectral remote sensing image classification technology has made great progress, forming a series of ground object classification algorithms targeting the characteristics of multi-spectral remote sensing images. The interpretation and processing tasks of multi-spectral remote sensing images largely rely on the feature extraction capabilities of the data itself.

[0005] With the development of deep neural networks, a series of improved convolutional neural network methods have been proposed and used in the field of multi-spectral remote sensing image classification, among which the spectral-spatial joint method is widely used. However, most networks use fixed convolution kernels to process images without considering the interfering background categories and redundant information, and do not enhance the representation of target feature information. Therefore, some researchers introduce the concept of multi-scale into multi-spectral remote sensing images to solve these problems. However, when extracting deeper semantic features based on CNN multi-scale deep learning methods, information loss will occur due to the limited training set and gradient vanishing problem. Therefore, feature extraction methods based on residual networks are applied to the field of image processing. Residual networks can achieve identity mapping by optimizing some convolutional layers using residual learning. Their powerful generalization ability is used to deal with the overfitting phenomenon in the field of multi-spectral remote sensing image classification based on neural networks. However, these models cannot model the larger range of spatial-spectral pixels of multi-spectral remote sensing images because they use limited receptive fields. Therefore, feature extraction methods based on Transformer are proposed, which originated from the field of natural language processing and were later applied to the image field, and exerted excellent feature extraction capabilities. Compared with the local modeling capability of CNN, Transformer can dynamically capture the global feature data of the input image through a multi-head self-attention mechanism. Although these Transformer-based methods have obvious advantages in modeling global feature relationships, they lack the mining of local related features, resulting in limited ability to obtain local high-level semantic information. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a feature extraction method based on a multi-view interactive neural network. The method integrates the features extracted by a multi-module lightweight multi-scale convolutional neural network and a feature interactive Transformer network, realizes information sharing between local and global features, enriches deep semantic information, and thereby obtains better multi-spectral target recognition results.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A feature extraction method based on a multi-view interactive neural network comprises the following steps:

[0009] Step 1: Set the relevant parameters of the neural network, including the learning rate , the batch size is 16, and the neural network includes a multi-scale convolutional neural network based on multi-module lightweight and a Transformer network based on local and global features interactive;

[0010] Step 2: Multispectral image data As input, the original multispectral image data Apply principal component analysis to reduce the dimension of features and obtain the image data after dimension reduction. ,This operation can extract a small number of spectral segments containing important information and reduce the computational complexity of subsequent image processing;

[0011] Step 3: For the image data after dimensionality reduction, construct a three-dimensional image block based on each pixel and its neighboring pixels as the input data of the network;

[0012] Step 4: Randomly select a certain amount of 3D image blocks from the sample set as the training sample set of the network, and the rest as the test sample set;

[0013] Step 5: For the image data in the training set, a multi-scale convolutional neural network based on multi-module lightweight is applied to extract multi-scale shared features: first, multiple branches are applied to extract multi-scale features. Each scale network branch consists of a spatial basic unit, a spectral basic unit and a jump connection. In each branch, the spatial basic unit and the spectral basic unit are used to replace the traditional 3D convolution block. Then, a single module is formed by a multi-branch network. The multi-module lightweight convolutional neural network constructs multiple modules. Cross-layer transmission is applied between multiple modules to perform feature interaction between different scales of a single module and between multiple modules.

[0014] Step 6: For the image data in the training set, an interactive Transformer network based on local and global features is applied to extract multi-view features. The network consists of multiple lightweight encoders connected in series and cross-layer transmission. First, a lightweight residual block is designed to replace the multi-layer perceptron, and the local spectrum spatial correlation is introduced into the global feature. Then, a lightweight convolutional layer is designed to replace the traditional convolutional block. Finally, feature interaction between encoders is performed through cross-layer transmission.

[0015] Step 7: Integrate multi-scale features and multi-view features by designing a fusion mechanism to obtain interactive fusion features;

[0016] Step 8: Input the interactive fusion features into the output module composed of two-dimensional global average pooling, fully connected layer and softmax function for processing, and obtain the target prediction result. The network parameters are optimized according to the predicted result and the real label data;

[0017] Step 9: Check whether the end condition of the optimal network training is met. If so, end the training to obtain the optimal neural network, otherwise proceed to step 5;

[0018] Step 10: After the network reaches the optimal state, the test set data is input and the output is the multi-spectral target recognition result.

[0019] The beneficial effects of the present invention are:

[0020] In order to improve the interactive limitations of the multi-scale feature extraction method, the present invention applies multiple network modules to cross-layer connections and multi-scale network branches to extract features, realize feature interaction between different perspectives, complete information sharing, and improve the network's ability to interpret information. Compared with existing methods, such a multi-module shared feature extraction network breaks the barriers between traditional multi-scale feature information and realizes feature information sharing.

[0021] When the present invention adopts the network for feature extraction, on the one hand, by applying lightweight residual blocks in the Transformer network, the local correlation of global features is taken into account to enrich the comprehensiveness of features; on the other hand, the traditional convolutional block is improved, that is, a lightweight residual layer is applied to reduce time-consuming costs.

[0022] The present invention integrates multi-view and multi-scale interactive features through a fusion mechanism, completes information sharing between features, and realizes the extraction of more discriminative feature information through two-dimensional global average pooling, fully connected layers and softmax functions. Applying it to multi-spectral target recognition can effectively improve the recognition accuracy and make the final multi-spectral target recognition result more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the feature extraction method based on multi-view interactive neural network of the present invention;

[0024] Figure 2 A schematic diagram of the construction process of a multi-module lightweight multi-scale convolutional neural network of the present invention;

[0025] Figure 3 It is a schematic diagram of a feature extraction module of a multi-module lightweight multi-scale convolutional neural network of the present invention;

[0026] Figure 4 A schematic diagram of the construction process of the interactive Transformer network based on local and global features of the present invention;

[0027] Figure 5 Comparison diagram of visual results of multi-spectral target recognition performed by the present invention and the existing feature extraction method, (a) is the truth map method, (b) is the SVM method, (c) is the RF method, (d) is the 2DCNN method, (e) is the 3DCNN method, and (f) is the method of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and implementation examples.

[0029] The present invention discloses a feature extraction method of a multi-perspective interactive neural network. In view of the shortcomings of the traditional feature extraction framework, such as insufficient utilization of extracted feature information, high time-consuming cost and imperfect information sharing, a neural network structure based on an interactive CNN and Transformer network is constructed. This algorithm extracts multi-perspective features through a multi-scale convolutional neural network of multiple modules and an interactive Transformer network based on local and global features, thereby increasing the richness of the extracted deep feature information. At the same time, cross-layer connections are made through multiple modules in each network to achieve feature interaction between multiple modules, thereby increasing information sharing capabilities. Compared with existing methods, the present invention can promote the feature extraction capability of the network through information sharing, can extract richer features, and extract features with more discriminative capabilities, and therefore has higher target recognition accuracy.

[0030] The specific flow chart of the feature extraction method based on multi-view interactive neural network is as follows: Figure 1 shown.

[0031] Step 1. Set relevant parameters;

[0032] Input the relevant parameters of the neural network, which includes a parallel multi-scale convolutional neural network and an interactive Transformer network based on local and global features, a learning rate , the batch size is 16.

[0033] Step 2: Dimensionality reduction by principal component analysis;

[0034] Assume that the size of the original image is , the original image data matrix is ,in For the number of samples, the present invention uses principal component analysis to reduce the dimensionality of all samples, synthesizes high-dimensional variables that may have correlation into linearly independent low-dimensional variables, that is, takes the principal components, and then uses the results after dimensionality reduction for subsequent processing.

[0035] First, calculate the mean of all samples, center the samples, and then calculate the covariance matrix :

[0036] (1)

[0037] in, For the matrix Then perform eigenvalue decomposition on the covariance matrix and sort the eigenvalues ​​in descending order, taking the largest one. The eigenvectors corresponding to the eigenvalues ​​form a transformation matrix , and finally apply the transformation matrix Pair Matrix Transform to obtain the image dataset after dimensionality reduction , calculated as follows:

[0038] (2)

[0039] That is, dimensionality reduction to After dimensioning, .

[0040] Step 3: Construct a sample set;

[0041] Each pixel point and its neighborhood of the image after PCA dimensionality reduction are traversed to form a three-dimensional image block. The process is as follows: The image data obtained after PCA dimensionality reduction is , traverse all pixels and get the three-dimensional image blocks of all pixels , Representing images Pixels The size of the center is All pixels in the neighborhood of .

[0042] Step 4: Select the training set;

[0043] Taking each pixel as the center, extract the 3D image block composed of the pixel and its neighboring pixels as the input of the network, and randomly select a part of the 3D image blocks as the image The training sample matrix ,in is the number of selected training samples, the rest is the test set, and the test sample matrix is .

[0044] Step 5: Multi-scale feature extraction of multiple modules;

[0045] like Figure 2 As shown, the multi-module lightweight multi-scale convolutional neural network includes multiple serially connected feature extraction modules, which are as follows:

[0046] First, if Figure 3 As shown in the figure, three parallel branches are used to form a feature extraction module. Each branch consists of a spatial basic unit, a spectral basic unit and a jump connection. The branches use spatial basic units and spectral basic units to replace the traditional 3D convolution blocks, effectively reducing the training parameters. Different branches use spatial basic units based on different convolution kernel sizes to realize small-scale spatial features. , mesoscale spatial characteristics and large-scale spatial features Extracted and integrated into global spatial features Similarly, different branches use spectrum basic units based on different convolution kernel sizes to realize small-scale spectral features. , mesoscale spectral characteristics and large-scale spectral features Extraction, after integration, is the global spectral feature Applying addition operations to aggregate spatial spectral features at different scales to obtain more discriminative and representative spectral spatial features The calculation process of each branch is as follows:

[0047] (3)

[0048] (4)

[0049] (5)

[0050] (6)

[0051] (7)

[0052] (8)

[0053] in, is the basic unit operation of space. For spectrum basic unit operation, For cross-scale transmission, For the addition operation, The final output result of the branch.

[0054] Then, the feature extraction modules are connected in series to realize the mining of deep information. In order to reduce the loss of the expression ability of low-level information and effectively transmit detailed features, a cross-layer transmission structure is designed to realize the information exchange between modules. This structure reduces the loss of required features in the layer-by-layer propagation process and realizes information exchange from shallow to deep layers. The cross-layer transmission is implemented as follows:

[0055] (9)

[0056] in, is the output of the multi-scale convolutional neural network, Represented as the output of the cross-layer transfer operation, is the number of feature extraction modules.

[0057] Step 6: Interactive local and global feature extraction;

[0058] like Figure 4As shown, the local and global feature interactive Transformer network includes a series of lightweight encoding blocks and a cross-layer transmission structure, the lightweight encoder includes a series of multi-head self-attention mechanism and a lightweight residual block, the lightweight residual block includes two groups of lightweight convolutional layers, a standard layer, and an activation layer connected in series in sequence, and a jump connection structure is adopted in the lightweight residual block, which specifically includes:

[0059] The lightweight encoder mainly includes a multi-head self-attention mechanism and a lightweight residual block. The traditional Transformer introduces a multi-layer perceptron after the multi-head self-attention mechanism, but this structure only focuses on the global features of the data and lacks the ability to mine local features. Therefore, a lightweight residual block is used instead of the traditional multi-layer perceptron to build the association between global features and local features to reduce time-consuming costs. The lightweight residual block is mainly composed of lightweight convolutional layers to reduce the amount of computational cost of feature extraction. At the same time, a jump connection structure is used in the lightweight residual block to reduce the information loss of shallow features. The present invention also introduces a cross-layer transmission structure into the Transformer feature extraction network based on the interaction of local and global features to exchange feature information and enhance the information propagation of features from shallow to deep. The calculation process is as follows:

[0060] First, the calculation of single-head attention is as follows:

[0061] (10)

[0062] (11)

[0063] (12)

[0064] First Take an attention head as an example, where For input data, input data Multiply the three weight matrices separately , and , get the three input vectors of the encoder , , , then Normalize and use the softmax function to obtain the self-attention score , is the softmax function operation, for Dimensions of the vector, self-attention score Multiply by the vector Get the output of single-head attention The multi-head attention mechanism is calculated based on the above, as follows:

[0065] (13)

[0066] in, represents the number of attention heads, Implement cascade operation and get the final output through cascade .

[0067] Then, the output of the lightweight convolutional layer is calculated , as follows:

[0068] (14)

[0069] (15)

[0070] (16)

[0071] in, For the input data, The kernel size is 2D convolution kernel, is the output of the 2D convolution kernel, The representative kernel size is 2D depth convolution kernel, is the output of the depthwise convolution kernel.

[0072] The lightweight residual block consists of two lightweight convolutions, two Batch Norm layers, and two ReLU activation layers. The output of the lightweight residual block is The calculation is as follows:

[0073] (17)

[0074] (18)

[0075] in, The integration of the output features and input data of the first activation layer, represents a lightweight convolutional layer operation, represents the activation layer operation, Represents the Batch Norm layer operation.

[0076] Finally, a lightweight encoder is constructed by applying a multi-head self-attention mechanism and a lightweight residual block, as follows:

[0077] (19)

[0078] (20)

[0079] in, is the normalization layer operation, is the dropout layer operation, For the reshape operation, Output features and input data for the intermediate layer after applying the multi-head attention mechanism Integration is the output of the lightweight encoder.

[0080] Step 7: Integrate the extracted features;

[0081] The local and global features extracted by the multi-module convolutional neural network and the feature interactive Transformer network are integrated through a fusion mechanism, including:

[0082] Integrate the multi-scale features extracted by the multi-scale convolutional neural network feature extraction method based on multi-module lightweight and the multi-view features extracted by the interactive Transformer network based on local and global features. First, unify the feature size, that is, align the features by sequentially connecting 2D convolutional layers, Batch Normal layers, and activation layers. Then perform feature cascading to obtain more discriminative multi-view and multi-scale interactive fusion features. , to achieve information sharing between different features.

[0083] Step 8: Output the result.

[0084] The fused multi-view and multi-scale features are input into the output module for processing to obtain the target prediction results, and the network parameters are adjusted according to the results and the true labels;

[0085] Step 9: Check whether the end condition is met. If so, the process ends. Otherwise, the process returns to step 5 based on the updated network parameters to continue extracting features.

[0086] Step 10: After the network reaches the optimal state, its output is the multi-spectral target recognition result. The output module is used to predict the target recognition result, which specifically includes:

[0087] The output module consists of an average pooling layer, a fully connected layer, and a Softmax layer. The interactive features fused in step 6 are After passing through the output module, the predicted category result of each sample is obtained. The calculation formula is as follows:

[0088] (twenty one)

[0089] in, represents the average pooling layer operation, is the fully connected layer, The output result information.

[0090] In order to verify the performance of the algorithm, the experiment was conducted on the NVIDIA RTX 2080Ti GPU system with 16.0 GB memory and the Tensorflow 2.1.0 programming platform. The experiment uses the Salinas Valley dataset, which is , with 204 spectral bands and 16 crop categories.

[0091] Figure 5 is the recognition result diagram of the present invention and other methods on the data set, Figure 5 The right side is a diagram of different target categories, where (a) is the truth map, (b)-(e) are the recognition result maps of the SVM, RF, 2DCNN and 3DCNN methods, and (f) is the result map of the present invention. The experimental results show that the recognition effect of the target using the method of the present invention is the best, the classification smoothness of different categories is the best, and there are fewer noise points. By comparison, it is found that compared with traditional methods and other feature extraction networks, the present invention can effectively reduce the probability of recognition errors when the categories Grapes_untrained and Vinyard_untrained with a large amount of data in the data set are compared, and it can also show strong feature representation capabilities when identifying the category Fallow with a small amount of data, and has a better recognition effect. The method of the present invention introduces the idea of ​​interactive features into the network feature extraction method, and designs a feature extraction method based on local and global shared collaboration, which can effectively improve the network's discriminability and robustness, and ultimately obtain better image recognition performance.

[0092] In summary, the present invention provides an efficient feature representation learning model through deep learning, which effectively combines a multi-module lightweight multi-scale convolutional neural network with an interactive Transformer network based on local and global features into a whole, cooperates with a fusion mechanism, and continuously optimizes it, so that the framework is finally optimized and the target recognition result is finally output, thereby realizing efficient interpretation of multi-spectral remote sensing image data.

[0093] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A feature extraction method based on a multi-view interactive neural network, characterized in that: The steps include: Step 1: Setting the relevant parameters of the neural network, including the learning rate and the batch size. The neural network includes a parallel multi-scale convolutional neural network based on multi-module lightweight and a local and global feature interactive Transformer network. Step 2: Use principal component analysis to reduce the dimension of the original multi-spectral image data to obtain the image data after dimension reduction; Step 3: For the image data after dimensionality reduction, taking each pixel point as the center, extracting a three-dimensional image block consisting of the pixel point and its neighboring pixel points as a sample set; Step 4: Randomly select a part of the 3D image blocks in the sample set as the training sample set, and the rest as the test sample set; Step 5: For the image data in the training set, extract multi-scale features based on a multi-module lightweight multi-scale convolutional neural network; Step 6: For the image data in the training set, extract multi-view features based on the interactive Transformer network of local and global features; Step 7: Integrate multi-scale features and multi-view features based on the fusion mechanism to obtain interactive fusion features; Step 8: Input the interactive fusion features into the output module for processing to obtain the target prediction results. According to the target prediction results and the true labels, the network structure parameters are optimized through continuous iterative training. Step 9: Check whether the optimal network end condition is met. If so, end the training to obtain the optimal neural network. Otherwise, go to step 5. Step 10: After the neural network reaches the optimal state, the three-dimensional image block to be tested is input into the optimal neural network, and the multi-spectral target recognition result is output.

2. The feature extraction method based on multi-view interactive neural network according to claim 1 is characterized in that: In step 4, the original multi-spectral image is assumed to be The image data after dimensionality reduction is , remember the three-dimensional image block ,in Represents image data after dimensionality reduction Pixels The center is and the neighborhood size is After traversing all the pixels, a small number of three-dimensional image blocks are randomly selected to obtain the training sample set matrix ,in is the number of selected training samples, and the test sample matrix is .

3. The feature extraction method based on multi-view interactive neural network according to claim 1 is characterized in that: In step 5, the multi-module lightweight multi-scale convolutional neural network includes a plurality of feature extraction modules connected in series, wherein: The feature extraction module includes three parallel branches, each branch includes a spatial basic unit and a spectrum basic unit with jump connections. The three parallel branches use spatial basic units and spectrum basic units based on different convolution kernel sizes to extract small-scale spectral spatial features, medium-scale branch spectral features and large-scale branch spectral spatial features, and apply addition operations to aggregate spectral features of different scales. The multiple serially connected feature extraction modules use a cross-layer transmission structure to exchange information between the feature extraction modules.

4. The feature extraction method based on multi-view interactive neural network according to claim 1 is characterized in that: In step 6, the local and global feature interactive Transformer network includes a plurality of lightweight encoders connected in series, wherein: The lightweight encoder includes a multi-head self-attention mechanism and a lightweight residual block connected in series. The lightweight residual block includes two groups of lightweight convolutional layers, a standard layer, and an activation layer connected in series in sequence. A jump connection structure is adopted in the lightweight residual block.

5. The feature extraction method based on multi-view interactive neural network according to claim 1 is characterized in that: In step 7, the step of integrating multi-scale features and multi-view features based on a fusion mechanism to obtain interactive fusion features includes: The feature sizes of multi-scale features and multi-view features are unified, that is, feature alignment is achieved through 2D convolution layer, Batch Normal layer, and activation layer, and then feature cascading is performed to obtain multi-view and multi-scale interactive features.

6. The feature extraction method based on multi-view interactive neural network according to claim 1, characterized in that: In step 8, the interactive fusion feature is input to the output module for processing to obtain the target prediction result, which includes: The output module includes an average pooling layer, a fully connected layer and a Softmax layer connected in series in sequence. After the interactive fusion features are input into the output module, the prediction recognition result of each sample is obtained.

Images