Feature extraction method based on multi-view interactive neural network

Abstract

The application discloses a feature extraction method based on a multi-view interactive neural network, which comprises the following steps: constructing a multi-scale convolutional neural network based on multiple modules, applying parallel multi-core to realize the extraction of multi-scale feature information, and designing cross-layer connection of multiple network modules to realize feature interaction between the multiple modules and complete information sharing; building a local and global feature interactive Transformer network, applying a lightweight residual block in the network to realize local correlation of global features, and applying a lightweight convolutional layer instead of a traditional convolutional block to reduce time-consuming cost; and fusing features extracted by the multi-module multi-scale convolutional neural network and the feature interactive Transformer network and applying the features to multispectral target recognition. In the network optimization process, global and local information is interactively interpreted, feature information is richer, and finally target recognition performance with relatively high precision is obtained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of network computing and representation learning, and particularly relates to a feature extraction method based on a multi-view interactive neural network, which can be used to solve the target recognition problem of multi-spectral remote sensing image data. BACKGROUND

[0002] As a kind of efficient feature representation method, the feature extraction method maps the original space data to a new feature space, and in this feature space, the features have better properties such as separability. Traditional methods use spatial-spectral feature prior knowledge to artificially construct information extraction models to represent the spectral, texture and geometric properties of images. However, in the face of high-dimensional nonlinear characteristics of data, these feature extraction methods set according to human experience are difficult to achieve the optimal balance between discriminability and robustness, and to realize the approximation of complex functions.

[0003] Deep learning, as a kind of layer-by-layer feature learning method with deep structure, has achieved better results than traditional feature extraction methods in the application based on feature extraction methods. Convolutional neural network (CNN) has the excellent characteristics of global parameter sharing and local connection, therefore, the feature extraction method based on convolutional neural network is one of the most popular deep learning methods in the field of image processing, and the remote sensing image classification based on convolutional neural network is one of the important applications of feature extraction network.

[0004] The principle of multi-spectral remote sensing image classification is to consider the spatial characteristics of image pixels on the basis of traditional conventional remote sensing image classification. After years of research by many scientists, multi-spectral remote sensing image classification technology has made great progress, and a series of ground object classification algorithms have been formed for the characteristics of multi-spectral remote sensing image. The interpretation and processing of multi-spectral remote sensing image largely depends on the feature extraction ability for the data itself.

[0005] With the development of deep neural networks, a series of improved convolutional neural network methods have been proposed and applied to the field of multi-spectral remote sensing image classification, among which the method of jointing spectral space is widely used. However, most networks use fixed convolution kernels to process images without considering interfering background categories and redundant information, and without enhancing the representation of target feature information. Therefore, some researchers have introduced the concept of multi-scale into multi-spectral remote sensing images to solve these problems. However, when the multi-scale deep learning method based on CNN extracts deep semantic features, the limited training set and the problem of gradient disappearance will lead to the loss of information. Therefore, the feature extraction method based on residual network is applied to the field of image processing. The residual network can achieve identity mapping by using residual learning to optimize part of the convolutional layer, and its strong generalization ability is used to cope with the overfitting phenomenon in the field of multi-spectral remote sensing image classification based on neural network. However, these models cannot model the larger range of spatial-spectral pixel relationships of multi-spectral remote sensing images due to the use of limited receptive fields. Therefore, the feature extraction method based on Transformer is proposed, which originates from the field of natural language processing and is then applied to the field of image processing and plays an excellent feature extraction ability. Compared with the local modeling ability of CNN, Transformer can dynamically capture the global feature data of the input image through the multi-head self-attention mechanism. Although these Transformer-based methods have obvious advantages in modeling global feature relationships, they lack the mining of local related features, resulting in limited ability to obtain local high-level semantic information. SUMMARY

[0006] To solve the above technical problems, the present application provides a feature extraction method based on a multi-view interactive neural network, which fuses the features extracted by a multi-module lightweight multi-scale convolutional neural network and a feature interactive Transformer network, realizes information sharing between local and global features, enriches deep semantic information, and further obtains better multi-spectral target recognition results.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0008] A feature extraction method based on a multi-view interactive neural network, comprising the following steps:

[0009] Step 1: setting the related parameters of the neural network, including the learning rate , the batch size is 16, the neural network includes a multi-module lightweight multi-scale convolutional neural network and a local and global feature interactive Transformer network;

[0010] Step 2: multi-spectral image data as input, the original multi-spectral image data The principal component analysis method is applied to reduce the dimension of features, and image data after dimension reduction is obtained The operation can extract a small number of spectral bands containing important information, and reduce the computational complexity of subsequent image processing

[0011] Step 3: For the image data after dimension reduction, a three-dimensional image block composed of each pixel point and its neighborhood pixel points is constructed as the input data of the network

[0012] Step 4: A certain amount of three-dimensional image blocks in the sample set are randomly selected as the training sample set of the network, and the remaining part is the test sample set

[0013] Step 5: For the image data in the training set, a multi-scale convolutional neural network based on multi-module lightweight is applied to extract multi-scale shared features: first, a plurality of branches are applied to extract multi-scale features, each scale network branch is composed of a spatial basic unit, a spectral basic unit and a jump connection, wherein the spatial basic unit and the spectral basic unit are used in each branch to replace the traditional 3D convolution block, then a single module is formed by a plurality of branch networks, a plurality of modules are formed by a multi-module lightweight convolutional neural network, and cross-layer transmission is applied between the plurality of modules, and then the feature interaction between different scales of a single module and between a plurality of modules is carried out

[0014] Step 6: For the image data in the training set, a local and global feature interactive Transformer network is applied to extract multi-view features, which is composed of a plurality of serial lightweight encoders and cross-layer transmission, first, a lightweight residual block is designed to replace the multi-layer perceptron, and the local spectral spatial correlation is introduced into the global feature, then a lightweight convolution layer is designed to replace the traditional convolution block, and finally the feature interaction between the encoders is carried out through cross-layer transmission

[0015] Step 7: The multi-scale features and multi-view features are integrated by designing a fusion mechanism to obtain interactive fusion features

[0016] Step 8: The interactive fusion features are input into an output module composed of a two-dimensional global average pooling, a fully connected layer and a softmax function for processing, and the target prediction result can be obtained, and the network parameters are optimized according to the predicted result and the real label data

[0017] Step 9: Check whether the end condition of optimal network training is met, if yes, end the training to obtain the optimal neural network, otherwise go to step 5

[0018] Step 10: After the network reaches the optimum, the test set data is input, and the output is the multispectral target recognition result

[0019] The beneficial effects of the present application are:

[0020] The application improves the interactive limitation of the multi-scale feature extraction method, applies multiple network modules cross-layer connection and multi-scale network branches for feature extraction, realizes feature interaction between different perspectives, completes information sharing, and improves the network interpretation information capability. Compared with the existing method, such a multi-module shared feature extraction network breaks the barrier of interaction between traditional multi-scale feature information, and realizes information sharing of features.

[0021] When the application adopts a network for feature extraction, on the one hand, a lightweight residual block is applied in the Transformer network, so that the local correlation of global features is considered, and the comprehensiveness of features is enriched, and on the other hand, the traditional convolution block is improved, that is, a lightweight residual layer is applied to reduce the time cost.

[0022] The application integrates multi-perspective and multi-scale interactive features through a fusion mechanism, completes information sharing between features, realizes extraction of more discriminative feature information through two-dimensional global average pooling, a fully connected layer and a softmax function, and applies it to multispectral target recognition to effectively improve the recognition accuracy, so that the final multispectral target recognition result is more reliable. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The application is a feature extraction method flowchart based on a multi-perspective interactive neural network.

[0024] Figure 2 The application is a multi-module lightweight multi-scale convolutional neural network construction process schematic diagram.

[0025] Figure 3 The application is a feature extraction module schematic diagram of the multi-module lightweight multi-scale convolutional neural network.

[0026] Figure 4 The application is a local and global feature interactive Transformer network construction process schematic diagram.

[0027] Figure 5 The application is a visual result comparison diagram of multispectral target recognition with the existing feature extraction method, (a) is a true value diagram method, (b) is an SVM method, (c) is an RF method, (d) is a 2DCNN method, (e) is a 3DCNN method, and (f) is the application method. DETAILED DESCRIPTION

[0028] The technical solutions of the application will be described in detail below in combination with the drawings and implementation cases.

[0029] The application discloses a feature extraction method of a multi-view interactive neural network.

[0030] The specific flowchart of the feature extraction method based on the multi-view interactive neural network is shown in the figure. Figure 1

[0031] Step 1, setting related parameters;

[0032] The related parameters of the input neural network include a multi-scale convolutional neural network and a local and global feature interactive Transformer network in parallel, the learning rate is 0.001, and the batch size is 16.

[0033] Step 2, principal component analysis dimension reduction;

[0034] Supposing that the size of an original image is 224*224, and the original image data matrix is X, wherein X is a matrix of 224*224, and n is the sample number. The application adopts principal component analysis to reduce the dimension of all samples, combines high-dimensional variables that may have correlation into linearly independent low-dimensional variables, that is, takes principal components, and then uses the results after dimension reduction for subsequent processing. Firstly, the mean of all samples is calculated, the samples are centrally processed, and then the covariance matrix is calculated.

[0035]

[0036] (1)

[0037] Wherein, X is the transpose matrix of the matrix X. Then, the eigenvalue decomposition is performed on the covariance matrix, the eigenvalues are sorted in descending order, the eigenvectors corresponding to the first k largest eigenvalues are taken to form a conversion matrix W, and finally the conversion matrix W is applied to the matrix X to obtain the result after dimension reduction. ​​​​​​​​​​Transforming to obtain a reduced dimension image dataset , is calculated as follows:

[0038] (2)

[0039] is the data reduced to dimension, .

[0040] Step 3, constructing a sample set;

[0041] Each pixel point and its neighborhood of the PCA reduced image form a three-dimensional image block, the process is as follows: the image data obtained after PCA reduction is , all pixel points are traversed to obtain all pixel three-dimensional image blocks , represents all pixel points in the neighborhood of the image with the pixel point as the center and the size of .

[0042] Step 4, selecting a training set;

[0043] After taking each pixel as the center and extracting the three-dimensional image block composed of the pixel and its neighborhood pixels as the input of the network, a part of the three-dimensional image blocks are randomly selected as the training sample matrix of the image , wherein is the number of selected training samples, and the rest is the test set, and the test sample matrix is .

[0044] Step 5, multi-module multi-scale feature extraction;

[0045] As shown in Figure 2 , the multi-module lightweight multi-scale convolutional neural network includes a plurality of serially connected feature extraction modules, specifically as follows:

[0046] First, as shown in Figure 3 , three parallel branches are applied to form a feature extraction module, and each branch is composed of a spatial basic unit, a spectral basic unit and a jump connection. The branch uses a spatial basic unit and a spectral basic unit to replace a traditional 3D convolution block, effectively reducing the training parameters. Different branches use spatial basic units based on different convolution kernel sizes to realize the extraction of small-scale spatial features , medium-scale spatial features and large-scale spatial features , and the integration is a global spatial feature Similarly, different branches employ spectral basis units based on different kernel sizes to realize small-scale spectral features , medium-scale spectral features , and large-scale spectral features , and the integration of the above is the global spectral feature . Additive operations are applied to aggregate different scale spectral features to obtain more discriminative and representative spectral spatial features . The calculation process of each branch is as follows:

[0047] (3)

[0048] (4)

[0049] (5)

[0050] (6)

[0051] (7)

[0052] (8)

[0053] wherein, is a spatial basis unit operation, is a spectral basis unit operation, is a cross-scale transmission, is an addition operation, is the final output result of the branch.

[0054] Then, the features extraction modules are implemented in series to mine deep information, in order to reduce the loss of expression ability of low-level information and effectively pass on the detailed features, a cross-layer transmission structure is designed to realize the information exchange between the modules, which reduces the loss of features required in the process of layer-by-layer propagation and realizes the exchange of information from shallow to deep. The cross-layer transmission is implemented as follows:

[0055] (9)

[0056] wherein, is the output of the multi-scale convolutional neural network, represents the output of the cross-layer transmission operation, is the number of feature extraction modules.

[0057] Step 6, interactive local and global feature extraction;

[0058] As Figure 4As shown, the local and global feature interactive Transformer network comprises a lightweight encoder and a cross-layer transmission structure in series, the lightweight encoder comprises a multi-head self-attention mechanism and a lightweight residual block in series, the lightweight residual block comprises two groups of lightweight convolutional layers, normalization layers and activation layers in series, and the lightweight residual block adopts a skip connection structure, specifically comprising:

[0059] The lightweight encoder mainly comprises a multi-head self-attention mechanism and a lightweight residual block, the traditional Transformer introduces a multi-layer perceptron after the multi-head self-attention mechanism, but this structure only focuses on the global features of data, lacks the mining ability of local features, therefore, the lightweight residual block is adopted to replace the traditional multi-layer perceptron to construct the correlation between the global features and the local features, and the time-consuming cost is reduced. The lightweight residual block is mainly composed of lightweight convolutional layers to reduce the calculation amount of feature extraction, and the skip connection structure is adopted in the lightweight residual block to reduce the information loss of shallow features. The cross-layer transmission structure is also introduced into the Transformer feature extraction network based on the interaction of local and global features to exchange feature information and enhance the information propagation from shallow to deep. The calculation process is as follows:

[0060] Firstly, the calculation of single-head attention is as follows:

[0061] (10)

[0062] (11)

[0063] (12)

[0064] Take the first attention head as an example, wherein, is the input data, the input data is multiplied by three weight matrices , and respectively to obtain three input vectors , , of the encoder, then is normalized, and a softmax function is used to obtain a self-attention score , is the operation of the softmax function, is the dimension of the vector, and the self-attention score is multiplied by the vector to obtain the output of the single-head attention . The multi-head attention mechanism is calculated based on the above, and the specific calculation is as follows:

[0065] (13)

[0066] in, Indicates the number of attention heads. Implement cascading operations to obtain the final output. .

[0067] Then, the output of the lightweight convolutional layer is calculated. The details are as follows:

[0068] (14)

[0069] (15)

[0070] (16)

[0071] in, For the input data, For the core size is 2D convolution kernels, The output of the 2D convolution kernel. The representative core size is 2D depthwise convolution kernels, This is the output of the depthwise convolution kernel.

[0072] The lightweight residual block consists of two lightweight convolutions, two batch Norm layers, and two ReLU activation layers. The output of the lightweight residual block... The calculation is as follows:

[0073] (17)

[0074] (18)

[0075] in, This involves integrating the output features of the first activation layer with the input data. This indicates a lightweight convolutional layer operation. Indicates activation layer operation, This indicates Batch Norm layer operations.

[0076] Finally, a lightweight encoder is constructed by applying a multi-head self-attention mechanism and lightweight residual blocks, as shown in the following formula:

[0077] (19)

[0078] (20)

[0079] in, is a normalized layer operation, is a discarded layer operation, is a reshape operation, is the integration of the intermediate layer output features after applying the multi-head attention mechanism and the input data is the integration of the intermediate layer output features after applying the multi-head attention mechanism and the input data is the output of the lightweight encoder.

[0080] Step 7, integrate the extracted features;

[0081] Integrate the local and global features extracted by the multi-module convolutional neural network and the feature interactive Transformer network through the fusion mechanism, specifically including:

[0082] Integrate the multi-scale features extracted by the multi-module lightweight multi-scale convolutional neural network and the multi-view features extracted by the local and global feature interactive Transformer network, first unify the feature size, that is, realize feature alignment through 2D convolution layer, Batch Normal layer, and activation layer in turn, then perform feature concatenation to obtain more discriminative multi-view multi-scale interactive fusion features to realize information sharing between different features.

[0083] Step 8, output the result;

[0084] Input the fused multi-view multi-scale features into the output module for processing to obtain the target prediction result, and adjust the network parameters according to the result and the true label;

[0085] Step 9, check whether the end condition is met, if yes, end, otherwise return to step 5 to continue extracting features based on the updated network parameters;

[0086] Step 10, after the network reaches the optimal, the output is the multispectral target recognition result. Apply the output module to predict the target recognition result, specifically including:

[0087] The output module is composed of an average pooling layer, a fully connected layer, and a Softmax layer. The interactive features fused in step 6 After passing through the output module, the prediction class result of each sample is obtained. The calculation formula is as follows:

[0088] (21)

[0089] wherein, represents the average pooling layer operation, is a fully connected layer, is the output result information.

[0090] In order to verify the performance of the algorithm, experiments are carried out under the NVIDIA RTX 2080Ti GPU system with a memory of 16.0 GB and a Tensorflow2.1.0 programming platform. The Salinas Valley dataset is used in the experiment, which has a size of , 204 spectral bands and 16 crop categories.

[0091] Figure 5 is a recognition result graph of the present application and other methods on the dataset, Figure 5 and the illustration of different target categories on the right, wherein (a) is a true value graph, (b)-(e) are recognition result graphs of the application of the SVM, RF, 2DCNN and 3DCNN methods respectively, and (f) is a result graph of the present application. The experimental results show that the target recognition effect of the present application is optimal, the classification smoothness of different categories is best, and there are fewer noise points. Through comparison, compared with the traditional method and other feature extraction networks, the present application can effectively reduce the probability of recognition error when the data volume of the dataset is large in the categories Grapes_untrained and Vinyard_untrained, and can also show strong feature representation ability when identifying the data volume of the Fallow category, and has a better recognition effect. The method of the present application introduces the idea of interactive features into the network feature extraction method, designs a feature extraction method based on local and global shared collaborative feature extraction, which can effectively improve the discriminability and robustness of the network, and finally obtain better image recognition performance.

[0092] In summary, the present application provides a deep learning efficient feature representation learning model, which effectively combines a multi-module lightweight multi-scale convolutional neural network and a local and global feature interactive Transformer network into a whole, cooperates with a fusion mechanism, continuously optimizes, finally makes the framework optimal, and finally outputs the target recognition result, and realizes efficient interpretation of multispectral remote sensing image data.

[0093] The above specific embodiments further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A feature extraction method based on a multi-view interactive neural network, characterized in that, Includes the following steps: Step 1: Set the relevant parameters of the neural network, including the learning rate and batch size. The neural network includes a parallel multi-module lightweight multi-scale convolutional neural network and an interactive Transformer network based on local and global features. Step 2: Principal component analysis is used to reduce the dimensionality of the original multi-spectral image data to obtain the dimensionality-reduced image data; Step 3: For the dimension-reduced image data, extract the three-dimensional image block composed of each pixel and its neighboring pixels as a sample set, with each pixel as the center. Step 4: Randomly select a portion of the 3D image patches from the sample set as the training sample set, and the remaining portion as the test sample set; Step 5: For the image data in the training set, extract multi-scale features based on a multi-module lightweight multi-scale convolutional neural network. The multi-module lightweight multi-scale convolutional neural network includes multiple cascaded feature extraction modules, wherein... The feature extraction module includes three parallel branches, each of which includes a spatial basic unit and a spectral basic unit with skip connections. The three parallel branches use spatial basic units and spectral basic units based on different convolution kernel sizes to extract small-scale spectral spatial features, medium-scale branch spectral features and large-scale branch spectral spatial features, and apply addition operations to aggregate spectral features at different scales. The multiple cascaded feature extraction modules adopt a cross-layer transmission structure to facilitate information exchange between the various feature extraction modules. Step 6: For the image data in the training set, extract multi-view features based on an interactive Transformer network of local and global features; the interactive Transformer network of local and global features includes multiple cascaded lightweight encoders, wherein... The lightweight encoder includes a series of multi-head self-attention mechanisms and lightweight residual blocks. The lightweight residual blocks include two sets of lightweight convolutional layers, normalization layers, and activation layers connected in series. A skip connection structure is used in the lightweight residual blocks. Step 7: Integrate multi-scale and multi-view features based on the fusion mechanism to obtain interactive fused features; Step 8: Input the interactive fusion features into the output module for processing to obtain the target prediction result. Based on the target prediction result and the real label, continuously iterate the training to optimize the network structure parameters. Step 9: Check if the optimal network termination condition is met. If it is met, end the training and obtain the optimal neural network; otherwise, proceed to step 5. Step 10: After the neural network reaches its optimal state, the 3D image block to be tested is input into the optimal neural network, and the multi-spectral target recognition result is output.

2. The feature extraction method based on a multi-view interactive neural network according to claim 1, characterized in that, In step 4, the original multi-spectral image is assumed to be... The dimensionality-reduced image data is Record three-dimensional image blocks ,in Represents the image data after dimensionality reduction In the middle of the pixel Centered on, the neighborhood size is After iterating through all pixels, a small number of 3D image patches are randomly selected to obtain the training sample set matrix. ,in The number of training samples selected is [number], and the test sample matrix is ​​[matrix]. .

3. The feature extraction method based on a multi-view interactive neural network according to claim 1, characterized in that, In step 7, the process of integrating multi-scale features and multi-view features based on the fusion mechanism to obtain interactive fused features includes: Multi-scale and multi-view features are unified in size by using 2D convolutional layers, batch normal layers, and activation layers to align features, and then feature concatenation is performed to obtain multi-view, multi-scale interactive features.

4. The feature extraction method based on a multi-view interactive neural network according to claim 1, characterized in that, In step 8, the step of inputting the interactive fusion features into the output module for processing to obtain the target prediction result includes: The output module includes an average pooling layer, a fully connected layer, and a softmax layer connected in series. After interactively fusing features are input to the output module, the prediction and recognition results for each sample are obtained.

Images