A remote sensing image fusion classification method based on multi-modal feature alignment

Abstract

The application discloses a remote sensing image fusion classification method based on multi-modal feature alignment, fully utilizes the convolutional neural network to extract features from different modes, and uses a Transformer network as a feature fusion framework; the traditional cross-attention method is improved, so that it is more suitable for the fusion classification task of hyperspectral images and LiDAR data; at the same time, in order to solve the problem that different modal features are not aligned when input into the Transformer, so that the multi-modal encoder is difficult to learn and model the interaction thereof, a method for aligning the multi-modal feature module is provided, so that the subsequent multi-modal Transformer is easier to perform cross-modal learning; in view of the problem that the utilization efficiency of multi-modal samples with different labels in current network training is not high, a feature matching module based on the most difficult negative sample contrast learning method is provided; and the precision of fusion classification is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of multi-source remote sensing image processing, and in particular to a remote sensing image fusion and classification method based on multi-modal feature alignment that can effectively explore the feature interactions between different modalities, achieve good fusion quality, and high classification accuracy. Background Technology

[0002] In recent years, driven by technological advancements, remote sensing technology has entered a new stage of development. The widespread adoption of Earth observation platforms and remote sensing sensors has simplified the acquisition of remote sensing data with different characteristics within the same geographical area. Among numerous sensor types, hyperspectral imagery stands out for its continuous and extensive spectral detail, superior surface cover, and outstanding target detection capabilities. On the other hand, LiDAR (Light Detection and Ranging) data captures elevation information, enabling the differentiation of land features with similar spectral responses but different heights. Integrating hyperspectral imagery and LiDAR data and fully utilizing this information to collaboratively complete land feature analysis is beneficial for combining the advantages of both types of sensors, effectively improving the performance of intelligent processing algorithms, and thus providing a more comprehensive analysis of land feature information. Currently, the fusion of hyperspectral imagery and LiDAR data has become a prominent research focus in the field of land use / land cover classification, with wide applications across various sectors.

[0003] In early research phases, the fusion of hyperspectral imagery and LiDAR data typically followed a strategy of feature extraction, feature transformation, and weight-based fusion. Ghamisi et al. performed morphological operations on the first principal component analysis (PCA) of hyperspectral imagery and the digital representation model (DSM) of LiDAR data to obtain the spatial features of the two images. Liao et al. extracted features from hyperspectral imagery and LiDAR data by constructing a graphical model to obtain fused features. Rasti et al. incorporated constraints into feature fusion for optimization, using a sparse low-rank method to reduce the dimensionality of the extracted features before classifying the resulting features. However, while these techniques help reduce data dimensionality to some extent, they lack generalization ability.

[0004] In recent years, the computing power and data acquisition capabilities of computing devices have rapidly improved. The significant increase in computing power can alleviate the inefficiency of training, while the substantial increase in training data can reduce the risk of overfitting. Salman et al. combined hyperspectral imagery with LiDAR data, using the initial convolutional layer filters of AlexNet to extract elevation details from the LiDAR data. Xu et al. proposed a two-branch convolutional neural network (TBCNN) that combines spectral and spatial branches. Roy et al. proposed a parallel feature learning and fusion method, utilizing convolutional neural networks and morphological blocks to extract LiDAR features. Although the above techniques have achieved good results, the current main approach uses single-level fusion networks. These networks use simple mechanisms, such as weight sharing and feature stacking, to combine different features at different network locations. However, this primitive fusion method mainly focuses on feature stacking and lacks true feature fusion. In most tasks involving the fusion and classification of hyperspectral imagery and LiDAR data, the focus remains on feature extraction.

[0005] The Transformer framework, as a novel model, effectively manipulates data using self-attention modules, significantly promoting the fusion of multimodal data. It has been applied to the field of multi-source remote sensing fusion and has yielded a series of results. However, current methods using the Transformer framework merely extract simple features and feed them to the Transformer model for fusion. This overly simplistic feature extraction mechanism may reduce the quality of the extracted features, thus affecting the fusion results.

[0006] Overall, although hyperspectral imagery and LiDAR data fusion classification techniques have made some progress over the past two decades, most existing methods suffer from limitations such as incomplete feature extraction, insufficient feature fusion, high algorithm redundancy, and computational complexity. Furthermore, while existing multimodal Transformers jointly model multimodal tokens, the lack of alignment between different modal features when input to the Transformer, with each modality's embedded features residing in its own space, makes it difficult for the multimodal encoder to learn and model their interactions. This hinders the multimodal Transformer from accurately learning the relationships between modalities. Additionally, current network training focuses only on paired samples to learn their relationships, failing to effectively utilize multimodal samples with different labels. Summary of the Invention

[0007] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing a remote sensing image fusion and classification method based on multimodal feature alignment that can effectively explore the feature interactivity between different modalities, achieve good fusion quality, and high classification accuracy.

[0008] The technical solution of this invention is: a remote sensing image fusion and classification method based on multimodal feature alignment, which is carried out according to the following steps:

[0009] Step 1. Establish and initialize the Transformer network N for multi-sensor remote sensing image fusion and classification. MF2AMT The N MF2AMT It contains 3 sub-networks N for feature extraction CH N CL and N TE One subnetwork N for feature fusion TF and one subnetwork N for classification cls ;

[0010] Step 1.1 Establish and initialize subnetwork N CH The subnetwork N CH It consists of 6 sets of convolutional layers, namely Conv1_0, Conv1_1, Conv1_2, Conv1_3, Conv1_4, and Conv1_5;

[0011] The Conv1_0 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0012] The Conv1_1 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0013] The Conv1_2 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0014] The Conv1_3 includes one convolutional layer, one BatchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0015] The Conv1_4 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0016] The Conv1_5 layer includes one convolution operation, one BatchNorm normalization operation, and one activation operation. The size of the convolution kernel in the convolution layer is 1×1, and each convolution kernel performs convolution operation with a stride of 1 pixel. The non-linear activation function GELU is selected as the activation function for the operation.

[0017] Step 1.2 Establish and initialize subnetwork N CL The subnetwork N CL It consists of 6 sets of convolutional layers, namely Conv2_0, Conv2_1, Conv2_2, Conv2_3, Conv2_4, and Conv2_5;

[0018] The Conv2_0 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0019] The Conv2_1 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0020] The Conv2_2 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0021] The Conv2_3 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0022] The Conv2_4 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0023] The Conv2_5 layer includes one convolution operation, one BatchNorm normalization operation, and one activation operation. The size of the convolution kernel in the convolution layer is 1×1, and each convolution kernel performs convolution operation with a stride of 1 pixel. The non-linear activation function GELU is selected as the activation function for the operation.

[0024] Step 1.3. Establish and initialize subnetwork N TE The subnetwork N TE It consists of a multi-head self-attention layer (MHSA) and a forward propagation layer (FFN);

[0025] The MHSA performs the following 5 steps:

[0026] (a) Perform cubic linear operations on the input three-dimensional tensor F to calculate the features respectively. in, and F Q F K and F V The i1th element is B, which represents the batch size of the input tensor, N, which represents the number of features in each tensor, and C, which represents the dimension of the input tensor.

[0027] (b) Using the reshape operation to reshape F Q F K and F K Mapping to R B×H×N×D Space, and calculate the multi-head attention matrix according to the definition of formula (1).

[0028]

[0029] in, F represents attn The element at position i1 in the middle, F represents K The transpose of the i1th element, where D represents the number of multi-head attention heads;

[0030] (c) Calculate the output of multi-head attention according to the definition of formula (2).

[0031]

[0032] (d) Use the reshape operation to reshape F out Mapping to R B×N×C space;

[0033] (e) For F out Perform a linear transformation, then execute a Dropout operation with a parameter of 0.5 to obtain the final feature F. A ;

[0034] The FFN includes two layers of linear transformation operations, one layer of activation operations, and two layers of Dropout operations. The parameter of the Dropout operation is 0.5, and the nonlinear activation function GELU is selected as the activation function for the operation.

[0035] Step 1.4 Establish and initialize the Transformer network N for feature fusion. TFIt includes a multi-head cross attention layer (MHCA) and a forward propagation layer (FFN);

[0036] The MHCA process involves the following seven steps:

[0037] (a) Perform cubic linear operations on the input three-dimensional tensor F1 to obtain the features respectively. in, and F 1Q F 1K and F 1V The i2th element is B, which represents the batch size of the input tensor, N, which represents the number of features in each tensor, and C, which represents the dimension of the input tensor.

[0038] (b) Perform cubic linear operations on the input three-dimensional tensor F2 to calculate the features respectively. in, and F 2Q F 2K and F 2V The i2th element is B, which represents the batch size of the input tensor, N, which represents the number of features in each tensor, and C, which represents the dimension of the input tensor.

[0039] (c) Use the reshape operation to reshape F 1Q F 1K F 1V F 2Q F 2K and F 2V Mapping to R B×H×N×D The space is used to calculate the multi-head attention matrix according to the definitions of formulas (3) and (4). and

[0040]

[0041]

[0042] in, F represents attn1 The element at position i2 in the middle, F represents 1K The transpose of the i2th element, F represents attn2 The element at position i2 in the middle, F represents 2K The transpose of the i2th element;

[0043] (d) Calculate the output of multi-head attention according to the definitions of formulas (4) and (5). and

[0044]

[0045]

[0046] (e) Use the reshape operation to reshape F out1 and F out2 Mapping to R B×N×C space;

[0047] (f) Using the concat operation, F out1 and F out2 By concatenating along the first dimension, we obtain F. out3 ;

[0048] (g) For F out3 Perform a linear transformation, then execute a Dropout operation with a parameter of 0.5 to obtain the final feature F. CA ;

[0049] The FFN performs two layers of linear transformation operations, one layer of activation operations, and two layers of Dropout operations. The parameter of the Dropout operation is 0.5, and the nonlinear activation function GELU is selected as the activation function for the operation.

[0050] Step 1.5 Establish and initialize subnetwork N cls The subnetwork N cls There is one fully connected layer, Dense1;

[0051] The Dense1 classifier has num classification units and uses Softmax as the activation function, where num represents the total number of land cover categories to be classified.

[0052] Step 2. Input the training set H of the hyperspectral image, the training set L of the LiDAR image, the manually labeled set of pixel coordinates and labels, and then input N. MF2AMT Conduct training;

[0053] Step 2.1. Based on the manually labeled set of pixel coordinates, extract the set of all labeled pixels from the training set H of the hyperspectral image. Extract all labeled pixel sets from the training set L of the LiDAR image. in, X represents H The i3rd pixel in the image, X represents LThe i3rd pixel in the array, M represents the total number of pixels with labels;

[0054] Step 2.2. Apply the definitions of formulas (7) and (8) to X. H and X L Standardization process is performed to obtain and in, This represents the standardized set of labeled hyperspectral image pixels. express The i3rd pixel, This represents the standardized set of labeled LiDAR pixels. express The i3rd pixel;

[0055]

[0056]

[0057] Step 2.3. H is divided into a series of 11×11 hyperspectral pixel blocks centered on each pixel. and with The L is divided into a series of 11×11 LiDAR pixel blocks centered on each pixel.

[0058] Will and This serves as the training set for the fusion classification neural network, and the samples in the training set are integrated into triples. The form is used as network data input, where, This represents a pixel pair consisting of hyperspectral images and LiDAR images in the training set, and and Their spatial coordinates are the same. express and Given the corresponding real category label, let iter ← 1, and execute steps 2.8 to 2.18;

[0059] Step 2.8. Use subnetwork N CH N CL Extract features from the training set;

[0060] Step 2.8.1 Use subnetwork N CH Training set of hyperspectral images Feature extraction is performed to obtain the feature F of the hyperspectral image. CH ;

[0061] Step 2.8.2 uses subnetwork N CL Training set of LiDAR images Feature extraction is performed to obtain the feature F of the LiDAR image. CL ;

[0062] Step 2.9. Combine with F CH and F CL Random numbers with the same dimension size and F CH and F CL Add them together so that F CH and F CL Obtain initial location encoding information;

[0063] Step 2.10. Use subnetwork N TE The features are then further extracted to obtain the features HS_feature and LiDAR_feature;

[0064] Step 2.11. Use subnetwork N TF Feature fusion is performed on the features;

[0065] Step 2.12. Calculate the hyperspectral image feature similarity sim according to the definitions of formula (9) and formula (10). H2L Feature similarity with LiDAR (sim) L2H ;

[0066] sim H2L =μ·HS_feature·LiDAR_features T (9)

[0067] sim L2H =μ·LiDAR_feature·HS_features T (10)

[0068] Where μ is a scaling factor to control the range and stability of similarity values;

[0069] Step 2.13. Calculate the objective function L according to the definition of formula (11). MCL ;

[0070]

[0071] Where H() is the standard cross-entropy function, and y is the true label of this type of land cover;

[0072] Step 2.14. Based on the definitions of formulas (12) and (13), use the softmax function to measure the similarity of hyperspectral image features. H2L Feature similarity with LiDAR (sim) L2HThe calculation yields the weight matrix W. H2L and w L2H ;

[0073] W H2L =softmax(sim H2L (12)

[0074] w L2H =softmax(sim H2L (13)

[0075] Step 2.15. Define the input set of samples as positive samples, with a label of 1, and set the weight matrix W... H2L and w L2H The diagonal elements are assigned a value of 0. In the remaining similarity matrix, the sample with the highest similarity value for each sample is selected as the negative sample, and its label is assigned a value of 0. This negative sample is then combined with the positive sample to form a triple. The training samples are used in the form of L to perform a binary classification task. The objective function L is calculated according to the definition of formula (14). MML ;

[0076] L MML =E (H,L)～D H(y MML ,p mml (H,L)) (14)

[0077] Step 2.16 uses subnetwork N cls The deep features are classified, and the classification prediction result TR is calculated. pred ;

[0078] Step 2.17 According to the definitions of formulas (15) and (16), the weighted cross-entropy is used as the loss function;

[0079]

[0080]

[0081] Where, ω j This represents the weight of the j-th class. Let n represent the probability that a pixel belongs to the j-th type of land cover. j This represents the number of the j-th type of land cover in the ground-truth training samples;

[0082] Step 2.18 If all pixels in the training set have been processed, proceed to step 2.19; otherwise, remove a set of unprocessed pixels from the training set and return to step 2.8.

[0083] Step 2.19 Let iter ← iter+1. If the number of iterations iter > Total_iter, then the trained convolutional neural network N is obtained. MF2AMT Proceed to step 3; otherwise, use the backpropagation algorithm based on stochastic gradient descent and the prediction loss L. ω-C +σ1L MCL +σ2L MML Update N MF2AMT The parameters, where σ1 and σ2 are scaling factors, proceed to step 2.8 to reprocess all pixel blocks in the training set, where Total_iter represents the preset number of iterations;

[0084] Step 3: Input the hyperspectral image H′ and LiDAR image L′ to be tested, perform data preprocessing on all pixels of H′ and L′, and use the pre-trained network N. MF2AMT Complete pixel classification;

[0085] Step 3.1. Extract all pixels from H′ to form a set. Extract all pixels from L′ to form a set. in, T represents H The i5th pixel, T represents L The i-th 5th pixel, where U represents the total number of all pixels;

[0086] Step 3.2. Apply the definitions of formulas (17) and (18) to T. H and T L Standardization process is performed to obtain and in, This represents the standardized set of pixels in the hyperspectral image to be measured. express The i5th pixel, This represents the standardized set of LiDAR pixels to be measured. express The i5th pixel;

[0087]

[0088]

[0089] Step 3.3. Centered on each pixel, H′ is divided into a series of 11×11 hyperspectral pixel blocks, forming the hyperspectral image set to be measured. And then with Centered on each pixel, L′ is divided into a series of 11×11 LiDAR pixel blocks, forming the LiDAR image set to be tested.

[0090] Step 3.4. Use the fully trained subnetwork N CH N CL Extract features from the image to be tested;

[0091] Step 3.4.1 Use a fully trained subnetwork N CH hyperspectral image set to be tested Feature extraction is performed to obtain the feature F of the hyperspectral image. CH ′;

[0092] Step 3.4.2 uses a fully trained subnetwork N CL hyperspectral image set to be tested Feature extraction is performed to obtain the feature F of the LiDAR image. CL ′;

[0093] Step 3.5. (This will be related to F) CH ′ and F CL Training complete positional encodings with consistent dimensionality and F CL ′ and F CL Add;

[0094] Step 3.6. Use the fully trained subnetwork N TE The features are then further extracted to obtain the features HS_feature′ and LiDAR_feature′;

[0095] Step 3.7. Use the fully trained subnetwork N TF Feature fusion is performed on the features;

[0096] Step 3.8. Use subnetwork N cls The features are classified to calculate the classification prediction result TE. pred .

[0097] Compared with existing technologies, this invention improves the accuracy of hyperspectral image and LiDAR data fusion classification at both the feature extraction and feature fusion levels, in the following three aspects: First, it proposes a multimodal feature alignment and feature matching Transformer network. This network uses CNN to extract features from different modalities and uses the Transformer network as a complex fusion method to combine the features. Simultaneously, the original cross-attention module in the multimodal Transformer is improved, enhancing the focus on the feature extraction and fusion process, ultimately improving classification accuracy. Second, it proposes a multimodal feature alignment strategy. The proposed method solves the problem of misalignment between features of different modalities when input to the Transformer. This helps to align the representations of hyperspectral image and LiDAR data before fusion, thus simplifying the subsequent cross-modal learning of the multimodal Transformer. Third, addressing the problem that current network training only focuses on paired samples to learn their relationships, failing to effectively utilize multimodal samples with different labels, it proposes a most difficult negative sample contrastive learning method. This method effectively utilizes unmatched samples and improves the quality of feature extraction, thereby improving the quality of fusion and classification. Therefore, this invention can effectively improve the alignment of multi-source features, alleviate the problems of spatial information fragmentation and poor utilization of unmatched data, and improve detection accuracy with a smaller training set. It features good feature extraction quality, high detection accuracy, and a small number of training samples required. Experimental results show that the overall accuracy of this invention on the Houston2013 and Trento datasets reached 94.11% and 99.31%, respectively, effectively improving the accuracy of fusion classification. Attached Figure Description

[0098] Figure 1 This is a flowchart of an embodiment of the present invention.

[0099] Figure 2 This is a comparison chart of the classification results of the embodiment of the present invention with the TBCNN method, CCR-Net method, CALC method, Glt-Net method, ExViT method and MFT method on the Houston2013 dataset.

[0100] Figure 3 This is a comparison chart of the classification results of the Trento dataset by the embodiments of the present invention and the TBCNN method, CCR-Net method, CALC method, Glt-Net method, ExViT method and MFT method. Detailed Implementation

[0101] The present invention provides a remote sensing image fusion classification method based on multimodal feature alignment, as follows: Figure 1 As shown, proceed as follows:

[0102] Step 1. Establish and initialize the Transformer network N for multi-sensor remote sensing image fusion and classification. MF2AMT The N MF2AMT It contains 3 sub-networks N for feature extraction CH N CL and N TE One subnetwork N for feature fusion TF and one subnetwork N for classification cls ;

[0103] Step 1.1 Establish and initialize subnetwork N CH The subnetwork N CH It consists of 6 sets of convolutional layers, namely Conv1_0, Conv1_1, Conv1_2, Conv1_3, Conv1_4, and Conv1_5;

[0104] The Conv1_0 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0105] The Conv1_1 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0106] The Conv1_2 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0107] The Conv1_3 includes one convolutional layer, one BatchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0108] The Conv1_4 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0109] The Conv1_5 layer includes one convolution operation, one BatchNorm normalization operation, and one activation operation. The size of the convolution kernel in the convolution layer is 1×1, and each convolution kernel performs convolution operation with a stride of 1 pixel. The non-linear activation function GELU is selected as the activation function for the operation.

[0110] Step 1.2 Establish and initialize subnetwork N CL The subnetwork N CL It consists of 6 sets of convolutional layers, namely Conv2_0, Conv2_1, Conv2_2, Conv2_3, Conv2_4, and Conv2_5;

[0111] The Conv2_0 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0112] The Conv2_1 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0113] The Conv2_2 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0114] The Conv2_3 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation.

[0115] The Conv2_4 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel;

[0116] The Conv2_5 layer includes one convolution operation, one BatchNorm normalization operation, and one activation operation. The size of the convolution kernel in the convolution layer is 1×1, and each convolution kernel performs convolution operation with a stride of 1 pixel. The non-linear activation function GELU is selected as the activation function for the operation.

[0117] Step 1.3. Establish and initialize subnetwork N TE The subnetwork N TE It consists of a multi-head self-attention layer (MHSA) and a forward propagation layer (FFN);

[0118] The MHSA performs the following 5 steps:

[0119] (a) Perform cubic linear operations on the input three-dimensional tensor F to calculate the features respectively. in, and F Q F K and F V The i1th element is B, which represents the batch size of the input tensor, N, which represents the number of features in each tensor, and C, which represents the dimension of the input tensor.

[0120] (b) Using the reshape operation to reshape F Q F K and F K Mapping to R B×H×N×D Space, and calculate the multi-head attention matrix according to the definition of formula (1).

[0121]

[0122] in, F represents attn The element at position i1 in the middle, F represents K The transpose of the i1th element, where D represents the number of multi-head attention heads;

[0123] (c) Calculate the output of multi-head attention according to the definition of formula (2).

[0124]

[0125] (d) Use the reshape operation to reshape F out Mapping to R B×N×C space;

[0126] (e) For F out Perform a linear transformation, then execute a Dropout operation with a parameter of 0.5 to obtain the final feature F. A ;

[0127] The FFN includes two layers of linear transformation operations, one layer of activation operations, and two layers of Dropout operations. The parameter of the Dropout operation is 0.5, and the nonlinear activation function GELU is selected as the activation function for the operation.

[0128] Step 1.4 Establish and initialize the Transformer network N for feature fusion. TF It includes a multi-head cross attention layer (MHCA) and a forward propagation layer (FFN);

[0129] The MHCA process involves the following seven steps:

[0130] (a) Perform cubic linear operations on the input three-dimensional tensor F1 to obtain the features respectively. in, and F 1Q F 1K and F 1V The i2th element is B, which represents the batch size of the input tensor, N, which represents the number of features in each tensor, and C, which represents the dimension of the input tensor.

[0131] (b) Perform cubic linear operations on the input three-dimensional tensor F2 to calculate the features respectively. in, and F 2Q F 2K and F 2V The i2th element is B, which represents the batch size of the input tensor, N, which represents the number of features in each tensor, and C, which represents the dimension of the input tensor.

[0132] (c) Use the reshape operation to reshape F 1Q F 1K F 1V F 2Q F 2K and F 2V Mapping to R B×H×N×D The space is used to calculate the multi-head attention matrix according to the definitions of formulas (3) and (4). and

[0133]

[0134]

[0135] in, F represents attn1 The element at position i2 in the middle, F represents 1K The transpose of the i2th element, F represents attn2 The element at position i2 in the middle, F represents 2K The transpose of the i2th element;

[0136] (d) Calculate the output of multi-head attention according to the definitions of formulas (4) and (5). and

[0137]

[0138]

[0139] (e) Use the reshape operation to reshape F out1 and F out2 Mapping to R B×N×C space;

[0140] (f) Using the concat operation, F out1 and F out2 By concatenating along the first dimension, we obtain F. out3 ;

[0141] (g) For F out3 Perform a linear transformation, then execute a Dropout operation with a parameter of 0.5 to obtain the final feature F. CA ;

[0142] The FFN performs two layers of linear transformation operations, one layer of activation operations, and two layers of Dropout operations. The parameter of the Dropout operation is 0.5, and the nonlinear activation function GELU is selected as the activation function for the operation.

[0143] Step 1.5 Establish and initialize subnetwork N cls The subnetwork N cls There is one fully connected layer, Dense1;

[0144] The Dense1 classifier has num classification units and uses Softmax as the activation function, where num represents the total number of land cover categories to be classified.

[0145] Step 2. Input the training set H of the hyperspectral image, the training set L of the LiDAR image, the manually labeled set of pixel coordinates and labels, and then input N. MF2AMT Conduct training;

[0146] Step 2.1. Based on the manually labeled set of pixel coordinates, extract the set of all labeled pixels from the training set H of the hyperspectral image. Extract all labeled pixel sets from the training set L of the LiDAR image. in, X represents H The i3rd pixel in the image, X represents L The i3rd pixel in the array, M represents the total number of pixels with labels;

[0147] Step 2.2. Apply the definitions of formulas (7) and (8) to X. H and X L Standardization process is performed to obtain and in, This represents the standardized set of labeled hyperspectral image pixels. express The i3rd pixel, This represents the standardized set of labeled LiDAR pixels. express The i3rd pixel;

[0148]

[0149]

[0150] Step 2.3. H is divided into a series of 11×11 hyperspectral pixel blocks centered on each pixel. and with The L is divided into a series of 11×11 LiDAR pixel blocks centered on each pixel.

[0151] Will and This serves as the training set for the fusion classification neural network, and the samples in the training set are integrated into triples. The form is used as network data input, where, This represents a pixel pair consisting of hyperspectral images and LiDAR images in the training set, and and Their spatial coordinates are the same. express and Given the corresponding real category label, let iter ← 1, and execute steps 2.8 to 2.18;

[0152] Step 2.8. Use subnetwork N CH N CL Extract features from the training set;

[0153] Step 2.8.1 Use subnetwork N CH Training set of hyperspectral images Feature extraction is performed to obtain the feature F of the hyperspectral image. CH ;

[0154] Step 2.8.2 uses subnetwork N CL Training set of LiDAR images Feature extraction is performed to obtain the feature F of the LiDAR image. CL ;

[0155] Step 2.9. Combine with F CH and F CLRandom numbers with the same dimension size and F CH and F CL Add them together so that F CH and F CL Obtain initial location encoding information;

[0156] Step 2.10. Use subnetwork N TE The features are then further extracted to obtain the features HS_feature and LiDAR_feature;

[0157] Step 2.11. Use subnetwork N TF Feature fusion is performed on the features;

[0158] Step 2.12. Calculate the hyperspectral image feature similarity sim according to the definitions of formula (9) and formula (10). H2L Feature similarity with LiDAR (sim) L2H ;

[0159] sim H2L =μ·HS_feature·LiDAR_features T (9)

[0160] sim L2H =μ·LiDAR_feature·HS_features T (10)

[0161] Where μ is a scaling factor to control the range and stability of similarity values;

[0162] Step 2.13. Calculate the objective function L according to the definition of formula (11). MCL ;

[0163]

[0164] Where H() is the standard cross-entropy function, and y is the true label of this type of land cover;

[0165] Step 2.14. Based on the definitions of formulas (12) and (13), use the softmax function to measure the similarity of hyperspectral image features. H2L Feature similarity with LiDAR (sim) L2H The calculation yields the weight matrix W. H2L and w L2H ;

[0166] W H2L =softmax(sim H2L (12)

[0167] w L2H=softmax(sim H2L (13)

[0168] Step 2.15. Define the input set of samples as positive samples, with a label of 1, and set the weight matrix W... H2L and w L2H The diagonal elements are assigned a value of 0. In the remaining similarity matrix, the sample with the highest similarity value for each sample is selected as the negative sample, and its label is assigned a value of 0. This negative sample is then combined with the positive sample to form a triple. The training samples are used in the form of L to perform a binary classification task. The objective function L is calculated according to the definition of formula (14). MML ;

[0169] L MML =E (H,L)～D H(y MML ,p mml (H,L)) (14)

[0170] Step 2.16 uses subnetwork N cls The deep features are classified, and the classification prediction result TR is calculated. pred ;

[0171] Step 2.17 According to the definitions of formulas (15) and (16), the weighted cross-entropy is used as the loss function;

[0172]

[0173]

[0174] Where, ω j This represents the weight of the j-th class. Let n represent the probability that a pixel belongs to the j-th type of land cover. j This represents the number of the j-th type of land cover in the ground-truth training samples;

[0175] Step 2.18 If all pixels in the training set have been processed, proceed to step 2.19; otherwise, remove a set of unprocessed pixels from the training set and return to step 2.8.

[0176] Step 2.19 Let iter ← iter+1. If the number of iterations iter > Total_iter, then the trained convolutional neural network N is obtained. MF2AMT Proceed to step 3; otherwise, use the backpropagation algorithm based on stochastic gradient descent and the prediction loss L. ω-C +σ1L MCL +σ2L MML Update N MF2AMTThe parameters, where σ1 and σ2 are scaling factors, proceed to step 2.8 to reprocess all pixel blocks in the training set, where Total_iter represents the preset number of iterations;

[0177] Step 3: Input the hyperspectral image H′ and LiDAR image L′ to be tested, perform data preprocessing on all pixels of H′ and L′, and use the pre-trained network N. MF2AMT Complete pixel classification;

[0178] Step 3.1. Extract all pixels from H′ to form a set. Extract all pixels from L′ to form a set. in, T represents H The i5th pixel, T represents L The i-th 5th pixel, where U represents the total number of all pixels;

[0179] Step 3.2. Apply the definitions of formulas (17) and (18) to T. H and T L Standardization process is performed to obtain and in, This represents the standardized set of pixels in the hyperspectral image to be measured. express The i5th pixel, This represents the standardized set of LiDAR pixels to be measured. express The i5th pixel;

[0180]

[0181]

[0182] Step 3.3. Centered on each pixel, H′ is divided into a series of 11×11 hyperspectral pixel blocks, forming the hyperspectral image set to be measured. And then with Centered on each pixel, L′ is divided into a series of 11×11 LiDAR pixel blocks, forming the LiDAR image set to be tested.

[0183] Step 3.4. Use the fully trained subnetwork N CH N CL Extract features from the image to be tested;

[0184] Step 3.4.1 Use a fully trained subnetwork N CHhyperspectral image set to be tested Feature extraction is performed to obtain the feature F of the hyperspectral image. CH ′;

[0185] Step 3.4.2 uses a fully trained subnetwork N CL hyperspectral image set to be tested Feature extraction is performed to obtain the feature F of the LiDAR image. CL ′;

[0186] Step 3.5. (This will be related to F) CH ′ and F CL Training complete positional encodings with consistent dimensionality and F CL ′ and F CL Add;

[0187] Step 3.6. Use the fully trained subnetwork N TE The features are then further extracted to obtain the features HS_feature′ and LiDAR_feature′;

[0188] Step 3.7. Use the fully trained subnetwork N TF Feature fusion is performed on the features;

[0189] Step 3.8. Use subnetwork N cls The features are classified to calculate the classification prediction result TE. pred .

[0190] To verify the effectiveness of this invention, experiments were conducted using the publicly available Houston2013 and Trento datasets as examples. Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient (KAPPA) were used as objective indicators to evaluate the transform detection results. The evaluation results of this invention were compared with those of TBCNN, CCR-Net, CALC, Glt-Net, ExViT, and MFT methods.

[0191] Tables 1 and 2 show the OA, AA, and KAPPA scores for the three datasets under different algorithms. The method proposed in this invention achieves results closest to the Ground Truth. As can be seen from Tables 1 and 2, the classification accuracy of this invention is improved, achieving 100% classification accuracy across multiple categories. Taking the Houston2013 dataset as an example, the method achieves an OA of 94.11%, and the classification accuracy for most land cover categories exceeds 85%. Through sufficient feature extraction and effective feature fusion, outliers in the classification results can be avoided to the greatest extent possible. Furthermore, the introduction of contrastive learning can better distinguish features that are difficult to differentiate from the surrounding environment, such as the road category in the Houston2013 dataset.

[0192] The embodiments of this invention are compared with the classification results of the TBCNN method, CCR-Net method, CALC method, Glt-Net method, ExViT method and MFT method on the Houston2013 dataset. Figure 2 As shown. Figure 2 In the table, (a) is a pseudo-color image of the hyperspectral imagery of the Houston2013 dataset; (b) is a digital representation model of the LiDAR imagery of the Houston2013 dataset; (c) is a label of the ground cover distribution of the Houston2013 dataset; (d) is a visualization of the TBCNN method on the Houston2013 dataset; (e) is a visualization of the CCR-Net method on the Houston2013 dataset; (f) is a visualization of the CALC method on the Houston2013 dataset; (g) is a visualization of the Glt-Net method on the Houston2013 dataset; (h) is a visualization of the ExViT method on the Houston2013 dataset; (i) is a visualization of the MFT method on the Houston2013 dataset; and (j) is a visualization of the method of this invention on the Houston2013 dataset.

[0193] This invention compares the classification results of the TBCNN, CCR-Net, CALC, Glt-Net, ExViT, and MFT methods on the Trento dataset. Figure 3 As shown. Figure 3In the table, (a) is a pseudo-color image of hyperspectral imagery from the Trento dataset; (b) is a digital representation model of LiDAR imagery from the Trento dataset; (c) is a label of real ground cover distribution from the Trento dataset; (d) is a visualization of the TBCNN method on the Trento dataset; (e) is a visualization of the CCR-Net method on the Trento dataset; (f) is a visualization of the CALC method on the Trento dataset; (g) is a visualization of the Glt-Net method on the Trento dataset; (h) is a visualization of the ExViT method on the Trento dataset; (i) is a visualization of the MFT method on the Trento dataset; and (j) is a visualization of the method of this invention on the Trento dataset.

[0194] from Figure 2 , Figure 3 As can be seen, a more accurate classification map was obtained for the Trento dataset, with more corresponding Ground Truth pixels and smoother boundaries. This is because, compared to the Houston2013 dataset, the land classification in these two datasets is simpler and the land distribution is more continuous. Furthermore, the classification map of this invention does not have obvious discrete pixels. By comparing the classification maps of various algorithms, it can be found that, compared to the Transformer framework, the resulting map extracted by this invention has smoother and more accurate classification results. This is because, compared to the traditional Transformer framework, this invention reduces the loss of useful information by considering stronger feature extraction and representation. Compared to CNN methods, this invention fully integrates and utilizes the unique information of different modalities, making the boundaries of the classification results clearer, especially in areas with complex terrain classification.

[0195] Combine Table 1 and Table 2 Figure 2 , Figure 3 The comparison results show that the present invention effectively improves the multi-source feature alignment effect, alleviates the problems of spatial information fragmentation and poor utilization of unmatched data, and achieves improved measurement accuracy with a small training set by constructing a multi-sensor remote sensing image fusion classification model based on the Transformer network framework of multi-modal feature alignment and feature matching.

[0196] Table 1 Objective evaluation metrics for the Houston dataset under different algorithms

[0197]

[0198] Table 2 Objective evaluation metrics for the Trento dataset under different algorithms

[0199]

Claims

1. A remote sensing image fusion classification method based on multi-modal feature alignment, characterized in that Follow these steps: Step 1. Establish and initialize a Transformer network for multi-sensor remote sensing image fusion and classification. The It contains 3 subnetworks for feature extraction. , and One subnetwork for feature fusion and one subnetwork for classification ; Step 1.1 Establish and initialize the subnetwork The sub-network It consists of 6 sets of convolutional layers, namely Conv1_0, Conv1_1, Conv1_2, Conv1_3, Conv1_4, and Conv1_5; The Conv1_0 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel; The Conv1_1 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation. The Conv1_2 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel; The Conv1_3 includes one convolutional layer, one BatchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation. The Conv1_4 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel; The Conv1_5 layer includes one convolution operation, one BatchNorm normalization operation, and one activation operation. The size of the convolution kernel in the convolution layer is 1×1, and each convolution kernel performs convolution operation with a stride of 1 pixel. The non-linear activation function GELU is selected as the activation function for the operation. Step 1.2 Establish and initialize the subnetwork The sub-network It consists of 6 sets of convolutional layers, namely Conv2_0, Conv2_1, Conv2_2, Conv2_3, Conv2_4, and Conv2_5; The Conv2_0 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel; The Conv2_1 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation. The Conv2_2 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel; The Conv2_3 includes one convolutional layer, one batchNorm normalization layer, and one activation layer. The size of the convolutional kernel in the convolutional layer is 1×1. Each convolutional kernel performs convolutional operations with a stride of 1 pixel and uses the non-linear activation function GELU as the activation function for the operation. The Conv2_4 contains one convolutional layer, wherein the size of the convolutional kernel in the convolutional layer is 3×3, and each convolutional kernel performs convolution operation with a stride of 1 pixel; The Conv2_5 layer includes one convolution operation, one BatchNorm normalization operation, and one activation operation. The size of the convolution kernel in the convolution layer is 1×1, and each convolution kernel performs convolution operation with a stride of 1 pixel. The non-linear activation function GELU is selected as the activation function for the operation. Step 1.3 Establish and initialize the subnetwork The sub-network It consists of a multi-head self-attention layer (MHSA) and a forward propagation layer (FFN); The MHSA performs the following 5 steps: (a) Input 3D tensor Performing three linear operations to calculate the features respectively , , ,in, , and They represent , and The One element, This indicates the batch size of the input tensor. This represents the number of features in each tensor. Indicates the dimension of the input tensor; (b) Using the reshape operation to... , and Mapped to Space, and calculate the multi-head attention matrix according to the definition of formula (1). ; (1) in, express The Middle The element at position 1 express The Transpose of an element This indicates the number of bullish heads. (c) Calculate the output of multi-head attention according to the definition of formula (2). ; (2) (d) Use the reshape operation to... Mapped to space; (e) Perform a linear transformation, then execute a Dropout operation with a parameter of 0.5 to obtain the final features. ; The FFN includes two layers of linear transformation operations, one layer of activation operations, and two layers of Dropout operations. The parameter of the Dropout operation is 0.5, and the nonlinear activation function GELU is selected as the activation function for the operation. Step 1.4: Establish and initialize the Transformer network for feature fusion. It includes a multi-head cross attention layer (MHCA) and a forward propagation layer (FFN); The MHCA process involves the following seven steps: (a) Input 3D tensor Performing three linear operations to calculate the features respectively , , ,in, , and They represent , and The One element, This indicates the batch size of the input tensor. Represents the number of features in each tensor and Indicates the dimension of the input tensor; (b) Input 3D tensor Performing three linear operations to calculate the features respectively , , ,in, , and They represent , and The One element, This indicates the batch size of the input tensor. Represents the number of features in each tensor and Indicates the dimension of the input tensor; (c) Using the reshape operation to... , , , , and Mapped to Space, and calculate the multi-head attention matrix according to the definitions of formulas (3) and (4). and ; (3) (4) in, express The Middle The element at position 1 express The Transpose of an element express The Middle The element at position 1 express The Transpose of each element; (d) Calculate the output of multi-head attention according to the definitions of formulas (4) and (5). and ; (5) (6) (e) Use the reshape operation to... and Mapped to space; (f) Using the concat operation, and By concatenating along the first dimension, we obtain... ; (g) Perform a linear transformation, then execute a Dropout operation with a parameter of 0.5 to obtain the final features. ; The FFN performs two layers of linear transformation operations, one layer of activation operations, and two layers of Dropout operations. The parameter of the Dropout operation is 0.5, and the nonlinear activation function GELU is selected as the activation function for the operation. Step 1.5 Establish and initialize the subnetwork The sub-network There is one fully connected layer, Dense1; The Dense1 classifier has num classification units and uses Softmax as the activation function, where num represents the total number of land cover categories to be classified. Step 2. Input the training set of hyperspectral images Training set of LiDAR images The manually labeled set of pixel coordinates and label set, for Conduct training; Step 2.1 Based on the manually labeled set of pixel coordinates, in the training set of the hyperspectral image... Extract the set of all labeled pixels. and in the training set of LiDAR images Extract the set of all labeled pixels. ,in, express The first in One pixel, express The first in One pixel, This represents the total number of labeled pixels; Step 2.2 Based on the definitions of formulas (7) and (8), and Standardization process is performed to obtain and ,in, This represents the standardized set of labeled hyperspectral image pixels. express The One pixel, This represents the standardized set of labeled LiDAR pixels. express The One pixel; (7) (8) Step 2.3 Centered on each pixel point Divided into a series of 11×11 hyperspectral pixel blocks and with Centered on each pixel point Divided into a series of 11×11 LiDAR pixel blocks ; Will and This serves as the training set for the fusion classification neural network, and the samples in the training set are integrated into triples. The form is used as network data input, where, This represents a pixel pair consisting of hyperspectral images and LiDAR images in the training set, and and Their spatial coordinates are the same. express and The corresponding real category label, causing the number of iterations. Proceed to steps 2.4 through 2.14; Step 2.4 Employing Subnetworks , Extract features from the training set; Step 2.4.1 Employing a subnetwork Training set of hyperspectral images Feature extraction is performed to obtain the features of the hyperspectral image. ; Step 2.4.2 uses a sub-network Training set of LiDAR images Feature extraction is performed to obtain the features of the LiDAR image. ; Step 2.5 will be with and Random numbers with the same dimension size and and Adding them together makes and Obtain initial location encoding information; Step 2.6 Employing Subnetworks Further feature extraction is performed on the features to obtain the final features. and ; Step 2.7 uses a subnetwork Feature fusion is performed on the features; Step 2.8 Calculate the hyperspectral image feature similarity according to the definitions of formulas (9) and (10). Feature similarity with LiDAR ; (9) (10) in, This is a scaling factor to control the range and stability of similarity values; Step 2.9 Calculate the objective function according to the definition of formula (11). ; (11) in, For the standard cross-entropy function, This serves as the true label for this type of land feature; Step 2.10: Based on the definitions of formulas (12) and (13), use the softmax function to evaluate the similarity of hyperspectral image features. Feature similarity with LiDAR The weight matrix is ​​obtained by calculation. and ; (12) (13) Step 2.11 Define the input set of samples as positive samples with a label of 1, and then apply the weight matrix... and The diagonal elements are assigned a value of 0. In the remaining similarity matrix, the sample with the highest similarity value for each sample is selected as the negative sample, and its label is assigned a value of 0. This negative sample is then combined with the positive sample to form a triple. The training samples are used in the form of [formula] for a binary classification task, and the objective function is calculated according to the definition of formula (14). ; (14) Step 2.12 uses a sub-network The deep features are classified, and the classification prediction results are calculated. ; Step 2.13 According to the definitions of formulas (15) and (16), the weighted cross-entropy is used as the loss function; (15) (16) in, Indicates the first Class weight, Indicates that the cell belongs to the first The probability of land-like features Indicates the first training sample in the ground-truth training sample The number of land features; Step 2.14 If all pixels in the training set have been processed, proceed to step 2.15; otherwise, remove a set of unprocessed pixels from the training set and return to step 2.

4. Step 2.15 If the number of iterations This yields a pre-trained convolutional neural network. Proceed to step 3; otherwise, utilize the backpropagation algorithm based on stochastic gradient descent and the prediction loss. renew The parameters, where and As the scaling factor, proceed to step 2.4 to reprocess all pixel blocks in the training set. Indicates the preset number of iterations; Step 3. Input the hyperspectral image to be measured. and LiDAR images ,right and All pixels were preprocessed, and a pre-trained network was used. Complete pixel classification; Step 3.1 in Extract all pixels to form a set ,exist Extract all pixels to form a set ,in, express The One pixel, express The One pixel, Represents the total number of all pixels; Step 3.2 Based on the definitions of formulas (17) and (18), and Standardization process is performed to obtain and ,in, This represents the standardized set of pixels in the hyperspectral image to be measured. express The One pixel, This represents the standardized set of LiDAR pixels to be measured. express The One pixel; (17) (18) Step 3.3 Centered on each pixel point The hyperspectral image set is divided into a series of 11×11 hyperspectral pixel blocks to form the image set to be measured. and then with Centered on each pixel point The LiDAR image set is composed of a series of 11×11 LiDAR pixel blocks. ; Step 3.4 Employ a fully trained subnetwork , Extract features from the image to be tested; Step 3.4.1 Use a fully trained subnetwork hyperspectral image set to be tested Feature extraction is performed to obtain the features of the hyperspectral image. ; Step 3.4.2 Employ a fully trained subnetwork hyperspectral image set to be tested Feature extraction is performed to obtain the features of the LiDAR image. ; Step 3.5 will be with and Training complete position encoding with consistent dimensionality and and Add; Step 3.6 Employ a fully trained subnetwork Further feature extraction is performed on the features to obtain the final features. and ; Step 3.7 Employ a fully trained subnetwork Feature fusion is performed on the features; Step 3.8 Employing Subnetworks The features are classified to calculate the classification prediction result. .

Images