Hyperspectral remote sensing cross-scene classification method and system based on multi-view differential convolution and side network migration strategy

Abstract

The invention discloses a hyperspectral remote sensing cross-scene classification method and system based on multi-view differential convolution and a side network migration strategy, and the method comprises the steps: obtaining a hyperspectral remote sensing image, and carrying out the preprocessing of the hyperspectral remote sensing image, and obtaining to-be-detected data; to-be-detected data is input into a pre-trained classification model for identification and classification, the classification model comprises a backbone network and a side network, the backbone network and the side network each comprise a feature extraction layer, a multi-view feature enhancement layer and a result output layer, the multi-view feature enhancement layer comprises a plurality of multi-view difference convolution modules, and the multi-view difference convolution modules comprise a plurality of multi-view difference convolution modules; the side network is obtained by training the backbone network through a side network migration strategy, and the backbone network and the side network are connected with each other through a gating connection layer. According to the method, the efficient and accurate classification of the complex hyperspectral remote sensing images is realized under the condition of not depending on a large amount of annotated data.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral image remote sensing technology, and in particular to a hyperspectral remote sensing cross-scene classification method and system based on multi-view differential convolution and side network migration strategy. Background Art

[0002] Hyperspectral remote sensing images (HSI) record a wide range of spectral information from ultraviolet to near-infrared wavelengths of surface objects using spectral imaging sensors carried by satellites or aircraft. Compared to traditional visible spectrum remote sensing images, hyperspectral remote sensing images possess richer spectral information, reflecting more detailed spectral reflectance characteristics of objects within the remote sensing scene. Their pixel features are also more capable of identifying the attributes of target objects. Therefore, object pixel classification technology based on hyperspectral images has enormous potential for development in remote sensing information processing and has been widely applied in industries such as geological exploration, military reconnaissance, precision agriculture, and urban planning. However, in practical applications, newly acquired hyperspectral remote sensing images generally face the challenge of obtaining labeled data. The high time and cost associated with manually labeling prior labeled samples often results in insufficient labeled training samples for hyperspectral remote sensing image classification tasks.

[0003] Hyperspectral remote sensing image classification methods based on deep learning typically rely on large amounts of labeled data for training. However, the available labeled data in practical applications is often very limited, which conflicts with the large-scale labeled data requirements of deep neural networks. Furthermore, existing hyperspectral image classification models based on convolutional neural networks struggle to adaptively extract the rich texture features resulting from changes in object distribution, further leading to a decline in classification performance with small labeled samples. Therefore, how to rationally learn the fine texture features of hyperspectral images based on changes in object distribution and how to incorporate other available prior labeling information are key to addressing the poor classification performance of hyperspectral images with small labeled samples.

[0004] Differential convolution and attention mechanisms are two approaches for improving the model's feature extraction capabilities by learning the fine-grained texture features inherent in the differential distribution of features in hyperspectral imagery. Traditional convolutional neural networks typically employ two-dimensional or three-dimensional convolutions, which only aggregate local information and fail to explicitly extract image gradients. Consequently, they cannot fully exploit fine-grained texture features. However, differential convolution excels at extracting image edges and detail features. Since the introduction of the LBP operator, differential convolution has evolved into various variants, including center-differential convolution, cross-center-differential convolution, spatiotemporal-differential convolution, and differential graph convolution, each of which exploits unique deep image features from a different convolutional perspective. Furthermore, the attention mechanism, inspired by the human visual system, enhances the model's focus on important regions and improves its ability to model long-range dependencies, thereby more effectively extracting useful information and suppressing useless information. Currently, various forms of attention mechanisms have been developed, including general attention, self-attention, channel attention, spatial attention, hybrid attention, and temporal attention. However, as high-dimensional data, hyperspectral remote sensing images not only have a variable spatial distribution structure of ground objects, but also present complex feature distribution in the high-dimensional spectral feature space. How to reasonably combine differential convolution and attention mechanism for efficient classification remains a challenge.

[0005] In addition to improving feature learning performance, the scarcity of annotated data for hyperspectral remote sensing images is another pressing issue that needs to be addressed. Transfer learning, which leverages knowledge from existing data to assist in learning new data, has become an important means of improving data utilization efficiency. The basic idea of transfer learning is to leverage prior knowledge of a known scene (source domain) to aid learning of a new scene (target domain). The source domain represents a known scene with abundant annotated data, while the target domain represents a new scene with scarce annotated data. Based on the different transfer methods, it can be further categorized into sample-based transfer, feature-based transfer, relationship-based transfer, and model-based transfer. Sample-based transfer reuses samples from the source and target domains through weight adjustment. Feature-based transfer aligns the feature distributions of the source and target domains through feature space transformation. Relation-based transfer leverages logical relationships within the source domain to assist learning in the target domain. Model-based transfer transfers data through parameter sharing. However, cross-scene transfer learning of hyperspectral images also faces many challenges, with negative transfer being one of the main issues. Negative transfer refers to the situation where, when the difference between the source and target domains is too large or the transfer strategy is inappropriate, transfer learning may not only fail to improve the classification performance of the target domain, but may also introduce noise, leading to a decrease in model performance. Therefore, how to rationally design a transfer strategy to avoid negative transfer and fully utilize source domain knowledge to achieve accurate cross-domain learning without significantly increasing the computational burden of training with source domain data is an important direction in current research on cross-scene classification of hyperspectral images.

[0006] Although differential convolution and cross-scene transfer have shown certain potential in solving the problem of small-sample hyperspectral remote sensing image classification, they still face a series of difficult problems that need to be solved. First, whether it is center difference convolution, cross center difference convolution or spatiotemporal difference convolution, they only learn image texture features from the perspective of gradient change, and are difficult to adapt to the complex and variable gradient changes formed by the differentiated distribution characteristics of objects in different regions of the image. The lack of combining differential convolution with the attention mechanism makes it difficult to suppress redundant features and learn key high-value fine-grained texture features. In addition, most of the current hyperspectral cross-scene transfer learning is based on the overall cross-domain transfer of all model parameters. This undifferentiated transfer method cannot fully consider the cross-scene adaptability problem, resulting in insufficient generalization ability in the target domain. While increasing the computational burden, it is also prone to negative transfer. Summary of the Invention

[0007] In order to address the shortcomings of the existing technology, the present invention provides a hyperspectral remote sensing cross-scene classification method and system based on multi-view differential convolution and side network migration strategy. By combining multi-view differential convolution and side network migration strategy, efficient and accurate classification of complex hyperspectral remote sensing images is achieved without relying on a large amount of labeled data.

[0008] In order to achieve the above object, the specific scheme adopted by the present invention is: a hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy, comprising the following steps: Acquire hyperspectral remote sensing images and perform preprocessing to obtain data to be detected; The data to be detected is input into a pre-trained classification model for identification and classification. The classification model includes a backbone network and a side network. Both the backbone network and the side network include a feature extraction layer, a multi-view feature enhancement layer, and a result output layer. The multi-view feature enhancement layer includes multiple multi-view differential convolution modules. The side network is trained by the backbone network through the side network migration strategy. The backbone network and the side network are connected to each other through a gated connection layer. The method for the classification model to identify and classify the data to be detected includes: The feature extraction layer of the side network and the backbone network is used to process the data to be detected to obtain the basic detection features; The multi-view feature enhancement layer of the backbone network is used to process the basic detection features to obtain the intermediate detection features; The gated connection layer is used to fuse the detection intermediate features of the side network and the backbone network; The multi-view feature enhancement layer of the side network is used to process the detection fusion features to obtain the classification fusion features; The result output layer of the side network is used to classify the classification fusion features to obtain the classification results and output them.

[0009] As a further optimization of the hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy, the method for training the classification model includes: Obtain source domain image data and target domain image data, and perform dimensionality reduction through principal component analysis to obtain low-dimensional image data; Randomly classify low-dimensional image data to obtain sample data and initialize the classification model; Train the classification model in the source domain based on sample data; Update and train the classification model based on the source domain loss; Use the side network transfer strategy to fine-tune the classification model on the target domain data and train the side network; Calculate the loss and update the side network parameters according to the loss to obtain the best classification model.

[0010] As a further optimization of the hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy, a method for obtaining source domain image data and target domain image data and performing dimensionality reduction through principal component analysis to obtain low-dimensional image data includes: Read source domain image data X S1 and target domain image data X T1 ; For the source domain image data X S1 and target domain image data X T1 Centralized processing; Calculate the source domain image data X after centralized processing S1 and target domain image data X T1 The covariance matrix of The covariance matrix is decomposed into eigenvalues in multiple dimensions, and the eigenvalues are arranged in descending order; multiple larger eigenvalues are selected from all eigenvalues, and the eigenvectors of the selected eigenvalues are used to form a transformation matrix; based on the transformation matrix, the source domain image data X is transformed into S1 and target domain image data X T1 Processing is performed to obtain low-dimensional image data, which includes low-dimensional source domain data X S2 ∈R m×h×w and low-dimensional target domain data X T2 ∈R m×h×w , where m is the spectral dimension, h and w are the spatial dimensions of the low-dimensional data.

[0011] As a further optimization of the hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy, the method of randomly classifying low-dimensional image data to obtain sample data includes: Calculate the number of samples corresponding to each category based on the number of low-dimensional image data and the number of categories; Based on the number of samples, the low-dimensional image data is randomly classified to obtain sample data, which includes the source domain trainable samples X S ∈R N×S and target domain trainable samples X T ∈R N×S , where N is the batch number and S is the feature size.

[0012] As a further optimization of the above-mentioned hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy: the method of source domain training of the classification model based on sample data includes: The source domain trainable samples X in the sample data S Input to the feature embedding layer of the classification model for processing to obtain the first layer input X of the multi-view differential convolution module 1 ∈R N×S , the processing method is: Among them, Maxpool 1×1 It is a maximum pooling operation with a kernel size of 1×1; The first layer input of the multi-view differential convolution module is X 1 ∈R N×S Diverse features are extracted by sequentially performing multiple multi-view differential convolutions, and basic features are extracted by ordinary convolution. Multi-view differential convolutions include pixel differential convolution (CPDC) based on center difference, pixel differential convolution (APDC) based on angle difference, and pixel differential convolution (RPDC) based on radial difference. The extracted diverse features are input into the multi-view interactive self-attention module, which is guided by the basic features and outputs the L-th layer feature O L ∈R N×S .

[0013] As a further optimization of the hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy, the extracted basic features and diverse features are: Among them, G, G CPDC , G APDC , G RPDC are the convolution kernel parameters corresponding to ordinary convolution, APDC, CPDC and RPDC respectively, while B, B CPDC 、B APDC 、B RPDC Represents the corresponding bias parameters, and the convolution kernel size used is 3×3; guide and output the L-th layer feature O through the basic features L ∈R N×S The methods include: Use three 1×1 2D convolutions and Get the L-th layer query vector Q extracted from the basic features L , the L-th layer key vector extracted from diverse features and the L-th layer numerical vector The formula is: Query vector Q L With key vector Do matrix multiplication and combine with the query vector Q L Addition further extracts data features; Use the softmax function to calculate the attention matrix of the obtained data features from three different pixel relationships and add them together: Where c, i, and j represent the cth channel, ith row, and jth column respectively; The output O is obtained by adding the multiple attention matrices and connecting them with residual connections between modules. L : in Represents the attention score of the L-th layer multi-view differential convolution module CPDC, APDC, and RPDC, X is the basic feature of the input multi-view interactive self-attention mechanism, L It is the input of the L-th layer multi-view differential convolution module.

[0014] As a further optimization of the above-mentioned hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy: the method for calculating the source domain loss is: Among them, L CEL is the cross entropy loss result, Indicates the predicted value of the i-th sample in the j-th category in the one-hot form, Y ij is the true value of the i-th sample in the j-th category.

[0015] As a further optimization of the hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy, a method for fine-tuning the classification model on target domain data and training the side network using the side network migration strategy includes: freezing the trained backbone network in the classification model at the parameter level; By introducing the side network, we can evaluate parameters based on the channel dimension and the backbone network parameters to extract effective parameters, and achieve adaptive migration of some parameters from the backbone network to the side network. The backbone network and the side network are connected through gated ladder connections.

[0016] A hyperspectral remote sensing cross-scene classification system based on multi-view differential convolution and side network migration strategy is used to implement the above-mentioned hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy. The system includes:

[0017] The data acquisition module is used to acquire hyperspectral remote sensing images and perform preprocessing to obtain the data to be detected;

[0018] The model running module is used to run the pre-trained classification model to identify and classify the objects to be detected.

[0019] Beneficial effects: The present invention proposes a cross-scene classification method for hyperspectral remote sensing images based on multi-view differential convolution and side network migration strategy to improve the classification accuracy under small sample conditions and reduce the computational burden of cross-scene migration. First, in view of the high dimensionality of the spectral characteristics of hyperspectral remote sensing images and the complexity of the distribution of objects, a multi-view differential convolution network is designed to extract texture features in different gradient directions through a variety of pixel differential calculation methods, so that the model pays more attention to the fine-grained edge details of the objects. At the same time, considering that the powerful learning ability of traditional two-dimensional convolution for local features can be synergistically complementary with the edge feature extraction of multi-view differential convolution, a multi-view interactive self-attention mechanism is established, and the traditional convolution local features are used to guide the feature learning of pixel differential convolution. The interaction between local spectral-spatial information and edge texture information is modeled in three-dimensional space, and the collaborative learning of local image information and edge structure is enhanced, thereby improving the feature expression ability of the model on complex hyperspectral data. In addition, considering that the cross-scene classification task of hyperspectral remote sensing images has challenges such as large data distribution differences and high risk of negative transfer, the present invention further proposes a side network migration strategy. This strategy introduces a lightweight side network after the backbone network learns the source domain knowledge. During the target domain training, only the side network parameters are updated, while the backbone network parameters remain frozen. Compared with the traditional transfer learning method that requires migrating the entire model parameters and then fine-tuning and updating them in the target domain, the side network migration strategy can effectively reduce the consumption of computing resources while improving the adaptability of cross-scene learning, effectively suppressing the negative transfer phenomenon, and ensuring the stability of the target domain classification performance. In summary, the present invention improves the feature extraction capability of hyperspectral remote sensing images through multi-view differential convolution, and optimizes the cross-scene knowledge transfer process in combination with the side network migration strategy, achieving efficient hyperspectral image classification in the case of insufficient prior knowledge, which is of great significance to the analysis and application of hyperspectral remote sensing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a diagram of the overall cross-scenario network transfer learning framework of the present invention; Figure 2This is a structural diagram of the multi-view differential convolution module of the present invention; Figure 3 Visualization results of different methods on the PaviaU dataset, including (a) MSCDC, (b) DEA-net, (c) DCFSL, (d) Tri-Former, (e) Lwnet, and (f) the result of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] like Figure 1 and 2 As shown in the figure, a hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy is proposed, including S100 to S200.

[0023] S100: Acquire a hyperspectral remote sensing image and perform preprocessing to obtain data to be detected.

[0024] S200. Input the data to be detected into a pre-trained classification model for identification and classification. The classification model includes a backbone network and a side network. Both the backbone network and the side network include a feature extraction layer, a multi-view feature enhancement layer and a result output layer. The multi-view feature enhancement layer includes multiple multi-view differential convolution modules. The side network is trained by the backbone network through a side network migration strategy. The backbone network and the side network are interconnected through a gated connection layer. The method for the classification model to identify and classify the data to be detected includes S201 to S205.

[0025] S201, using the feature extraction layer of the side network and the backbone network to process the data to be detected to obtain basic detection features;

[0026] S202, using the multi-view feature enhancement layer of the backbone network to process the detection basic features to obtain detection intermediate features;

[0027] S203, using the gated connection layer to fuse the detection intermediate features of the side network and the backbone network;

[0028] S204, using the multi-view feature enhancement layer of the side network to process the detection fusion feature to obtain the classification fusion feature;

[0029] S205. Use the result output layer of the side network to classify the classification fusion features to obtain the classification results, and output them.

[0030] Furthermore, the method for training the classification model includes T1 to T6.

[0031] T1. Acquire source domain image data and target domain image data, and perform dimensionality reduction through principal component analysis to obtain low-dimensional image data. The method of acquiring source domain image data and target domain image data and performing dimensionality reduction through principal component analysis to obtain low-dimensional image data includes T11 to T16.

[0032] T11. Read source domain image data X S1 and target domain image data X T1 .

[0033] T12, source domain image data X S1 and target domain image data X T1 Centralized processing.

[0034] T13. Calculate the source domain image data X after centralized processing S1 and target domain image data X T1 The covariance matrix of .

[0035] T14. Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues in multiple dimensions, and arrange the eigenvalues in descending order.

[0036] T15. Select multiple larger eigenvalues from all eigenvalues, and use the eigenvectors of the selected eigenvalues to form a transformation matrix.

[0037] T16, based on the transformation matrix to the source domain image data X S1 and target domain image data X T1 Processing is performed to obtain low-dimensional image data, which includes low-dimensional source domain data X S2 ∈R m×h×w and low-dimensional target domain data X T2 ∈R m×h×w , where m is the spectral dimension, h and w are the spatial dimensions of the low-dimensional data.

[0038] T2. Randomly classify the low-dimensional image data to obtain sample data, and initialize the classification model. The method of randomly classifying the low-dimensional image data to obtain sample data includes T21 to T22.

[0039] T21. Calculate the number of samples corresponding to each category based on the amount of low-dimensional image data and the number of categories.

[0040] T22, randomly classify the low-dimensional image data based on the number of samples to obtain sample data, which includes the source domain trainable sample X S ∈R N×S and target domain trainable samples XT ∈R N×S , where N is the number of batches and S is the feature size.

[0041] T3. Perform source domain training on the classification model based on the sample data. The method for performing source domain training on the classification model based on the sample data includes T31 to T33.

[0042] T31, the source domain trainable sample X in the sample data S Input to the feature embedding layer of the classification model for processing to obtain the first layer input X of the multi-view differential convolution module 1 ∈R N×S , the processing method is: Among them, Maxpool 1×1 is a maximum pooling operation with a kernel size of 1×1.

[0043] More specifically, The first one is the convolution formula for the pixel at the i-th row and j-th column position, and the second one is the maximum pooling formula, X L [i,j] represents the feature vector of the pixel at the i-th row and j-th column position of the L-th layer multi-view differential convolution module, m and n represent the kernel size, G[u,v] is the weight value of the convolution kernel function at the i-th row and v-th column position, X L [i+u,j+v] is the pixel value of the data feature of the L-th layer multi-view differential convolution module at the i+u-th row and j+v-th column position, B represents the bias of the 2D convolution, X L-1 [i+u,j+v] is the pixel value at the i+uth row and j+vth column position of the data feature of the L-th layer multi-view differential convolution module.

[0044] T32, the first layer input X of the multi-view differential convolution module 1 ∈R N×S Diverse features are extracted by sequentially performing multiple multi-view differential convolutions, and basic features are extracted through ordinary convolution. Multi-view differential convolution includes pixel differential convolution (CPDC) based on center difference, pixel differential convolution (APDC) based on angle difference, and pixel differential convolution (RPDC) based on radial difference. Based on T32, it can fully mine fine-grained texture features and extract edge information. The multi-view feature enhancement layer consists of three multi-view differential convolution modules. Pixel differential convolution first calculates the pixel difference of the image data, and then performs convolution with special kernel weights to extract features that are more conducive to capturing image gradient information and edge information, thereby achieving more stable and accurate edge detection while retaining the powerful learning ability of CNN. The mathematical formulas of the three differential convolution modules are as follows.

[0045] The mathematical formula of CPDC is: where ω i is the original kernel weight, is the kernel weight after mathematical calculation. The pixel difference convolution based on center difference CPDC is to perform center difference calculation on the neighborhood pixel features with a range of 3×3 based on the center pixel value, and then perform convolution calculation on the data.

[0046] The mathematical formula for APDC is:

[0047] where ω i is the original kernel weight, is the kernel weight after mathematical calculation. The pixel differential convolution based on perspective difference APDC is based on the clockwise difference calculation of the neighborhood pixel features in the range of 3×3, and the convolution operation is performed with the designed kernel size.

[0048] The mathematical formula for RPDC is:

[0049] where ω i is the original kernel weight, is the kernel weight after mathematical calculation. The radial difference pixel differential convolution RPDC is expanded to a larger receptive field of 5×5, and the difference calculation is performed between the outer ring of the 5×5 range and the inner ring of the 3×3 range, and the convolution operation is performed with the designed convolution kernel size. The processed feature X L The input is fed into three pixel differential convolution modules to extract three types of diverse features. At the same time, basic features are extracted through traditional convolution. The original diverse features and basic features are processed through a batch normalization layer and a sigmoid activation function σ1. The processed features are then further extracted through a perception layer constructed using a 3×3 convolution kernel, a batch normalization layer, and a ReLU activation function σ, where n = {1, 2, 3}.

[0050] T33, input the extracted diverse features into the multi-view interactive self-attention module, guide and output the L-th layer feature O through the basic features L ∈R N×S .

[0051] Furthermore, the extracted basic features and diverse features are: Among them, G, G CPDC , G APDC , G RPDCare the convolution kernel parameters corresponding to ordinary convolution, APDC, CPDC and RPDC respectively, while B, B CPDC 、B APDC 、B RPDC Represents the corresponding bias parameters, and the convolution kernel size used is 3×3.

[0052] Guided by basic features and outputting the L-th layer feature O L ∈R N×S The method includes T331 to T334.

[0053] T331, using three 1×1 size 2D convolutions and Get the L-th layer query vector Q extracted from the basic features L , the L-th layer key vector extracted from diverse features and the L-th layer numerical vector The formula is:

[0054] T332, query vector Q L With key vector Do matrix multiplication and combine with the query vector Q L The addition further extracts the data features.

[0055] T333. Use the softmax function to calculate the attention matrix of the obtained data features from three different pixel relationships and add them together: Where c, i, and j represent the cth channel, ith row, and jth column respectively.

[0056] T334, add the various attention matrices to get the output O L Specifically, the probability of pixel values between the three dimensions of the three-dimensional feature is calculated by using a variety of softmax functions. Then the attention coefficient matrix is combined with the numerical vector Perform another special multiplication operation, specifically multiplying the attention coefficient matrix with the numerical vector Do element-wise multiplication with a numeric vector Add together to obtain a diverse attention matrix, and then add the diverse attention matrices to get the output O′ L , and output O′ L With basic features Perform special multiplication operations, accompanied by residual connections between modules, to obtain the result O L , the formula is: in Represents the attention score of the L-th layer multi-view differential convolution module CPDC, APDC, and RPDC, X is the basic feature of the input multi-view interactive self-attention mechanism, L It is the input of the L-th layer multi-view differential convolution module.

[0057] Furthermore, the source domain data features are extracted and fused through a three-layer multi-view differential convolution module to fully extract the complex and rich features of the hyperspectral image data and perform deep fusion. 3 Perform adaptive average pooling (AdaptiveAvgPool, AAP), the formula for adaptive average pooling is as follows: Where X[c,i,j] is the value of the output feature map in the i-th row and j-th column of the c-channel, and X′[c,h,w] is the value of the input feature map in the h-th row and w-th column of the c-channel. is the region of the i-th row and j-th column of the input feature map, is the number of elements in the region. The kernel size and the step size of each kernel movement are adaptively calculated based on the input data features. Samples are extracted by calculating the average value to achieve the purpose of compressing the data feature size and extracting features. Finally, a fully connected layer is used to map the extracted features to the sample label space and integrate the features extracted by the network. The process formula is as follows: O CEL =FC(AAP(O 3 )); Among them O CEL ∈R N×m is the final feature prediction result to be fed into the cross entropy loss function, where m is the number of categories, O 3 is the deep feature extracted by three multi-view differential convolution modules, AAP(g) is the adaptive average pooling layer, and FC(g) is the fully connected layer.

[0058] T4. Update and train the classification model based on the source domain loss. The method for calculating the source domain loss is: Among them, L CEL is the cross entropy loss result, Indicates the predicted value of the i-th sample in the j-th category in the one-hot form, Y ij is the true value of the i-th sample in the j-th category.

[0059] T5. Fine-tune the classification model with target domain data using the side network migration strategy and train the side network. The method of fine-tuning the classification model with target domain data and training the side network using the side network migration strategy includes T51 to T53.

[0060] T51. Freeze the trained backbone network in the classification model at the parameter level.

[0061] T52: Introduce a side network, which is a lightweight ladder network. Parameter evaluation is performed based on the channel dimension and the backbone network parameter level to extract effective parameters, and achieve adaptive migration of some parameters from the backbone network to the side network.

[0062] T53, connect the backbone network and the side network through a gated ladder connection.

[0063] Furthermore, the specific process of T5 can be described as follows.

[0064] The trained backbone network is first frozen at the parameter level, and the side network is introduced. Parameter evaluation is performed based on the channel dimension and the backbone network parameters to extract effective parameters, enabling the adaptive transfer of some parameters from the backbone network to the side network. The backbone and side networks are connected via a gated ladder connection, consisting of an average pooling (AP) layer, a 2D convolution with a 1×1 kernel, and a batch normalization (BN) layer. This connection adapts the intermediate features learned by the backbone network to facilitate their entry into the side network. During transfer learning, both the backbone and side networks are trained simultaneously on a small number of target domain training samples. At the parameter level, since the ladder network is trained as the side network while the backbone network is frozen, backpropagation during parameter training only calculates and updates all parameters of the small side network, without consuming more computing resources to calculate and update the parameters of the large backbone network. In this way, the goal is to save computing resources and effectively utilize the prior knowledge learned from the rich source domain data to achieve better knowledge transfer effects, effectively suppress the negative transfer phenomenon, and improve the classification effect of the network in the cross-scene classification task of hyperspectral remote sensing images. The intermediate features extracted by the backbone network will be transmitted to the side network through the gated trapezoidal connection and the intermediate features extracted by the side network for fusion operation. Specifically, the fusion operation is performed by setting a learnable parameter to perform element-wise multiplication with the matching feature and adding it to the intermediate features extracted by the side network. The intermediate data features of the backbone network will be used as supplementary features to generate fused intermediate features with the intermediate data features of the side network module, and the fused intermediate features will be used as the input of the next module of the side network and the formula for training the gated trapezoidal connection is as follows: in is a 2D convolutional layer with a kernel size of 1×1, The L+1th layer intermediate features are extracted when the backbone network that has learned the prior knowledge of the source domain processes the target domain training samples. are the corresponding convolution kernel weights and bias parameters, is the Lth layer target domain intermediate feature extracted by the side network, The L+1th layer input feature after the intermediate features of the backbone network that match the side network dimension are transformed by the gated ladder connection and the intermediate features of the side network are fused, μ L+1 is the learnable parameter of the L+1th layer of the backbone network.

[0065] T6. Calculate the loss and update the side network parameters based on the loss to obtain the optimal classification model.

[0066] In order to verify the effect of the present invention, experiments were conducted using existing data. Figure 3 The following figure shows the experimental results of the method described in the present invention on the open source hyperspectral feature dataset PaviaU. It can be seen that the feature categories are well distinguished. The classification effect of the present invention can be further demonstrated through comparative experiments. On the PaviaU dataset, the method was compared with several existing methods, including the non-transfer methods MSCDC and DEA-net and the transfer methods DCFSL, Tri-Former, and Lwnet. The overall accuracy OA, average accuracy AA, and Kappa coefficient were calculated. The larger the values of these three indicators, the better the classification effect. Table 1 shows the classification results of different methods on the PaviaU dataset.

[0067] Table 1 Comparison of various methods on the PaviaU dataset

[0068] As can be seen from Table 1, the method of the present invention achieves the highest accuracy on this dataset and has stronger classification ability.

[0069] The present invention further provides a hyperspectral remote sensing cross-scene classification system based on multi-view differential convolution and side network migration strategy, which is used to implement the above-mentioned hyperspectral remote sensing cross-scene classification method and system based on multi-view differential convolution and side network migration strategy. The system includes a data acquisition module and a model operation module.

[0070] The data acquisition module is used to acquire hyperspectral remote sensing images and perform preprocessing to obtain the data to be detected;

[0071] The model running module is used to run the pre-trained classification model to identify and classify the objects to be detected.

[0072] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy, characterized by: The steps include: Acquire hyperspectral remote sensing images and perform preprocessing to obtain data to be detected; The data to be detected is input into a pre-trained classification model for identification and classification. The classification model includes a backbone network and a side network. Both the backbone network and the side network include a feature extraction layer, a multi-view feature enhancement layer, and a result output layer. The multi-view feature enhancement layer includes multiple multi-view differential convolution modules. The side network is trained by the backbone network through the side network migration strategy. The backbone network and the side network are connected to each other through a gated connection layer. The method for the classification model to identify and classify the data to be detected includes: The feature extraction layer of the side network and the backbone network is used to process the data to be detected to obtain the basic detection features; The multi-view feature enhancement layer of the backbone network is used to process the basic detection features to obtain the intermediate detection features; The gated connection layer is used to fuse the detection intermediate features of the side network and the backbone network; The detection fusion features are processed using the multi-view feature enhancement layer of the side network to obtain the classification fusion features; The result output layer of the side network is used to classify the classification fusion features to obtain the classification results and output them.

2. The hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy according to claim 1 is characterized in that: The method of training the classification model includes: Obtain source domain image data and target domain image data, and perform dimensionality reduction through principal component analysis to obtain low-dimensional image data; Randomly classify low-dimensional image data to obtain sample data and initialize the classification model; Train the classification model in the source domain based on sample data; Update and train the classification model based on the source domain loss; Use the side network transfer strategy to fine-tune the classification model on the target domain data and train the side network; Calculate the loss and update the side network parameters according to the loss to obtain the best classification model.

3. The hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy according to claim 2 is characterized in that: The method of obtaining source domain image data and target domain image data and performing dimensionality reduction through principal component analysis to obtain low-dimensional image data includes: Read source domain image data X S1 and target domain image data X T1 ; For the source domain image data X S1 and target domain image data X T1 Centralized processing; Calculate the source domain image data X after centralized processing S1 and target domain image data X T1 The covariance matrix of The covariance matrix is decomposed into eigenvalues in multiple dimensions, and the eigenvalues are arranged in descending order; multiple larger eigenvalues are selected from all eigenvalues, and the eigenvectors of the selected eigenvalues are used to form a transformation matrix; based on the transformation matrix, the source domain image data X is transformed into S1 and target domain image data X T1 Processing is performed to obtain low-dimensional image data, which includes low-dimensional source domain data X S2 ∈R m×h×w and low-dimensional target domain data X T2 ∈R m×h×w , where m is the spectral dimension, h and w are the spatial dimensions of the low-dimensional data.

4. The hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy according to claim 3 is characterized in that: Methods for randomly classifying low-dimensional image data to obtain sample data include: Calculate the number of samples corresponding to each category based on the number of low-dimensional image data and the number of categories; Based on the number of samples, the low-dimensional image data is randomly classified to obtain sample data, which includes the source domain trainable samples X S ∈R N×S and target domain trainable samples X T ∈R N×S , where N is the number of batches and S is the feature size.

5. The hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy as described in claim 4 is characterized in that: Methods for training classification models in the source domain based on sample data include: The source domain trainable samples X in the sample data S Input to the feature embedding layer of the classification model for processing to obtain the first layer input X of the multi-view differential convolution module 1 ∈R N×S , the processing method is: Among them, Maxpool 1×1 It is a maximum pooling operation with a kernel size of 1×1; The first layer input of the multi-view differential convolution module is X 1 ∈R N×S Multiple multi-view differential convolutions are used in sequence to extract diverse features, and basic features are extracted through ordinary convolution. Multi-view differential convolutions include pixel differential convolution (CPDC) based on center difference, pixel differential convolution (APDC) based on angle difference, and pixel differential convolution (RPDC) based on radial difference. The extracted diverse features are input into the multi-view interactive self-attention module, which is guided by the basic features and outputs the L-th layer feature O L ∈R N×S .

6. The hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy according to claim 5 is characterized in that: The extracted basic features and diverse features are: Among them, G, G CPDC , G APDC , G RPDC are the convolution kernel parameters corresponding to ordinary convolution, APDC, CPDC and RPDC respectively, while B, B CPDC 、B APDC 、B RPDC Indicates the corresponding bias parameters, and the convolution kernel size used is 3×3; Guided by basic features and outputting the L-th layer feature O L ∈R N×S The methods include: Use three 1×1 2D convolutions and Get the L-th layer query vector Q extracted from the basic features L , the L-th layer key vector extracted from diverse features and the L-th layer numerical vector The formula is: Query vector Q L With key vector Do matrix multiplication and combine with query vector Q L Addition further extracts data features; Use the softmax function to calculate the attention matrix of the obtained data features from three different pixel relationships and add them together: Where c, i, and j represent the cth channel, ith row, and jth column respectively; The output O is obtained by adding the multiple attention matrices and connecting them with residuals. L : in Represents the attention score of the L-th layer multi-view differential convolution module CPDC, APDC, and RPDC, X is the basic feature of the input multi-view interactive self-attention mechanism, L It is the input of the L-th layer multi-view differential convolution module.

7. The hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy according to claim 1 is characterized in that: The method for calculating the source domain loss is: Among them, L CEL is the cross entropy loss result, Indicates the predicted value of the i-th sample in the j-th category in the one-hot form, Y ij is the true value of the i-th sample in the j-th category.

8. The hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy according to claim 2 is characterized in that: Methods for using the side network migration strategy to fine-tune the classification model on target domain data and train the side network include: Freeze the trained backbone network in the classification model at the parameter level; By introducing the side network, we can evaluate parameters based on the channel dimension and the backbone network parameters to extract effective parameters, and achieve adaptive migration of some parameters from the backbone network to the side network. The backbone network and the side network are connected through gated ladder connections.

9. Hyperspectral remote sensing cross-scene classification system based on multi-view differential convolution and side network migration strategy, characterized by: A system for implementing a hyperspectral remote sensing cross-scene classification method based on multi-view differential convolution and side network migration strategy as described in any one of claims 1 to 8, comprising: The data acquisition module is used to acquire hyperspectral remote sensing images and perform preprocessing to obtain the data to be detected; The model running module is used to run the pre-trained classification model to identify and classify the objects to be detected.

Images