Hyperspectral remote sensing image classification method and device, and storage medium

Abstract

The invention provides a hyperspectral remote sensing image classification method and device and a storage medium, and the method comprises the steps: obtaining a source domain data set and a target domain data set of a to-be-detected hyperspectral remote sensing image, and enabling the sample size of the source domain data set to be larger than that of the target domain data set; performing multi-scale center sample information enhancement processing on the source domain data set and the target domain data set to obtain a first multi-scale center sample set and a second multi-scale center sample set; the first multi-scale center sample set is input into a classification model for pre-training, the classification model comprises a branch fusion network and is used for executing feature fusion processing, and fused spatial spectrum features are output; and a pre-trained classification model is migrated to the second multi-scale center sample set, lightweight fine tuning processing is executed, and the classification model outputs a final target classification result. According to the method, an excellent classification effect is achieved under the condition that the parameter quantity is small, and the classification accuracy of the hyperspectral remote sensing image is improved.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image classification and relates to a hyperspectral remote sensing image classification method, device and storage medium. Background Art

[0002] Hyperspectral images cover hundreds of narrow and continuous spectral bands, and each pixel contains rich spatial and spectral features of ground objects. Hyperspectral image classification aims to classify each pixel according to these features, which has important application value in military and civilian fields.

[0003] According to the different ways of feature acquisition, the classification methods of hyperspectral images can be roughly divided into two categories: methods based on manual feature extraction and methods based on deep learning. The method based on manual feature extraction extracts discriminative features from the data through artificial design. This method requires more human intervention and experience accumulation and is not suitable for processing the growing massive remote sensing data. The classification method based on deep learning can learn the potential information representation from the data and automatically obtain complex and abstract features. As one of the commonly used classification methods, the classification method based on spatial spectrum features uses three-dimensional cube blocks as input and makes full use of the structural characteristics of hyperspectral images, so it has been widely used.

[0004] However, the existing hyperspectral classification methods based on spatial-spectral features still have the following problems: First, in the classification methods based on spatial-spectral features, in order to make full use of the spatial-spectral information of the hyperspectral image, a three-dimensional cube block is used as a sample, and the label of the central pixel is used as the sample label of the entire three-dimensional cube block. In the classification process, the prediction result is determined by all the pixels in the three-dimensional cube block. Therefore, not all pixels contribute equally to the classification result, and the contribution rate of the central pixel and its neighboring area is the largest. However, the existing hyperspectral classification methods based on spatial-spectral features generally have the problem of insufficient attention to the central sample. Second, the classification method based on spatial-spectral features uses a three-dimensional cube block as input, which requires more parameters and calculations than the classification method based on spectral or spatial features. In addition, with the development of technology, the design of the hyperspectral classification network structure is becoming increasingly complex. How to achieve satisfactory classification results under lightweight conditions is an urgent problem to be solved. Third, the current hyperspectral classification method needs to train the model from scratch for different classification scenarios, which consumes a lot of computing resources. Therefore, the existing hyperspectral classification methods based on spatial spectrum features have defects such as insufficient attention to central samples, large amount of computational data, and high complexity, which lead to poor image classification accuracy, low accuracy, and poor adaptability in different scenarios. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a hyperspectral remote sensing image classification method, device, and storage medium, which can efficiently communicate the source domain and the target domain, achieve excellent classification effect with a small number of parameters, and improve the classification accuracy of hyperspectral remote sensing images.

[0006] In order to achieve the above object, the present invention provides a hyperspectral remote sensing image classification method, comprising:

[0007] A source domain data set and a target domain data set of a hyperspectral remote sensing image to be tested are obtained, wherein the sample size of the source domain data set is larger than that of the target domain data set;

[0008] Performing multi-scale center sample information enhancement processing on the source domain data set and the target domain data set respectively to obtain a first multi-scale center sample set and a second multi-scale center sample set;

[0009] Inputting the first multi-scale center sample set into a classification model for pre-training, the classification model comprising a branch fusion network for performing feature fusion processing and outputting fused spatial spectrum features;

[0010] The pre-trained classification model is migrated to the second multi-scale central sample set, and a lightweight fine-tuning process is performed, and the classification model outputs a final target classification result.

[0011] In some embodiments, a three-dimensional block model is performed on the source domain data set to construct a first three-dimensional cube sample, and a position of a center point of the first three-dimensional cube sample is calculated;

[0012] At different scales, the upper and lower boundary positions of the central area corresponding to each scale are determined according to the position of the center point and the given spatial dimension;

[0013] Extract central samples in the central area corresponding to each scale to obtain the first multi-scale central sample set; and / or,

[0014] Modeling a three-dimensional block of the target domain data set, constructing a second three-dimensional cube sample, and calculating the position of the center point of the second three-dimensional cube sample;

[0015] At different scales, the upper and lower boundary positions of the central area corresponding to each scale are determined according to the position of the center point and the given spatial dimension;

[0016] A central sample is extracted from the central area corresponding to each scale to obtain the second multi-scale central sample set.

[0017] In some embodiments, the branch fusion network includes a plurality of parallel curl branch modules and a feature aggregation module connected to the output ends of the plurality of curl branch modules; the first multi-scale center sample set or the second multi-scale center sample set is input into the plurality of curl branch modules to obtain multi-scale spatial spectrum features; the multi-scale spatial spectrum features are input into the feature aggregation module for feature fusion to obtain fused spatial spectrum features;

[0018] Each of the convolution branch modules includes: a 3D convolution unit and a Transformer unit, and different convolution branch modules use different numbers of 3D convolution units.

[0019] In some embodiments, the 3D convolution unit includes a spatial convolution block, a spectral convolution block and a convolution aggregation block cascaded in sequence, wherein the spatial convolution block and the spectral convolution block both use residual connection as the main structure, and both use 3D convolution kernels of different sizes; the convolution aggregation block is used for feature aggregation;

[0020] Transformer unit, using the Transformer encoder structure;

[0021] The feature aggregation module includes a cross-attention unit, a fully connected unit and an addition unit which are cascaded in sequence, wherein the cross-attention unit captures the correlation between multi-scale spatial-spectral features from different branches to obtain shallow fusion features; the fully connected unit maps the shallow fusion features to a new feature space; the addition unit performs deep feature fusion on the new feature space through addition operations to generate the final fused spatial-spectral features.

[0022] In some embodiments, performing lightweight processing includes:

[0023] The second multi-scale center sample set is adaptively feature learned in an iterative manner from shallow to deep according to given shallow prompt words and deep prompt words, and only the prompt words initialized to zero vectors are adjusted.

[0024] In some embodiments, performing lightweight processing further includes: adding a classification network to the pre-trained classification model to perform low-rank adaptation processing on the second multi-scale center sample set;

[0025] The classification network includes a linear layer and a three-dimensional convolutional layer, and a rank decomposition matrix is ​​added to both the linear layer and the three-dimensional convolutional layer;

[0026] And the linear layer adds the first rank decomposition matrix to update the weight of the linear layer through low rank decomposition;

[0027] The three-dimensional convolutional layer adds a second rank decomposition matrix to update the weights of the three-dimensional convolutional layer through low-rank decomposition, and during the low-rank decomposition process, the dimension of the second rank decomposition matrix is ​​adjusted to be the same as the dimension of the weights of the three-dimensional convolutional layer before updating.

[0028] In some embodiments, the classification model uses the cross entropy loss of the source domain dataset combined with the KL divergence loss of the adversarial sample to construct a target loss function.

[0029] In some embodiments, wherein:

[0030] Cross entropy loss of source domain dataset X It is expressed as:

[0031]

[0032] Among them, N is the total number of samples, and the predicted value of the i-th source domain dataset is The true label Y i ;

[0033] KL divergence loss of adversarial sample X' corresponding to source domain dataset X It is expressed as:

[0034]

[0035] Among them, when the total number of samples is N, the predicted value of the i-th adversarial sample is With the true label Y' i ;

[0036] The objective loss function is expressed as:

[0037]

[0038] in, is the target loss function.

[0039] Another aspect of the present invention also discloses a hyperspectral remote sensing image classification device, comprising:

[0040] A data set construction module is used to obtain a source domain data set and a target domain data set of a hyperspectral remote sensing image to be tested, wherein the sample size of the source domain data set is larger than that of the target domain data set;

[0041] An enhancement processing module is used to perform multi-scale center sample information enhancement processing on the source domain data set and the target domain data set respectively to obtain a first multi-scale center sample set and a second multi-scale center sample set;

[0042] A pre-training module, used for inputting the first multi-scale center sample set into a classification model for pre-training, wherein the classification model comprises a branch fusion network for performing feature fusion processing and outputting fused spatial spectrum features;

[0043] The fine-tuning module is used to migrate the pre-trained classification model to the second multi-scale central sample set and perform lightweight fine-tuning processing, and the classification model outputs the final target classification result.

[0044] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the above-mentioned hyperspectral remote sensing image classification method when executed by a processor.

[0045] It can be seen from the above scheme that the advantages of the present invention are:

[0046] The hyperspectral remote sensing image classification method disclosed by the present invention can efficiently communicate the source domain and the target domain, achieve excellent classification effect with a small number of parameters and a small amount of calculation, and improve the classification accuracy of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of the overall process of a hyperspectral remote sensing image classification method provided by an embodiment of the present invention;

[0048] Figure 2 It is a schematic diagram of the principle of the hyperspectral remote sensing image classification method;

[0049] Figure 3 This is a schematic diagram of the principle of multi-scale center sample information enhancement;

[0050] Figure 4 It is a schematic diagram of the branch fusion structure;

[0051] Figure 5 Schematic diagram of the structure of the Convolution Branch module, where (a) represents a 3D convolution unit and (b) represents a Transformer unit.

[0052] Figure 6 This is a schematic diagram of the tuning principle of progressive zero initialization prompt;

[0053] Figure 7 Schematic diagram of the principle of low-rank adaptation technology for hyperspectral images, where (a) represents Linear-LoRA and (b) represents 3DCNN-LoRA;

[0054] Figure 8 A comparison chart showing the results of the method of the present invention on the Pavia University dataset and different classification methods of related technologies;

[0055] Fig. 9 A comparison chart showing the results of the method of the present invention on the Houston 2013 dataset and different classification methods of related technologies;

[0056] Fig.10 A comparison chart showing the results of the method of the present invention on the Indian Pines dataset with different classification methods of related technologies;

[0057] Fig.11 A comparison chart showing the results of the method of the present invention on the Botswana data set and different classification methods of related technologies;

[0058] Fig.12 A schematic diagram of the structure of a hyperspectral remote sensing image classification device provided by an embodiment of the present invention;

[0059] in:

[0060] 200: Hyperspectral remote sensing image classification device;

[0061] 210: Dataset construction module;

[0062] 220: Enhanced processing module;

[0063] 230: pre-training module;

[0064] 240: Fine-tuning module. DETAILED DESCRIPTION

[0065] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0066] See also Figure 1-Figure 11 , Figure 1 A schematic diagram of the overall process of a hyperspectral remote sensing image classification method provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of the principle of the hyperspectral remote sensing image classification method; Figure 3 This is a schematic diagram of the principle of multi-scale center sample information enhancement; Figure 4 It is a schematic diagram of the branch fusion structure; Figure 5 Schematic diagram of the structure of the CFormerBranch module, where (a) represents a 3D convolution unit and (b) represents a Transformer unit; Figure 6 This is a schematic diagram of the tuning principle of progressive zero initialization prompt; Figure 7 Schematic diagram of the principle of low-rank adaptation technology for hyperspectral images, where (a) represents Linear-LoRA and (b) represents 3DCNN-LoRA; Figure 8 A comparison chart showing the results of the method of the present invention on the Pavia University dataset with different classification methods of related technologies; Fig. 9 A comparison chart showing the results of the method of the present invention on the Houston 2013 dataset and different classification methods of related technologies; Fig.10 It shows the result comparison diagram of the method of the present invention and different classification methods of related technologies on the Indian Pines dataset; Fig.11 It shows the result comparison diagram of the method of the present invention and different classification methods of related technologies on the Botswana dataset.

[0067] Please refer to Figure 1 、 Figure 2 As shown in

[0068] Step S1, obtain the source domain dataset and the target domain dataset of the hyperspectral remote sensing image to be measured, and the sample size of the source domain dataset is larger than that of the target domain dataset.

[0069] Step S2, perform multi-scale central sample information enhancement processing on the source domain dataset and the target domain dataset respectively to obtain a first multi-scale central sample set and a second multi-scale central sample set.

[0070] Figure 3 It is a schematic diagram of the multi-scale central sample information enhancement strategy. In the previous methods for classifying hyperspectral remote sensing images based on 3DCNN, in order to effectively utilize the spatial-spectral information, most methods use the method of taking the central pixel as the label of the entire 3D sample for classification. However, in the subsequent classification process, it is the predicted value obtained by the entire 3D sample passing through the network that is used to calculate the loss with the label of the 3D sample to complete the backpropagation process. Therefore, in space, not all samples contribute equally to the final classification result. Among them, the central pixel makes the greatest contribution. In this case, in this embodiment, a method of multi-scale central sample information enhancement is proposed to solve the above problems.

[0071] Specifically, in some embodiments, model the source domain dataset into three-dimensional cubic blocks to construct the first three-dimensional cube sample, and calculate the position of the center point of the first three-dimensional cube sample For a sample of the source domain dataset where B represents the spectral dimension, and P represents the length and width of the sample respectively. Then, at different scales, according to the position C of the center point and the given spatial dimension M i (M i <P), determine the upper and lower boundary positions of the central region corresponding to each scale, where the upper boundary position The lower boundary position is and extract central samples in the central region corresponding to each scale to obtain the first multi-scale central sample set

[0072] In some embodiments, the target domain data set is modeled as a three-dimensional block, a second three-dimensional cube sample is constructed, and the position of the center point of the second three-dimensional cube sample is calculated. Then, at different scales, the upper and lower boundary positions of the central area corresponding to each scale are determined based on the position of the center point and the given spatial dimension. The central sample is extracted from the central area corresponding to each scale to obtain the second multi-scale central sample set. In addition, the various calculation formulas for multi-scale central sample information enhancement of the target domain data set are similar to those of the source domain data set, and will not be repeated here.

[0073] In this embodiment, after the multi-scale center samples are extracted, they pass through a branch fusion network composed of different branches to enhance the focus on the center pixel.

[0074] Step S3: input the first multi-scale center sample set into a classification model for pre-training. The classification model includes a branch fusion network for performing feature fusion processing and outputting fused spatial spectrum features.

[0075] Figure 4 Schematic diagram of branch fusion structure. In order to effectively obtain the spatial spectrum features of the hyperspectral remote sensing image, in some embodiments, the branch fusion network includes a plurality of parallel curl branch modules (CFormer Branch) and a feature aggregation module (Cross Fusion) connected to the output ends of the plurality of curl branch modules.

[0076] During the pre-training stage of the classification model, the branch fusion network receives the first multi-scale central sample set and inputs it into the multiple convolution branch modules to obtain multi-scale spatial spectral features. Different convolution branch modules output spatial spectral features of different scales. Then, the multi-scale spatial spectral features are input into the feature aggregation module for feature fusion to obtain the fused spatial spectral features corresponding to the first multi-scale central sample set.

[0077] Each of the convolution branch modules includes: a 3D convolution unit (3D Conv) and a Transformer unit, and different convolution branch modules use different numbers of 3D convolution units. As the spatial dimension of the multi-sample increases, the number of 3D convolution units also gradually increases.

[0078] Figure 5 This is the structure diagram of the volume transformation branch module (CFormerBranch), where Figure 5Figure (a) represents a 3D convolution unit, and figure (b) represents a Transformer unit. In some embodiments, the 3D convolution unit includes a spatial convolution block (Spatial Conv Block), a spectral convolution block (Spectral Conv Block) and a convolution aggregation block (ConvAggregation Block) cascaded in sequence, wherein the spatial convolution block and the spectral convolution block both use residual connection as the main structure, and are composed of a series of operations including 3D convolution, batch normalization, GeLU activation function, and Dropout. Both use 3D convolution kernels of different sizes, the spatial convolution block uses a convolution kernel size of 1×3×3, and the spectral convolution block uses a convolution kernel size of 3×1×1. The convolution aggregation block is composed of a series of operations including 3D convolution (3D Conv), batch normalization (BN), GeLU activation function, and Dropout, and is used for feature aggregation. The Transformer unit adopts the traditional Transformer encoder structure, which is mainly composed of layer normalization (layerNorm), multi-head attention mechanism (Multi-HeadAttention), and fully connected layer (MLP).

[0079] The feature aggregation module (Cross Fusion) includes a cross attention unit (CrossAttention), a fully connected unit (MLP) and an addition unit, wherein the cross attention unit captures the correlation between multi-scale spatial-spectral features from different branches to obtain shallow fusion features; the fully connected unit maps the shallow fusion features to a new feature space; the addition unit performs deep feature fusion on the new feature space through addition operations to generate the final fused spatial-spectral features.

[0080] In addition, in the classification model fine-tuning stage, the branch fusion network receives the second multi-scale center sample set, inputs it into the multiple CFormer Branch modules, obtains multi-scale spatial spectrum features, and different CFormer Branch modules output spatial spectrum features of different scales; then, the multi-scale spatial spectrum features are input into the feature aggregation module for feature fusion, and the fused spatial spectrum features corresponding to the second multi-scale center sample set are obtained. This process is similar to the feature fusion process of the first multi-scale center sample set mentioned above, and this embodiment will not be described in detail.

[0081] Step S4: Migrate the pre-trained classification model to the second multi-scale central sample set, perform lightweight fine-tuning processing, and the classification model outputs the final target classification result.

[0082] In this embodiment, after pre-training the classification model in step S3, the pre-trained classification model is further migrated to the second multi-scale center sample set to perform lightweight fine-tuning. In the fine-tuning stage of the target domain, the present invention proposes progressive zero-initialization hint tuning and hyperspectral image low-rank adaptation technology for lightweight fine-tuning.

[0083] Figure 6 Schematic diagram of progressive zero-initialization prompt tuning. Specifically, progressive zero-initialization prompt tuning uses a small number of trainable parameters to perform adaptive feature learning in a progressive manner. When fine-tuning using this strategy, shallow prompt words and deep prompt words are constructed and initialized. In the specific adjustment process, only shallow prompt words and deep prompt words initialized to zero vectors are selected, while other parts of the model remain frozen. In this embodiment, shallow prompt words are constructed. Deep Tips The overall prompt words for construction are: d represents the dimension information. The second multi-scale center sample set is adaptively learned in an iterative manner from shallow to deep according to the given shallow prompt words and deep prompt words, and only the prompt words initialized to zero vectors are adjusted, while the other parts of the model remain frozen, and the output corresponding to the shallow prompt words is obtained. It can be expressed as: Among them, [·,·] represents the connection operation. Similarly, the output corresponding to the deep prompt word is It can be expressed as:

[0084] In this embodiment, a progressive zero-initialized hint fine-tuning strategy is adopted, iterating from shallow to deep layers, which effectively reduces computational overhead, improves training efficiency, and improves model stability. In addition, initializing the hint to a zero matrix not only saves parameter space, but also simplifies computational complexity.

[0085] In addition, in one embodiment, a classification network is added to the classification model after pre-training to perform low-rank adaptation processing on the second multi-scale center sample set. At this time, the classification model includes two parts: a branch fusion network and a classification network. The target domain data set is subjected to multi-scale center sample information enhancement processing to obtain a second multi-scale center sample set, and then the second multi-scale center sample set is subjected to the above-mentioned progressive zero-initialization prompt tuning processing, and then respectively input into the branch fusion network and the classification network for feature fusion and low-rank adaptation processing. Among them, the processing process of the branch fusion network is described in the above step S3, and will not be described in detail here. In this embodiment, the low-rank adaptation processing process of the classification network is described in detail.

[0086] Specific reference Figure 7 As shown in Figure 7 A schematic diagram of the low-rank adaptation technique for hyperspectral images is shown, where Figure 7 Figure (a) shows that the linear layer adds a rank decomposition matrix to represent the Linear-LoRA layer; Figure (b) shows that the three-dimensional convolutional layer adds a rank decomposition matrix to represent the 3DCNN-LoRA layer. The classification network contains a linear layer and a three-dimensional convolutional layer, and the linear layer and the three-dimensional convolutional layer are added with a rank decomposition matrix. During the training process, under the premise of freezing the pre-trained classification model, these rank decomposition matrices are used as optimization parameters, and only these rank decomposition matrices are optimized, thereby reducing the number of parameters in the target domain task training process.

[0087] Among them, the linear layer adds the first rank decomposition matrix {A L ,B L}Update the weights of the linear layer through low-rank decomposition. Specifically, for the linear layer, let is the input tensor, is the output tensor, is the weight of the linear layer, then the calculation formula of the original linear layer can be expressed as: Y = XW. Linear-LoRA updates the original weight W through low-rank decomposition to

[0088]

[0089] Among them, W is frozen during the training process, Represents the low-rank matrix of trainable parameters for the linear layer, which is initialized using the Kaiming normal distribution; Indicates that an all-zero matrix is ​​used for the linear layer; where r < < min (C I ,C O ), represents the scaling ratio, and α represents a constant, which defaults to 1. After adding the first rank decomposition matrix, the expression of the linear layer forward propagation is updated to:

[0090]

[0091] Similarly, for the three-dimensional convolutional layer, the three-dimensional convolutional layer adds a second rank decomposition matrix {A C ,B C The weights of the three-dimensional convolutional layer are updated through low-rank decomposition, and during the low-rank decomposition process, the dimension of the second rank decomposition matrix is ​​adjusted to be the same as the dimension of the weights of the three-dimensional convolutional layer before updating.

[0092] Specifically, is the input tensor, is the output tensor, is the weight of the 3D convolutional layer, K b , K h , Kw Represents the index of the convolution kernel spectral dimension, height, and width, then the calculation formula of the original three-dimensional convolution can be expressed as:

[0093]

[0094] Through 3DCNN-LoRA, low-rank decomposition updates the original weight W to

[0095]

[0096] Likewise, the original weights W are frozen during training. Represents the low-rank matrix of trainable parameters for 3DCNN, which is initialized using the Kaiming normal distribution; Indicates the use of an all-zero matrix for 3DCNN, where r C <<min(C I ,C O ), Indicates the scaling ratio, α indicates a constant, which defaults to 1 here. Compared with Linear-LoRA, 3DCNN-LoRA has one more dimension adjustment operation. In the low-rank decomposition process, the dimension of the second-rank decomposition matrix is ​​adjusted to be the same as the dimension of the weight of the three-dimensional convolutional layer before the update, that is, A C B C Adjust to the same dimension size as the original weight W of the 3D convolutional layer. After 3DCNN-LoRA, the expression of forward propagation is updated as:

[0097]

[0098] In summary, in this embodiment, based on the low-rank adaptation technology of hyperspectral images, a classification network is added to the classification model, and specifically, a rank decomposition matrix is ​​added to the linear layer and the three-dimensional convolutional layer of the classification network to perform low-rank adaptation processing on the second multi-scale center sample set. During the training process, only these rank decomposition matrices are optimized, thereby reducing the amount of parameters in the target domain task training process. By combining Linear-LoRA and 3DCNN-LoRA, it fully considers the structural characteristics of hyperspectral remote sensing images and can give full play to the advantages of hyperspectral remote sensing image classification models. It can significantly reduce storage requirements and task overhead by projecting the linear layer and convolutional layer weights of the hyperspectral remote sensing image classification model to a smaller subspace, effectively reducing computational complexity.

[0099] In addition, in some embodiments, see Figure 2As shown in , the source domain dataset is enhanced by multi-scale center sample information, and the first multi-scale center sample set is input into the branch fusion network. The information is aggregated at different scales around the center position by multiple branches, which enhances the focus on the center pixel. Then, the initial classification result obtained by pre-training the classification model with the source domain dataset is obtained by optimizing the cross entropy loss. Specifically, let the input source domain dataset be X S Indicates that the cross entropy loss function is used Represents the cross entropy loss of the source domain dataset It is expressed as: Among them, N is the total number of samples, and the predicted value of the i-th sample in the source domain dataset is The true label Y i . Then the optimization formula of the source domain can be expressed as:

[0100] For the tuning stage of the target domain, the classification model uses the cross entropy loss of the source domain dataset combined with the KL divergence loss of the adversarial sample to construct the target loss function. At this time, the KL divergence loss of the adversarial sample X' corresponding to the sample X of the source domain dataset is It is expressed as:

[0101]

[0102] Among them, when the total number of samples is N, the predicted value of the i-th adversarial sample is With the true label Y' i ; represents the KL divergence loss designed for adversarial samples X', which aims to enhance the robustness of the model by learning the distribution characteristics of these samples.

[0103] The objective loss function It is expressed as: Then the optimization formula of the target domain is: Among them, |Φ|+|Ψ|<<|Θ|, which means that the number of parameters during fine-tuning in the target domain is much smaller than the number of parameters during pre-training in the source domain.

[0104] In addition, in this embodiment, the fast gradient sign method is used to generate adversarial samples. This method determines the perturbation by calculating the gradient of the loss function relative to the input sample, as shown in the following formula:

[0105]

[0106] Where X' represents the adversarial sample, ∈ is the amplitude of the perturbation, sign(·) represents the sign function of the gradient, Represents the loss function The gradient of sample X of the source domain dataset. By calculating the gradient of the loss function with respect to the input sample and generating adversarial perturbations based on the sign of the gradient, adversarial samples can be generated efficiently and quickly. The use of adversarial samples can enhance the model's ability to resist adversarial attacks and effectively improve its robustness and generalization capabilities. In summary, the hyperspectral adversarial robustness loss enables the model to quickly adapt to the data distribution of the target domain, thereby ensuring strong generalization performance even in the case of domain shift.

[0107] The effect of the present invention is verified by comparing the method of the present invention with related technologies.

[0108] in, Figure 8-Figure 11 The figure is a comparison chart of the results of the method of the present invention on the Pavia University, Houston 2013, Indian Pines, and Botswana datasets with different classification methods using the same technology. Figure 8 A comparison chart showing the results of the method of the present invention on the Pavia University dataset with different classification methods of related technologies; Fig. 9 A comparison chart showing the results of the method of the present invention on the Houston 2013 dataset and different classification methods of related technologies; Fig.10 A comparison chart showing the results of the method of the present invention on the Indian Pines dataset with different classification methods of related technologies; Fig.11 The result comparison chart of the method of the present invention on the Botswana dataset and different classification methods of related technologies is shown. The bold font in the figure is the optimal result. In order to verify the performance advantage of the method of the present invention, it is compared with related technologies of the prior art. These related technology comparison methods are such as HybridSN, FSKNet, S3EResBOF, FuSENet, MSDN, SSRN, SSFTT, HSI-BERT. In order to make a fair comparison, each method is trained with 10% of the training samples. The evaluation indicators used in the present invention for evaluating the accuracy of the model are: average accuracy (OA), overall accuracy (AA), Kappa coefficient (κ); the evaluation indicators used for evaluating the size of the model are: parameter amount (Param), calculation amount (FLOPs), and test time (TT). It can be seen from the figure that from the perspective of model accuracy, compared with other deep learning comparison methods, the present invention has the highest average accuracy, overall accuracy, and Kappa coefficient under the condition of minimum parameter amount, calculation amount, and test time, reaching the leading level. In summary, the present invention achieves leading classification performance in both the source domain and the target domain under the premise of lightweight design.

[0109] In summary, the hyperspectral remote sensing image classification method disclosed in the present invention can efficiently communicate the source domain and the target domain, achieve excellent classification results with a small number of parameters and a small amount of calculation, and improve the classification accuracy of hyperspectral images. Specifically:

[0110] First, the present invention proposes a branch fusion network based on a multi-scale central sample information enhancement strategy. By superimposing multiple branches composed of inputs of different scales, the focus on central samples is enhanced, thereby improving the classification ability of the model.

[0111] Second: for the target domain dataset, the number of model parameters and computational complexity are reduced through progressive zero-initialization hint tuning and hyperspectral image low-rank adaptation technology. Among them, the progressive zero-initialization hint tuning strategy, under the premise of lightweight, adopts a progressive iterative method to perform adaptive learning of features from shallow to deep. The hyperspectral image low-rank adaptation technology consists of Linear-LoRA and 3DCNN-LoRA, which can effectively reduce the computational complexity while ensuring the accuracy of the model based on the characteristics of hyperspectral remote sensing images.

[0112] Third: The cross entropy loss calculated by the source domain dataset and the KL divergence loss calculated by the adversarial sample constitute the target loss function of the classification model, which aims to balance the classification accuracy of the model and the robustness of adversarial attacks. It can improve the generalization ability for different datasets while ensuring the accuracy of the model.

[0113] In addition, it should be noted that the hyperspectral remote sensing image classification method disclosed in the present invention is not limited to being executed in the order of the above steps S1-S4. It can reorder, add or delete steps. As long as the expected results of the technical solution of the present invention can be achieved, the present invention is not limited here.

[0114] In addition, corresponding to the embodiment of the aforementioned hyperspectral remote sensing image classification method, the present invention also provides an embodiment of an inference acceleration device of an embedded edge intelligent computing system.

[0115] See also Fig.12 , Fig.12 The schematic diagram of the structure of the hyperspectral remote sensing image classification device is shown. The hyperspectral remote sensing image classification device 200 at least comprises:

[0116] The data set construction module 210 is used to obtain a source domain data set and a target domain data set of the hyperspectral remote sensing image to be tested, wherein the sample size of the source domain data set is larger than that of the target domain data set.

[0117] The enhancement processing module 220 is used to perform multi-scale center sample information enhancement processing on the source domain data set and the target domain data set respectively to obtain a first multi-scale center sample set and a second multi-scale center sample set.

[0118] The pre-training module 230 is used to input the first multi-scale center sample set into a classification model for pre-training. The classification model includes a branch fusion network for performing feature fusion processing and outputting fused spatial spectrum features.

[0119] The fine-tuning module 240 is used to migrate the pre-trained classification model to the second multi-scale central sample set and perform lightweight fine-tuning processing, and the classification model outputs a final target classification result.

[0120] Those skilled in the art can clearly understand that for the convenience and simplicity of describing the hyperspectral remote sensing image classification method, the specific working process of each module of the described device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0121] According to an embodiment of the present invention, the present invention also provides a machine-readable medium, which may be a tangible medium, which may contain or store a program for use by an instruction execution system, device or equipment or used in conjunction with an instruction execution system, device or equipment, and the computer program implements the steps of the above-mentioned hyperspectral remote sensing image classification method when executed by a processor.

[0122] In addition, the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A hyperspectral remote sensing image classification method, characterized in that: Include: A source domain data set and a target domain data set of a hyperspectral remote sensing image to be tested are obtained, wherein the sample size of the source domain data set is larger than that of the target domain data set; Performing multi-scale center sample information enhancement processing on the source domain data set and the target domain data set respectively to obtain a first multi-scale center sample set and a second multi-scale center sample set; Inputting the first multi-scale center sample set into a classification model for pre-training, the classification model comprising a branch fusion network for performing feature fusion processing and outputting fused spatial spectrum features; The pre-trained classification model is migrated to the second multi-scale central sample set, and a lightweight fine-tuning process is performed, and the classification model outputs a final target classification result.

2. The method according to claim 1, characterized in that Modeling a three-dimensional block of the source domain data set, constructing a first three-dimensional cube sample, and calculating the position of the center point of the first three-dimensional cube sample; At different scales, the upper and lower boundary positions of the central area corresponding to each scale are determined according to the position of the center point and the given spatial dimension; Extracting central samples in the central area corresponding to each scale to obtain the first multi-scale central sample set; and / or, Modeling a three-dimensional block of the target domain data set, constructing a second three-dimensional cube sample, and calculating the position of the center point of the second three-dimensional cube sample; At different scales, the upper and lower boundary positions of the central area corresponding to each scale are determined according to the position of the center point and the given spatial dimension; A central sample is extracted from the central area corresponding to each scale to obtain the second multi-scale central sample set.

3. The method according to claim 1 or 2, characterized in that: The branch fusion network includes a plurality of parallel curl branch modules and a feature aggregation module connected to the output ends of the plurality of curl branch modules; the first multi-scale center sample set or the second multi-scale center sample set is input into the plurality of curl branch modules to obtain multi-scale spatial spectrum features; The multi-scale spatial spectrum features are input into the feature aggregation module for feature fusion to obtain fused spatial spectrum features; Each of the convolution branch modules includes: a 3D convolution unit and a Transformer unit, and different convolution branch modules use different numbers of 3D convolution units.

4. The method according to claim 3, characterized in that in, The 3D convolution unit includes a spatial convolution block, a spectral convolution block and a convolution aggregation block which are cascaded in sequence, wherein the spatial convolution block and the spectral convolution block both use residual connection as the main structure, and both use 3D convolution kernels of different sizes; the convolution aggregation block is used for feature aggregation; Transformer unit, using the Transformer encoder structure; The feature aggregation module includes a cross-attention unit, a fully connected unit and an addition unit which are cascaded in sequence, wherein the cross-attention unit captures the correlation between multi-scale spatial-spectral features from different branches to obtain shallow fusion features; the fully connected unit maps the shallow fusion features to a new feature space; the addition unit performs deep feature fusion on the new feature space through addition operations to generate the final fused spatial-spectral features.

5. The method according to claim 1, characterized in that Perform lightweight processing, including: The second multi-scale center sample set is adaptively feature learned in an iterative manner from shallow to deep according to given shallow prompt words and deep prompt words, and only the prompt words initialized to zero vectors are adjusted.

6. The method according to claim 1 or 5, characterized in that: Performing lightweight processing also includes: adding a classification network to the pre-trained classification model to perform low-rank adaptation processing on the second multi-scale center sample set; The classification network includes a linear layer and a three-dimensional convolutional layer, and a rank decomposition matrix is ​​added to both the linear layer and the three-dimensional convolutional layer; And the linear layer adds the first rank decomposition matrix to update the weight of the linear layer through low rank decomposition; The three-dimensional convolutional layer adds a second rank decomposition matrix to update the weights of the three-dimensional convolutional layer through low-rank decomposition, and during the low-rank decomposition process, the dimension of the second rank decomposition matrix is ​​adjusted to be the same as the dimension of the weights of the three-dimensional convolutional layer before updating.

7. The method according to claim 1, characterized in that The classification model uses the cross entropy loss of the source domain dataset and the KL divergence loss of the adversarial sample to construct the target loss function.

8. The method according to claim 7, characterized in that in: Cross entropy loss of source domain dataset X It is expressed as: Among them, N is the total number of samples, and the predicted value of the i-th source domain dataset is The true label Y i ; KL divergence loss of adversarial sample X' corresponding to source domain dataset X It is expressed as: Among them, when the total number of samples is N, the predicted value of the i-th adversarial sample is With the true label Y' i ; The objective loss function is expressed as: in, is the target loss function.

9. A hyperspectral remote sensing image classification device, characterized in that: Include: A data set construction module is used to obtain a source domain data set and a target domain data set of a hyperspectral remote sensing image to be tested, wherein the sample size of the source domain data set is larger than that of the target domain data set; An enhancement processing module is used to perform multi-scale center sample information enhancement processing on the source domain data set and the target domain data set respectively to obtain a first multi-scale center sample set and a second multi-scale center sample set; A pre-training module, used for inputting the first multi-scale center sample set into a classification model for pre-training, wherein the classification model comprises a branch fusion network for performing feature fusion processing and outputting fused spatial spectrum features; The fine-tuning module is used to migrate the pre-trained classification model to the second multi-scale central sample set and perform lightweight fine-tuning processing, and the classification model outputs the final target classification result.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the hyperspectral remote sensing image classification method described in any one of claims 1 to 8 are implemented.

Images