A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method, system, device and medium

Abstract

A method, system, device and medium for detecting hyperspectral changes by integrating lightweight spatial-spectral features across time periods, the method comprising: preprocessing hyperspectral image data and dividing them into training sets and test sets; constructing a feature extraction network; constructing a classification network; using the training set, training the feature extraction network and the classification network with a gradient descent algorithm, calculating the accuracy of the training set in each iteration, and using the weight of the first generation network model with the highest accuracy on the training set as the final detection model weight to obtain a trained model; inputting the test set into the trained model for testing, obtaining a detection result, and outputting a predicted label map of the hyperspectral image data; the system, device and medium are used to implement the method; the present invention designs a model based on spatial-spectral feature extraction and cross-temporal feature fusion from the two aspects of refined feature extraction and cross-temporal feature fusion, and simplifies the model using a lightweight method, thereby realizing a lightweight hyperspectral image change detection method with good performance.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a method, system, device and medium for detecting cross-temporal lightweight spatial-spectral feature fusion hyperspectral changes. Background Art

[0002] Change detection can identify differences between multi-temporal remote sensing images and has been widely used in forestry and agricultural monitoring, natural disaster assessment, surface dynamics analysis and other fields. Hyperspectral images typically have hundreds of dimensional spectral channels and the unique property of one spectrum per object. They contain rich spectral information and have inherent advantages over multispectral images in detecting land cover changes. The main challenge of using hyperspectral images for change detection is that the high dimensionality of the images and the redundancy of the spectral information make it difficult for traditional models to fully and effectively extract spatial-spectral features. Deep learning methods pay less attention to the correlation between features between hyperspectral images across time, and cannot fully capture the relationship between the two input images to obtain change information. At the same time, they require a large computational cost.

[0003] The paper “López- In the paper "Stacked autoencoders for multiclass change detection in hyperspectral images" (IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, IEEE, 2018: 1906-1909), J. Garea AS, Heras DB, et al. used a stacked autoencoder to extract features from the difference image of dual-temporal hyperspectral images to detect changes. Compared with traditional models, this network has a deeper network structure and can more fully extract features. However, this method only considers the spectral characteristics of pixels and fails to consider spatial characteristics, resulting in minimal performance improvement.

[0004] In the paper "Song A, Choi J, Han Y, et al. Change detection in hyperspectral images using recurrent 3D fully convolutional networks[J]. Remote Sensing, 2018, 10(11):1827," a recurrent 3D unsupervised fully convolutional network framework was proposed. This framework combines the advantages of a 3D unsupervised fully convolutional network and a long short-term memory convolutional neural network to extract joint spectral and spatiotemporal features. However, this method increases computational cost and fails to fully consider information redundancy in the spectral and spatial domains.

[0005] In the paper "Wang F, Jiang M, Qian C, et al. Residual attention network for image classification [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 3156-3164," they proposed enhancing the representational capabilities of convolutional neural networks through an attention mechanism. They proposed a residual attention network with an encoder-decoder attention module to refine feature maps to improve network performance. However, generating three-dimensional attention maps in the residual attention network requires more parameters, increasing computational cost. Furthermore, the network fails to address the correlation between features across temporal hyperspectral images.

[0006] In summary, existing hyperspectral ground object change detection methods have problems such as incomplete spatial-spectral feature extraction, less attention to the feature correlation between cross-temporal hyperspectral images, and complex network models. Summary of the Invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method, system, equipment and medium, by making full use of point convolution to extract spectral information, using the Ghost module to extract spatial feature information, reducing the number of model parameters, simplifying the network structure, and improving the network operation efficiency; by using the cross-temporal feature fusion module to fully capture the correlation relationship between cross-temporal images to obtain change information, thereby improving the model detection accuracy.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method includes the following steps:

[0010] Step S1: preprocess the hyperspectral image data and divide it into a training set and a test set;

[0011] Step S2: constructing a feature extraction network consisting of two point convolution modules and four spatial spectrum feature fusion modules to extract image feature information;

[0012] Step S3: construct a classification network consisting of an attention weighted module, a global average pooling layer, and a fully connected layer to classify the image feature information extracted by the feature extraction network;

[0013] Step S4: Using the training set in step S1, the feature extraction network constructed in step S2 and the classification network constructed in step S3 are trained using the gradient descent algorithm. During the training iteration process, the accuracy of the training set is calculated for each iteration, and the weight of the first generation network model with the highest accuracy on the training set is used as the final detection model weight to obtain a trained model;

[0014] Step S5: Input the test set in step S1 into the model trained in step S4 for testing to obtain the final detection result, and output the predicted label map of the hyperspectral image data based on the detection result.

[0015] The implementation steps of step S1 include:

[0016] Step S1.1, obtain hyperspectral image data, and normalize the obtained hyperspectral image x, perform linear transformation on the hyperspectral image data, map the hyperspectral data to [0,1], and obtain the normalized hyperspectral image Where μ and σ are the mean and variance of the hyperspectral image data respectively;

[0017] Step S1.2: Set the sampling window of the hyperspectral image to the spatial size of the patch block, and sample samples containing spectral and spatial dimension information from the normalized hyperspectral image obtained in step S1.1;

[0018] Step S1.3: Use the window sampling in step S1.2 to obtain the patch block, use the label of the central pixel of the patch block as the label of the patch sample pair, and randomly select 10%-40% of the changed pixels and 60%-90% of the unchanged pixels of all labeled patch sample pairs in the hyperspectral image as the training set and the test set for testing.

[0019] In the step S2:

[0020] Two point convolution modules are used to preliminarily extract the features of the patch block obtained in step S1.3. Four spatial spectrum feature fusion modules are connected in series after the point convolution modules.

[0021] The spatial-spectral feature fusion module consists of a dual-branch spatial-spectral feature extraction module and a cross-temporal feature fusion module. The spatial-spectral feature extraction module is used to extract the spatial-spectral fusion features of the hyperspectral image, and the cross-temporal feature fusion module is used to explore the correlation feature information between cross-temporal images.

[0022] Each branch of the dual-branch spatial spectrum feature extraction module consists of point convolution and Ghost lightweight convolution. The spatial spectrum feature extraction module uses a dual-branch parallel feature extraction method to extract spectral and spatial features respectively. Among them, point convolution is used for spectral feature extraction, and the Ghost module is used for spatial feature extraction. The specific process is as follows:

[0023]

[0024] Among them, F l is the input feature map, Represents the point convolution operation, W l represents the parameter matrix, Extract features for spectral branches;

[0025]

[0026] Among them, F l is the input feature map, Represents the point convolution operation, W l represents the parameter matrix, It is the inherent feature map of the Ghost module after point dimensionality reduction;

[0027]

[0028] in, is the i-th feature map of the intrinsic feature map, i is 1 or 2;

[0029]

[0030] Among them, F l+3 Extract features from spectra and spatial feature extraction The fusion characteristics of

[0031] The cross-temporal feature fusion module includes an attention weighting module and a point convolution module. The cross-temporal feature fusion module first parallelizes the image features of different temporal phases. The parallel features are weighted by the attention weighting module. The weighted features are then fused and reduced in dimension using point convolution. The specific process is as follows:

[0032]

[0033] in, Represent the image features of the two phases respectively, For parallel operation, F l+1 Characteristics obtained after parallel connection;

[0034]

[0035] in, is the weighted operation on the i-th phase, is the feature obtained after weighted operation of the i-th phase;

[0036]

[0037] in, is the point convolution of the i-th phase, is the feature of the i-th phase after convolution.

[0038] In step S3, a classification network consisting of an attention weighted module, a global average pooling layer, and a fully connected layer is constructed. The specific process is as follows:

[0039]

[0040] in, Represents the image features of the two phases, F l+1 It is the feature obtained by subtracting the image features of the two phases;

[0041]

[0042] in, represents the weighted module, F l+2 Represents the weighted features;

[0043] F l+3 =GAP(F l+2 )

[0044] Among them, GAP represents the global average pooling layer, F l+3 is the feature after pooling;

[0045] F out =Liner(F l+3 )

[0046] Among them, Liner represents the fully connected layer, F out is the predicted probability distribution of the output;

[0047] Y P =argmax(F out )

[0048] Among them, argmax means taking F out The dimension with the largest value, Y P is the predicted label.

[0049] The specific implementation method of step S4 is:

[0050] Step S4.1, setting training parameters;

[0051] Step S4.2: Input the training set obtained in step S1 into the feature extraction model constructed in step S2, and input the output of the feature extraction model into the classification network model obtained in step S3 to obtain the model's predicted label Y P ;

[0052] Step S4.3: Calculate the cross entropy loss between the predicted label and the true label in step S4.2:

[0053]

[0054] Where c represents the number of categories, here c = 2, and are the true label and the predicted label, respectively, and N is the number of samples input in a single shot;

[0055] Step S4.4: During iterative optimization, the model loss Loss is calculated using the cross entropy loss function obtained in step S4.3, and the gradient of the cross entropy loss to the model weight is obtained. Update weights according to model weight gradients;

[0056] Step S4.5: Calculate the overall classification accuracy of the current model for all training sets obtained in step S1 in each iteration; take the weight of the generation model with the highest classification accuracy of training samples during the iteration as the final model weight.

[0057] In step S5, 3 to 10 independent inferences are performed on the test set, and the prediction accuracy of the inference results is averaged as the final prediction accuracy.

[0058] A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection system, comprising:

[0059] Data processing module: used to preprocess hyperspectral image data and divide it into training set and test set;

[0060] Network construction module: used to build a feature extraction network consisting of point convolution and four spatial-spectral feature fusion modules, and a classification network consisting of an attention weighting module, a global average pooling layer, and a fully connected layer;

[0061] Network training module: It is used to train the constructed feature extraction network and classification network using the training set and the gradient descent algorithm. During the training iteration process, the accuracy of the training set is calculated in each iteration, and the weight of the first generation network model with the highest accuracy on the training set is used as the final detection model weight to obtain the trained model;

[0062] Network testing module: used to input the test set into the trained model for testing, obtain the final detection results, and output the predicted label map of the hyperspectral image data based on the detection results.

[0063] The present invention also provides a cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection device, comprising:

[0064] Memory: a computer-readable device storing a computer program for the above-mentioned method for detecting cross-temporal lightweight spatial-spectral feature fusion hyperspectral change;

[0065] Processor: used to implement the cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method when executing the computer program.

[0066] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for detecting cross-temporal lightweight spatial-spectral feature fusion hyperspectral changes.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. Based on the unique properties of hyperspectral images, such as multiple spectral dimensions and one spectrum for each object, the present invention uses a multi-layer point convolution lightweight model to extract spectral information from hyperspectral images, which can better fuse spectral dimension information. At the same time, due to the small number of point convolution parameters, it can effectively simplify the classification network and reduce the complexity of the model.

[0069] 2. Based on the property of high-dimensional hyperspectral images with redundant information, the present invention uses Ghost convolution to extract spatial feature information, and extracts rich feature information through conventional convolution operations. For redundant feature information, cheaper linear transformation operations are used to generate it, which can effectively reduce the computing resources required for the model and achieve lightweight.

[0070] 3. Based on the rich spatial spectrum features of hyperspectral images, the present invention adopts a dual-branch structure to fuse the features extracted by the point convolution and Ghost convolution modules to fully extract the effective information of the hyperspectral images.

[0071] 4. Based on the dual-phase nature of change detection data, the present invention connects the spatial-spectral fusion features of different phases in parallel, performs attention weighting on the parallel features, enhances effective features, suppresses useless features, and reduces the dimension of the weighted features so that they match the features of different phases respectively, thereby fully capturing the correlation relationship between cross-phase images, better obtaining change information, and improving the model detection accuracy.

[0072] 5. The present invention only involves the basic knowledge of convolution and attention to achieve effective model lightweight design. It has low requirements for professional knowledge and does not require a complex module design process. It is suitable for tasks related to hyperspectral change detection, is easy to reproduce, and has strong versatility.

[0073] In summary, the present invention designs a model based on spatial-spectral feature extraction and cross-temporal feature fusion from the two aspects of refined feature extraction and cross-temporal feature fusion, and uses a lightweight method to simplify the model, thereby realizing a lightweight hyperspectral image change detection method with good performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of the process of the present invention.

[0075] Figure 2 This is a model structure diagram of the present invention.

[0076] Figure 3 This is a diagram analyzing the impact of Patch size.

[0077] Figure 4 This is the classification effect diagram of Jiangsu, Bay Area and Santa Barbara in the present invention, where: Figure 4 (a) is the detection effect diagram of the Jiangsu dataset. Figure 4 (b) is the detection effect diagram of the BayArea dataset. Figure 4 (c) is the detection effect diagram of the Santa Barbara dataset. DETAILED DESCRIPTION

[0078] The following is a clear and complete description of the embodiments and effects of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0079] like Figure 1 As shown, the implementation steps of this embodiment are as follows:

[0080] A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method includes the following steps:

[0081] Step S1: preprocess the hyperspectral image data and divide it into a training set and a test set;

[0082] Since the spectra of the data in the hyperspectral image have different value ranges, which will interfere with model reasoning, the hyperspectral image data needs to be preprocessed first. The specific implementation is as follows:

[0083] Step S1.1, data preprocessing:

[0084] Obtain hyperspectral image data from a public website, normalize the acquired hyperspectral image x, perform linear transformation on the hyperspectral image data, map the hyperspectral data to [0,1], and obtain the normalized hyperspectral image Where μ and σ are the mean and variance of the hyperspectral image data respectively;

[0085] Step S1.2, data sampling:

[0086] The original hyperspectral image contains all the ground objects. It needs to be sampled first to obtain the data and labels related to each pixel before it can be input into the model for inference operation. The implementation is as follows:

[0087] Step S1.2: Set the sampling window of the hyperspectral image to the spatial size of the patch block, and sample samples containing spectral and spatial dimension information from the normalized hyperspectral image obtained in step S1.1;

[0088] Step S1.3: Data division:

[0089] Use the window sampling in step S1.2 to obtain the patch block, and use the label of the central pixel of the patch block as the label of the patch sample pair. In this embodiment, 20% of all labeled patch sample pairs in the hyperspectral image are randomly selected from the changed pixels and the unchanged pixels as the training set, and the remaining 80% are used as the test set for testing.

[0090] In this embodiment, a patch block with a size of 5×5 is used for sampling.

[0091] Step S2: constructing a feature extraction network consisting of two point convolution modules and four spatial spectrum feature fusion modules to extract image feature information;

[0092] like Figure 2 As shown, in step S2:

[0093] Two point convolution modules are used to preliminarily extract the features of the patch block obtained in step S1.3. Four spatial spectrum feature fusion modules are connected in series after the point convolution modules.

[0094] The spatial-spectral feature fusion module consists of a dual-branch spatial-spectral feature extraction module and a cross-temporal feature fusion module. The spatial-spectral feature extraction module is used to extract the spatial-spectral fusion features of the hyperspectral image, and the cross-temporal feature fusion module is used to explore the correlation feature information between cross-temporal images.

[0095] Each branch of the dual-branch spatial spectrum feature extraction module consists of point convolution and Ghost lightweight convolution. The spatial spectrum feature extraction module uses a dual-branch parallel feature extraction method to extract spectral and spatial features respectively. Among them, point convolution is used for spectral feature extraction, and the Ghost module is used for spatial feature extraction. The specific process is as follows:

[0096]

[0097] Among them, F l is the input feature map, Represents the point convolution operation, W l represents the parameter matrix, Extract features for spectral branches;

[0098]

[0099] Among them, F l is the input feature map, Represents the point convolution operation, W l represents the parameter matrix, It is the inherent feature map of the Ghost module after point dimensionality reduction;

[0100]

[0101] in, is the i-th feature map of the intrinsic feature map, i is 1 or 2;

[0102]

[0103] Among them, F l+3 Extract features from spectra and spatial feature extraction The fusion characteristics of

[0104] The cross-temporal feature fusion module includes an attention weighting module and a point convolution module. The cross-temporal feature fusion module first parallelizes the image features of different temporal phases. The parallel features are weighted by the attention weighting module. The weighted features are then fused and reduced in dimension using point convolution. The specific process is as follows:

[0105]

[0106] in, Represent the image features of the two phases respectively, For parallel operation, F l+1 Characteristics obtained after parallel connection;

[0107]

[0108] in, is the weighted operation on the i-th phase, is the feature obtained after weighted operation of the i-th phase;

[0109]

[0110] in, is the point convolution of the i-th phase, is the feature of the i-th phase after convolution.

[0111] Step S3: construct a classification network consisting of an attention weighted module, a global average pooling layer, and a fully connected layer to classify the image feature information extracted by the feature extraction network;

[0112] In step S3, a classification network consisting of an attention weighted module, a global average pooling layer, and a fully connected layer is constructed. The specific process is as follows:

[0113]

[0114] in, Represents the image features of the two phases, F l+1 It is the feature obtained by subtracting the image features of the two phases;

[0115]

[0116] in, represents the weighted module, F l+2 Represents the weighted features;

[0117] F l+3 =GAP(F l+2 )

[0118] Among them, GAP represents the global average pooling layer, F l+3 is the feature after pooling;

[0119] F out =Liner(F l+3 )

[0120] Among them, Liner represents the fully connected layer, F out is the predicted probability distribution of the output;

[0121] Y P =argmax(F out )

[0122] Among them, argmax means taking F out The dimension with the largest value, Y P is the predicted label.

[0123] Step S4: Using the training set in step S1, the feature extraction network constructed in step S2 and the classification network constructed in step S3 are trained using the gradient descent algorithm. During the training iteration process, the accuracy of the training set is calculated for each iteration, and the weight of the first generation network model with the highest accuracy on the training set is used as the final detection model weight to obtain a trained model;

[0124] The specific implementation method of step S4 is:

[0125] Step S4.1: In this embodiment, the number of training generations is set to 300, the single sample input amount is 72, and the initial learning rate is 0.01;

[0126] Step S4.2: Input the training set obtained in step S1 into the feature extraction model constructed in step S2, and input the output of the feature extraction model into the classification network model obtained in step S3 to obtain the model's predicted label Y P ;

[0127] Step S4.3: Calculate the cross entropy loss between the predicted label and the true label in step S4.2:

[0128]

[0129] Where c represents the number of categories, here c = 2, and are the true label and the predicted label, respectively, and N is the number of samples input in a single shot;

[0130] Step S4.4: During iterative optimization, the model loss Loss is calculated using the cross entropy loss function obtained in step S4.3, and the gradient of the cross entropy loss to the model weight is obtained. Update weights according to model weight gradients;

[0131] Step 4.5: Calculate the overall classification accuracy of the current model for all training sets obtained in step S1 in each iteration; take the weight of the generation model with the highest classification accuracy of training samples during the iteration as the final model weight.

[0132] In step 5, the test set in step S1 is input into the model trained in step S4 for testing to obtain the final detection results, and the predicted label map of the hyperspectral image data is output based on the detection results. Three independent inferences are performed on the test set, and the prediction accuracy of the three inference results is averaged as the final prediction accuracy.

[0133] The effects of the present invention can be further illustrated by the following experiments.

[0134] 1. Experimental conditions:

[0135] Hardware environment: Intel W-2123 processor, main frequency 3.60GHz, memory 64GB, NVIDIA GeForce GTX2080Ti graphics processing unit GPU, memory 11GB.

[0136] Software environment: 64-bit Windows 10 system and Pytorch 1.6.0 deep learning framework.

[0137] Dataset: The Jiangsu Dataset, collected by Hyperion sensors, primarily records material changes in a river in Jiangsu Province, China. The two multi-temporal images were captured on May 3, 2013, and December 31, 2013, respectively, with a size of 463 × 241 pixels. In the experiment, after removing noise bands, 198 bands were used for change detection.

[0138] BayArea dataset: acquired by AVIRIS, mainly covering urban and farmland areas in Patterson California, USA. The two HSIs were acquired in 2013 and 2015 respectively. The two images are 600×500 pixels, and the experiment uses 224 available bands.

[0139] Santa Barbara dataset: Acquired by AVIRIS, two HSIs were taken over the Santa Barbara area of ​​California in 2013 and 2014. The size is 984×740 pixels and there are 224 spectral bands.

[0140] The evaluation indicators include overall accuracy OA, Kappa coefficient KAPPA, true positive examples TP, true negative examples TN, false positive examples FP, false negative examples FN, where:

[0141] Overall accuracy OA indicates the proportion of correctly classified samples to all samples. The larger the value, the better the classification effect.

[0142]

[0143] Kappa coefficient KAPPA represents the different weights in the confusion matrix. The larger the value, the better the classification effect.

[0144]

[0145]

[0146] Among them, the true positive example TP represents the number of pixels whose changes are correctly detected;

[0147] True negative examples TN represents the number of unchanged pixels that are correctly detected;

[0148] False positives FP represents the number of unchanged pixels in the true category map that are falsely detected;

[0149] False negatives FN represents the number of changed pixels that are incorrectly classified as unchanged pixels.

[0150] 2. Experimental content

[0151] Experiment 1. In the patch-based method, the patch size has a great impact on the result accuracy.

[0152] First, we analyze the effect of patch size on classification results. For each data set, we try patch sizes of 1, 3, 5, 7, and 9. 1×1 can be regarded as using only spectral information for classification. Its OA value is as follows: Figure 3 As shown, all three datasets achieved the worst classification accuracy with a 1×1 patch, indicating that the introduction of spatial information helps enhance the discernibility of change detection tasks. As the patch size increases, the OA of the Jiangsu dataset first increases and then decreases, achieving optimal classification accuracy with a 5×5 patch. The OA values ​​of the other two datasets first increase and then remain stable, indicating that the actual patch size has a certain impact on model performance, with optimal performance achieved after the patch size reaches a certain threshold. Therefore, based on this analysis, a 5×5 patch size was used as the optimal parameter for subsequent experiments.

[0153] Experiment 2: The number of model channels affects model complexity, parameter quantity and operating efficiency.

[0154] Generally speaking, if the model uses too many channels, it is easy to cause overfitting, while if it uses too few channels, the amount of information is insufficient and it is difficult to extract features more fully.

[0155] In this embodiment, we try to set the number of model channels to 4, 8, 16, 32, 64, and 128 respectively, and the results are shown in Table 1. As can be seen from the table, when the number of channels is 4, the worst classification effect is achieved on all three data sets. As the number of channels increases, OA gradually increases to a stable level, and then gradually decreases. This shows that when the number of channels is low, the model parameters are relatively small, and the knowledge capacity is relatively small, resulting in the failure to fully utilize the model performance; when the number of channels is large, the model parameters are too many, resulting in overfitting. In addition, further exploration of the relationship between the number of channels, OA and the number of parameters shows that this method can still maintain acceptable classification performance even when the number of channels is too low, indicating that the model has good feature extraction capabilities and can extract discriminative features with extremely low parameter amounts. In order to balance model performance and efficiency, the number of channels is set to 64 in subsequent experiments.

[0156] Table 1 Comparison of OA (%) and parameters of models with different channel numbers

[0157]

[0158] Experiment 3: Residual connections are widely used in convolutional neural network models. They can alleviate the gradient vanishing problem, accelerate training speed, improve model training results, and enhance the generalization ability of the network. However, when the network becomes deeper, since residual connections allow gradients to flow more easily, it may cause some gradient explosion problems. Therefore, we experimented with the number of spatial-spectral feature fusion modules and the use of residual connections. In this embodiment, we tried to use 2, 3, 4, and 5 spatial-spectral feature fusion modules, and tried with or without residual connections to verify the model performance. The results are shown in Table 2. As can be seen from the table, with the increase in the number of spatial-spectral feature fusion modules, the effect on the three data sets first increased and then leveled off. The addition of residual connections will also improve a certain degree of accuracy, so we adopted four spatial-spectral feature fusion modules and added residual connections.

[0159] Table 2 Number of spatial-spectral feature fusion modules and detection accuracy of the residual connection model

[0160]

[0161] Experiment 4: On the Jiangsu hyperspectral change detection data, the cross-temporal lightweight spatial spectrum feature fusion hyperspectral change detection method of the present invention is used to obtain the detection result graph and calculate the evaluation index. The results are shown in Table 3 and Figure 4 (a).

[0162] Table 3 Accuracy of different detection methods on the Jiangsu dataset

[0163]

[0164]

[0165] As shown in Table 3, the OA and KAPPA coefficients of the proposed method on the Jiangsu hyperspectral change detection data are both higher than those of existing methods. Compared with traditional methods, the OA of the proposed method is 2.72% higher than the PCA method and 4.26% higher than the IRMAD method. The KAPPA coefficient is 12.48% higher than the PCA method and 19.87% higher than the IRMAD method. Compared with deep learning methods, the OA of the proposed method is 1.94% higher than the LCNN method and 0.21% higher than the BCNNs method. The KAPPA coefficient is 14.57% higher than the LCNN method and 1.59% higher than the BCNNs method.

[0166] like Figure 4(a) shows that to more clearly compare the effects of each method, we represent TP in white, TN in black, FP in red, FN in green, and unmarked areas in gray. In contrast, the results of the present invention (far right) contain fewer misclassified pixels, indicating that the proposed method has the best performance and can well identify the changed area.

[0167] Experiment 5: On the BayArea hyperspectral change detection data, the cross-temporal lightweight spatial spectrum feature fusion hyperspectral change detection method of this embodiment is used to obtain the detection result graph and calculate the evaluation index. The results are shown in Table 4 and Figure 4 (b).

[0168] Table 4 Accuracy of different detection methods on the BayArea dataset

[0169]

[0170]

[0171] As shown in Table 4, the OA and KAPPA coefficients of the present invention on the BayArea hyperspectral change detection data are both higher than those of existing methods. Compared with traditional methods, the OA of the present invention is 15.25% higher than that of the PCA method and 15.87% higher than that of the IRMAD method. The KAPPA coefficient is 30.18% higher than that of the PCA method and 31.33% higher than that of the IRMAD method. Compared with deep learning methods, the OA of the present invention is 0.64% higher than that of the LCNN method and 0.34% higher than that of the BCNNs method. The KAPPA coefficient is 1.28% higher than that of the LCNN method and 0.78% higher than that of the BCNNs method.

[0172] like Figure 4 (b) To more clearly compare the effects of the various methods, we represent TP in white, TN in black, FP in red, FN in green, and unmarked areas in gray. In contrast, the results of our invention (far right) contain fewer misclassified pixels, indicating that the proposed method has the best performance and can well identify the changed areas.

[0173] Experiment 6: On the Santa Barbara hyperspectral change detection data, the cross-temporal lightweight spatial spectrum feature fusion hyperspectral change detection method of this example is used to obtain the detection result map and calculate the evaluation index. The results are shown in Table 5 and Figure 4 (c).

[0174] Table 5 Accuracy of different detection methods on the Santa Barbara dataset

[0175] method OA KAPPA TP TN FP FN PCA 80.03 55.62 30425 75663 4755 21709 IRMAD 86.07 71.61 47094 66996 13422 5040 LCNN 99.32±0.04 98.58±0.08 51760 79932 486 373 BCNNs 99.56±0.02 99.10±0.05 51812 80194 223 322 Ours 99.77±0.03 99.52±0.07 51974 80320 98 160

[0176] As shown in Table 5, the OA and KAPPA coefficients of the present invention on the Santa Barbara hyperspectral change detection data are both higher than those of existing methods. Compared with traditional methods, the OA of the present invention is 19.74% higher than that of the PCA method and 13.70% higher than that of the IRMAD method. The KAPPA coefficient is 43.90% higher than that of the PCA method and 27.91% higher than that of the IRMAD method. Compared with deep learning methods, the OA of the present invention is 0.45% higher than that of the LCNN method and 0.21% higher than that of the BCNNs method. The KAPPA coefficient is 0.94% higher than that of the LCNN method and 0.42% higher than that of the BCNNs method.

[0177] like Figure 4 (c) To more clearly compare the effects of the various methods, we represent TP in white, TN in black, FP in red, FN in green, and unmarked areas in gray. In contrast, the results of our invention (far right) contain fewer misclassified pixels, indicating that the proposed method has the best performance and can well identify the changed areas.

[0178] Experiment 7: This example verifies the effectiveness of the proposed method by conducting ablation experiments on the proposed model's spatial-spectral feature extraction module, cross-temporal feature fusion module, and attention weighting module. The experimental results are shown in Table 6. Single spectrum and single space refer to the spatial-spectral feature extraction module using only the spectral feature extraction branch or the spatial feature extraction branch for feature extraction, respectively. No cross-temporal feature fusion module refers to the deletion of the cross-temporal feature fusion module. No attention weighting module refers to the deletion of the classification network attention weighting module.

[0179] Table 6 Comparison of ablation test results of each module

[0180]

[0181] Table 6 shows that each of the aforementioned modules improves model performance to some extent. The cross-temporal feature fusion module achieves the highest performance gain among all modules, demonstrating the importance of integrating cross-temporal feature correlations in hyperspectral image change detection. The parallel spectral and spatial branches extract spatial spectral features, which helps the model refine more refined change features. After subtracting some classifier features, the change features are weighted by local spatial regions in the attention weighting module to enhance the discriminability of important features.

[0182] Experimental results demonstrate that the proposed cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method addresses the issues of existing hyperspectral ground object change detection methods, such as incomplete spatial-spectral feature extraction, insufficient attention to feature correlations between cross-temporal hyperspectral images, and complex network models. This method achieves superior results compared to existing technologies while requiring fewer computational resources and lowering storage costs. The proposed method can also be used for hyperspectral ground object change detection across different temporal phases.

[0183] A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection system, comprising:

[0184] Data processing module: used to implement the preprocessing of hyperspectral image data in step S1 and divide it into training set and test set;

[0185] Network construction module: used to implement the feature extraction network composed of point convolution and four spatial-spectral feature fusion modules in steps S2 and S3, and to construct a classification network composed of an attention weighting module, a global average pooling layer, and a fully connected layer;

[0186] Network training module: used to implement the training set in step S1 in step S4, and use the gradient descent algorithm to train the feature extraction network constructed in step S2 and the classification network constructed in step S3. During the training iteration process, the accuracy of the training set is calculated in each iteration, and the weight of the first generation network model with the highest accuracy on the training set is used as the final detection model weight to obtain the trained model;

[0187] Network testing module: used to implement step S5 in which the test set in step S1 is input into the model trained in step S4 for testing, obtain the final detection result, and output the predicted label map of the hyperspectral image data according to the detection result.

[0188] The present invention also provides a cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection device, comprising:

[0189] Memory: a computer-readable device storing a computer program for the above-mentioned method for detecting cross-temporal lightweight spatial-spectral feature fusion hyperspectral change;

[0190] Processor: used to implement the cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method when executing the computer program.

[0191] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for detecting cross-temporal lightweight spatial-spectral feature fusion hyperspectral changes.

[0192] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method, characterized by: The following steps are involved: Step S1: preprocess the hyperspectral image data and divide it into a training set and a test set; Step S2: constructing a feature extraction network consisting of two point convolution modules and four spatial spectrum feature fusion modules to extract image feature information; Step S3: construct a classification network consisting of an attention weighted module, a global average pooling layer, and a fully connected layer to classify the image feature information extracted by the feature extraction network; The classification network consisting of an attention weighted module, a global average pooling layer, and a fully connected layer is constructed as follows: in, Represents the image features of the two phases, F l+1 It is the feature obtained by subtracting the image features of the two phases; in, represents the weighted module, F l+2 Represents the weighted features; F l+3 =GAP(F l+2 ) Among them, GAP represents the global average pooling layer, F l+3 is the feature after pooling; F out =Liner(F l+3 ) Among them, Liner represents the fully connected layer, F out is the predicted probability distribution of the output; Y P =argmax(F out ) Among them, argmax means taking F out The dimension with the largest value, Y P is the predicted label; Step S4: Using the training set in step S1, the feature extraction network constructed in step S2 and the classification network constructed in step S3 are trained using the gradient descent algorithm. During the training iteration process, the accuracy of the training set is calculated for each iteration, and the weight of the first generation network model with the highest accuracy on the training set is used as the final detection model weight to obtain a trained model; Step S5: Input the test set in step S1 into the model trained in step S4 for testing to obtain the final detection result, and output the predicted label map of the hyperspectral image data based on the detection result.

2. The method for detecting hyperspectral changes across time using lightweight spatial-spectral feature fusion according to claim 1 is characterized by: The implementation steps of step S1 include: Step S1.1, obtain hyperspectral image data, and normalize the obtained hyperspectral image x, perform linear transformation on the hyperspectral image data, map the hyperspectral data to [0,1], and obtain the normalized hyperspectral image Where μ and σ are the mean and variance of the hyperspectral image data respectively; Step S1.2: Set the sampling window of the hyperspectral image to the spatial size of the patch block, and sample samples containing spectral and spatial dimension information from the normalized hyperspectral image obtained in step S1.1; Step S1.3: Use the window sampling in step S1.2 to obtain the patch block, use the label of the central pixel of the patch block as the label of the patch sample pair, and randomly select 10%-40% of the changed pixels and 60%-90% of the unchanged pixels of all labeled patch sample pairs in the hyperspectral image as the training set and the test set for testing.

3. The method for detecting hyperspectral changes across time using lightweight spatial-spectral feature fusion according to claim 1 is characterized by: In the step S2: Two point convolution modules are used to preliminarily extract the features of the patch block obtained in step S1.

3. Four spatial spectrum feature fusion modules are connected in series after the point convolution modules. The spatial-spectral feature fusion module consists of a dual-branch spatial-spectral feature extraction module and a cross-temporal feature fusion module. The spatial-spectral feature extraction module is used to extract the spatial-spectral fusion features of the hyperspectral image, and the cross-temporal feature fusion module is used to explore the correlation feature information between cross-temporal images. Each branch of the dual-branch spatial spectrum feature extraction module consists of point convolution and Ghost lightweight convolution. The spatial spectrum feature extraction module uses a dual-branch parallel feature extraction method to extract spectral and spatial features respectively. Among them, point convolution is used for spectral feature extraction, and the Ghost module is used for spatial feature extraction. The specific process is as follows: Among them, F l is the input feature map, Represents the point convolution operation, W l represents the parameter matrix, Extract features for spectral branches; Among them, F l is the input feature map, Represents the point convolution operation, W l represents the parameter matrix, It is the inherent feature map of the Ghost module after point dimensionality reduction; in, is the i-th feature map of the intrinsic feature map, i is 1 or 2; Among them, F l+3 Extract features from spectra and spatial feature extraction The fusion characteristics of The cross-temporal feature fusion module includes an attention weighting module and a point convolution module. The cross-temporal feature fusion module first parallelizes the image features of different temporal phases. The parallel features are weighted by the attention weighting module. The weighted features are then fused and reduced in dimension using point convolution. The specific process is as follows: in, Represent the image features of the two phases respectively, For parallel operation, F l+1 Characteristics obtained after parallel connection; in, is the weighted operation on the i-th phase, is the feature obtained after weighted operation of the i-th phase; in, is the point convolution of the i-th phase, is the feature of the i-th phase after convolution.

4. The method for detecting hyperspectral changes across time using lightweight spatial-spectral feature fusion according to claim 1 is characterized by: The specific implementation method of step S4 is: Step S4.1, setting training parameters; Step S4.2: Input the training set obtained in step S1 into the feature extraction model constructed in step S2, and input the output of the feature extraction model into the classification network model obtained in step S3 to obtain the model's predicted label Y P ; Step S4.3: Calculate the cross entropy loss between the predicted label and the true label in step S4.2: Where c represents the number of categories, here c = 2, and are the true label and the predicted label, respectively, and N is the number of samples input in a single shot; Step S4.4: During iterative optimization, the model loss Loss is calculated using the cross entropy loss function obtained in step S4.3, and the gradient of the cross entropy loss to the model weight is obtained. Update weights according to model weight gradients; Step S4.5: Calculate the overall classification accuracy of the current model for all training sets obtained in step S1 in each iteration; take the weight of the generation model with the highest classification accuracy of training samples during the iteration as the final model weight.

5. The method for detecting hyperspectral changes across time by integrating lightweight spatial-spectral features according to claim 1 is characterized by: In step S5, 3 to 10 independent inferences are performed on the test set, and the prediction accuracy of the inference results is averaged as the final prediction accuracy.

6. A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection system, characterized by: include: Data processing module: used to preprocess hyperspectral image data and divide it into training set and test set; Network construction module: used to build a feature extraction network consisting of point convolution and four spatial-spectral feature fusion modules, and a classification network consisting of an attention weighting module, a global average pooling layer, and a fully connected layer; The classification network consisting of an attention weighted module, a global average pooling layer, and a fully connected layer is constructed as follows: in, Represents the image features of the two phases, F l+1 It is the feature obtained by subtracting the image features of the two phases; in, represents the weighted module, F l+2 Represents the weighted features; F l+3 =GAP(F l+2 ) Among them, GAP represents the global average pooling layer, F l+3 is the feature after pooling; F out =Liner(F l+3 ) Among them, Liner represents the fully connected layer, F out is the predicted probability distribution of the output; Y P =argmax(F out ) Among them, argmax means taking F out The dimension with the largest value, Y P is the predicted label; Network training module: It is used to train the constructed feature extraction network and classification network using the training set and the gradient descent algorithm. During the training iteration process, the accuracy of the training set is calculated in each iteration, and the weight of the first generation network model with the highest accuracy on the training set is used as the final detection model weight to obtain the trained model; Network testing module: used to input the test set into the trained model for testing, obtain the final detection results, and output the predicted label map of the hyperspectral image data based on the detection results.

7. A cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection device, characterized by: include: Memory: a computer-readable device storing a computer program for a method for detecting cross-temporal lightweight spatial-spectral feature fusion hyperspectral changes according to any one of claims 1 to 5; Processor: used to implement the cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method described in any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the cross-temporal lightweight spatial-spectral feature fusion hyperspectral change detection method described in any one of claims 1 to 5.

Images