Cross-scene hyperspectral image classification method, device and equipment and storage medium

Abstract

The application relates to a cross-scene hyperspectral image classification method, device and equipment and a storage medium. The method comprises the following steps: acquiring source domain data and target domain data; constructing a spectral space confrontation multi-domain synthesis network; the multi-domain synthesis network comprises a first branch and a second branch which have the same network structure and input spectral blocks; each branch comprises a domain generation network, a multi-scale mutual information regularization network, a multi-domain mixing module and a semantic consistency discrimination network; the domain generation network parameters in the two branches of the multi-domain synthesis network are not shared, and the semantic consistency discrimination network parameters are shared; the spectral space confrontation multi-domain synthesis network is adversarially trained according to a pre-constructed loss function and the source domain data, so that a trained spectral space confrontation multi-domain synthesis network is obtained; and the target domain data is input into the trained spectral space confrontation multi-domain synthesis network, so that a classification result is obtained. The method can improve the classification accuracy of a cross-scene hyperspectral image classification task facing single-source domain generalization.

Description

Technical Field

[0001] This application relates to the field of hyperspectral image classification technology, and in particular to a method, apparatus, device and storage medium for cross-scene hyperspectral image classification. Background Technology

[0002] Hyperspectral remote sensing is a multi-dimensional information acquisition technology that combines imaging and spectral techniques, enabling both imaging and spectral measurement. Hyperspectral images obtained through imaging spectrometers possess the characteristic of unified image and spectrum; each pixel corresponds to a nearly continuous spectral curve, reflecting diagnostic spectral absorption differences in ground features. Thanks to the rich spatial and spectral information that hyperspectral images can provide, the use of hyperspectral images for fine-grained classification of ground features is widely applied in fields such as wetland monitoring, precision agriculture, urban planning, and resource surveys.

[0003] Many past studies have focused on single-scene classification, that is, selecting samples in a specific scene to train the model and then testing it in that scene. These methods include traditional machine learning approaches (such as k-nearest neighbors, support vector machines, and sparse representations) and deep learning-based methods (such as convolutional neural networks, graph neural networks, and transformer networks), achieving excellent performance in land cover classification. However, for new scenes, it is often necessary to retrain the model with sufficient samples containing labeled information. Accurate labeling of a large number of categories in a new scene is both time-consuming and expensive, and the resulting model can only be applied to that scene, making it difficult to generalize to other scenes.

[0004] Due to differences in sensors capturing two scenes, as well as variations in time, season, and weather conditions, similar ground features exhibit spectral drift and spatial texture variations across different scenes. Directly transferring a model trained in the source domain to the target domain for testing results in a significant drop in classification performance. To improve the model's generalization ability to the target domain, thanks to the rapid development of transfer learning, researchers have recently utilized domain adaptation techniques for cross-scene hyperspectral image classification. Domain adaptation, as a type of transductive transfer learning, learns shared cross-domain invariant knowledge between the source and target domains to enable the model to generalize well to the target domain. Domain adaptation-based cross-scene hyperspectral image classification methods mainly include probability distribution alignment, feature selection, subspace learning, and deep domain adaptation. Although domain adaptation-based hyperspectral cross-scene classification methods have achieved satisfactory classification performance, they not only require data from the target domain to participate in model training, but the acquired model knowledge can only be used for the specific target scene involved in the model training process. When facing other scenes not involved in model training, the model needs to be retrained for those specific scenes.

[0005] To free models from dependence on target scene data, it is desirable for models to learn using only source domain data to exhibit better generalization performance in unknown domains. Domain generalization is a promising direction for solving this problem. However, traditional domain generalization (multi-source domain generalization) requires multiple labeled source domains during training, which is very difficult for cross-scene hyperspectral image classification because it is hard to obtain labeled hyperspectral data from multiple different scenes with the same category. Single-source domain generalization aims to train a deep model from a single source domain that can generalize to unseen domains, which is more in line with the practical application scenario of cross-scene hyperspectral image classification. The most popular approach to solving single-source domain generalization is domain expansion, which extends a single source domain by generating diverse pseudo-domains.

[0006] Domain expansion methods designed for traditional images only focus on changes in spatial information. However, for cross-scene hyperspectral image classification, the domain differences caused by changes in spatial texture information and spectral drift lead to poor performance in cross-scene hyperspectral image classification tasks. SDEnet, designed for cross-scene hyperspectral image classification tasks that generalize from a single source domain, generates expanded domains with changes in spatial and spectral information through spatial and spectral randomization in the generator. However, to ensure the class invariance information of the expanded domain, SDEnet only uses morphological knowledge extracted by a simple morphological encoder as template features to constrain the essential class semantic information in the expanded domain. Since it only constrains the expanded domain on the spatial information and not on the spectral information, it cannot effectively ensure that the essential class semantic information of the expanded domain is consistent with the source domain by relying solely on the morphological knowledge of the source domain. This is detrimental to the model's learning of class semantic consistency information. Summary of the Invention

[0007] Therefore, it is necessary to provide a method, apparatus, device, and storage medium for cross-scene hyperspectral image classification to address the aforementioned technical problems.

[0008] A cross-scene hyperspectral image classification method, the method comprising:

[0009] Acquire source domain data and target domain data; the source domain data includes the original spectral blocks and their corresponding category labels; the target domain data includes the spectral blocks to be classified and their corresponding category labels.

[0010] A spectral-spatial adversarial multi-domain synthesis network is constructed. The multi-domain synthesis network includes two branches with identical network structures and identical input spectral blocks: a first branch and a second branch. Each branch includes a domain generation network, a multi-scale mutual information regularization network, a multi-domain mixing module, and a semantic consistency discrimination network. The parameters of the domain generation network in the two branches of the multi-domain synthesis network are not shared, while the parameters of the semantic consistency discrimination network are shared. The domain generation network is used to extract the spectral and spatial information of the original spectral block to obtain spectral and spatial features. These spectral and spatial features are then fused to obtain spectral-spatial fusion features. Adaptive instance normalization, activation, and decoding are performed on the spectral-spatial fusion features sequentially to obtain output features. The output features and the original spectral block are then adaptively fused to obtain generated domain data. The multi-scale mutual information regularization network is used to calculate the multi-scale mutual information between the original spectral block and the generated domain data. The multi-domain mixing module is used to mix the generated domain data and the source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network is used to extract spectral and spatial features, fuse features, flatten, and classify the mixed domain data and the generated domain data to obtain corresponding classification results.

[0011] The spectral-space adversarial multi-domain synthesis network is adversarially trained based on the pre-constructed loss function and the source domain data to obtain the trained spectral-space adversarial multi-domain synthesis network.

[0012] The target domain data is input into a trained spectral-space adversarial multi-domain synthesis network to obtain the classification result corresponding to the spectral block to be classified.

[0013] A cross-scene hyperspectral image classification device, the device comprising:

[0014] The data acquisition module is used to acquire source domain data and target domain data; the source domain data includes raw spectral blocks and their corresponding category labels; the target domain data includes spectral blocks to be classified and their corresponding category labels.

[0015] The network construction module is used to construct a spectral-spatial adversarial multi-domain synthesis network. The multi-domain synthesis network includes a first branch and a second branch with identical network structures and input spectral blocks. Each branch includes a domain generation network, a multi-scale mutual information regularization network, a multi-domain hybridization module, and a semantic consistency discrimination network. The domain generation network parameters are not shared between the two branches, while the semantic consistency discrimination network parameters are shared. The domain generation network is used to extract the spectral and spatial information of the original spectral block to obtain spectral features and spatial features, and then fuses these features to obtain spectral-spatial fusion features. The spectral-spatial fusion features are sequentially subjected to adaptive instance normalization, activation, and decoding to obtain output features. The output features and the original spectral block are then adaptively fused to obtain the generated domain data. The multi-scale mutual information regularization network is used to calculate the multi-scale mutual information between the original spectral block and the generated domain data. The multi-domain mixing module is used to mix the generated domain data and the source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network is used to extract spectral and spatial features, fuse features, flatten, and classify the mixed domain data and the generated domain data respectively to obtain corresponding classification results.

[0016] The network training module is used to perform adversarial training on the spectral-space adversarial multi-domain synthesis network based on a pre-constructed loss function and the source domain data, so as to obtain a trained spectral-space adversarial multi-domain synthesis network.

[0017] The target domain classification module is used to input the target domain data into the trained spectral spatial adversarial multi-domain synthesis network to obtain the classification result corresponding to the spectral block to be classified.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0019] Acquire source domain data and target domain data; the source domain data includes the original spectral blocks and their corresponding category labels; the target domain data includes the spectral blocks to be classified and their corresponding category labels.

[0020] A spectral-spatial adversarial multi-domain synthesis network is constructed. The multi-domain synthesis network includes two branches with identical network structures and identical input spectral blocks: a first branch and a second branch. Each branch includes a domain generation network, a multi-scale mutual information regularization network, a multi-domain mixing module, and a semantic consistency discrimination network. The parameters of the domain generation network in the two branches of the multi-domain synthesis network are not shared, while the parameters of the semantic consistency discrimination network are shared. The domain generation network is used to extract the spectral and spatial information of the original spectral block to obtain spectral and spatial features. These spectral and spatial features are then fused to obtain spectral-spatial fusion features. Adaptive instance normalization, activation, and decoding are performed on the spectral-spatial fusion features sequentially to obtain output features. The output features and the original spectral block are then adaptively fused to obtain generated domain data. The multi-scale mutual information regularization network is used to calculate the multi-scale mutual information between the original spectral block and the generated domain data. The multi-domain mixing module is used to mix the generated domain data and the source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network is used to extract spectral and spatial features, fuse features, flatten, and classify the mixed domain data and the generated domain data to obtain corresponding classification results.

[0021] The spectral-space adversarial multi-domain synthesis network is adversarially trained based on the pre-constructed loss function and the source domain data to obtain the trained spectral-space adversarial multi-domain synthesis network.

[0022] The target domain data is input into a trained spectral-space adversarial multi-domain synthesis network to obtain the classification result corresponding to the spectral block to be classified.

[0023] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0024] Acquire source domain data and target domain data; the source domain data includes the original spectral blocks and their corresponding category labels; the target domain data includes the spectral blocks to be classified and their corresponding category labels.

[0025] A spectral-spatial adversarial multi-domain synthesis network is constructed. The multi-domain synthesis network includes two branches with identical network structures and identical input spectral blocks: a first branch and a second branch. Each branch includes a domain generation network, a multi-scale mutual information regularization network, a multi-domain mixing module, and a semantic consistency discrimination network. The parameters of the domain generation network in the two branches of the multi-domain synthesis network are not shared, while the parameters of the semantic consistency discrimination network are shared. The domain generation network is used to extract the spectral and spatial information of the original spectral block to obtain spectral and spatial features. These spectral and spatial features are then fused to obtain spectral-spatial fusion features. Adaptive instance normalization, activation, and decoding are performed on the spectral-spatial fusion features sequentially to obtain output features. The output features and the original spectral block are then adaptively fused to obtain generated domain data. The multi-scale mutual information regularization network is used to calculate the multi-scale mutual information between the original spectral block and the generated domain data. The multi-domain mixing module is used to mix the generated domain data and the source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network is used to extract spectral and spatial features, fuse features, flatten, and classify the mixed domain data and the generated domain data to obtain corresponding classification results.

[0026] The spectral-space adversarial multi-domain synthesis network is adversarially trained based on the pre-constructed loss function and the source domain data to obtain the trained spectral-space adversarial multi-domain synthesis network.

[0027] The target domain data is input into a trained spectral-space adversarial multi-domain synthesis network to obtain the classification result corresponding to the spectral block to be classified.

[0028] The aforementioned cross-scene hyperspectral image classification methods, devices, equipment, and storage media improve the robustness of the model to unseen domains by expanding the diverse distribution of the source domain. Specifically, to generate diverse generated domains spatially and spectrally, a spectral-spatial domain generation network (S2DGN) is designed. A multi-domain hybrid module is used to further expand the diversity of generated domains and enhance their class-domain semantic consistency information. In addition, a multi-scale mutual information regularization network is used to constrain the S2DGN so that the essential class semantic information of the generated domains does not deviate from the source domain. A semantic consistency discriminator with spectral-spatial feature extraction capabilities is used to capture class-domain semantic consistency information from the diverse generated domains to obtain cross-domain invariant knowledge. This method can improve the classification accuracy of cross-scene hyperspectral image classification tasks that generalize to single-source-domain applications. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating a cross-scene hyperspectral image classification method in one embodiment;

[0030] Figure 2 This is a schematic diagram of the spectral-space adversarial multi-domain synthesis network structure in one embodiment;

[0031] Figure 3 This is a schematic diagram of the structure of a multi-scale mutual information regularization network in one embodiment;

[0032] Figure 4 In one embodiment, S 2 Classification images obtained by AMSnet and contrasting methods on the Pavia Center dataset for the target scene;

[0033] Figure 5 This is a structural block diagram of a cross-scene hyperspectral image classification device in one embodiment;

[0034] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] In one embodiment, such as Figure 1 As shown, a cross-scene hyperspectral image classification method is provided, including the following steps:

[0037] Step 102: Obtain source domain data and target domain data.

[0038] Source domain data includes the original spectral patches and their corresponding class labels. Target domain data includes the spectral patches to be classified and their corresponding class labels. In hyperspectral image classification, the source domain typically refers to a pre-labeled hyperspectral image dataset used to train the model. This dataset may come from previous tasks or public datasets, where each sample has a clear label indicating its class. The model trained on the source domain dataset learns some general features and knowledge that can be applied to the target domain. The target domain is typically the domain where the model is intended to be applied, i.e., the domain to which the model is transferred. The target domain dataset may differ from the source domain; it could be images from different regions, different satellites, or data collected at different times.

[0039] Step 104: Construct a spectral-space adversarial multi-domain synthesis network.

[0040] like Figure 2 The diagram shown illustrates the structure of a spectral-space adversarial multi-domain synthesis network. (S) 2AMSnet comprises two branches with identical network structures and input spectral blocks: a first branch and a second branch. Each branch includes a Spectral-Spatial Domain Generation Network, a Multiscale Mutual Information Regularization Network, a Multidomain Mixing Module, and a Semantic Consistency Discriminator. In the multidomain synthesis network, the parameters of the domain generation network are not shared between the two branches, while the parameters of the semantic consistency discriminator network are shared.

[0041] The domain generation network extracts spectral and spatial information from the original spectral blocks to obtain spectral and spatial features. These features are then fused to obtain spectral-spatial fusion features. Adaptive instance normalization, activation, and decoding are performed on these fusion features sequentially to obtain output features. The output features and the original spectral blocks are then adaptively fused to obtain the generated domain data. The multi-scale mutual information regularization network calculates the multi-scale mutual information between the original spectral blocks and the generated domain data. The multi-domain mixing module mixes the generated domain data and source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network extracts, fuses, flattens, and classifies the mixed domain data and the generated domain data, respectively, to obtain the corresponding classification results.

[0042] Specifically, in order to generate diverse data to improve the model's generalization ability, in S 2 In AMSnet, two S-classes with identical structures but non-shared parameters were designed. 2 DGN, to generate diverse generation domains. 2 The model structure of DGN is as follows: Figure 2 S in 2 As shown in DGN. Hyperspectral images possess rich spectral and spatial information. In order to generate diverse generative domain data at both the spatial and spectral levels, S... 2 DGN includes spectral feature branches and spatial feature branches. For any sampled spectral block... First, the spectral and spatial features of the spectral patch are extracted through spectral feature branching and spatial feature branching, respectively. Spectral features It is obtained through involution, which extracts the spectral features of each pixel in the spectral patch along the band direction to obtain the spectral information of the spectral patch. This process can be represented as:

[0043]

[0044] in Δ is the convolution kernel for inner convolution, k is the number of output channels, and G is the number of groups where each group shares the same convolution kernel. K It is the offset set of the surrounding neighborhood centered on the pixel at (i, j), and B is the band number of the spectral patch. Spatial characteristics It is obtained by convolving the spatial dimension of the spectral patch using 2D convolution. This process can be represented as:

[0045]

[0046] in It is a 2D convolution kernel, C out It is the number of output channels, C in It is the size of the number of input channels, in S 2 DGN in C out =C in =B. After spectral feature branching and spatial feature branching, the spectral and spatial features of the spectral block are fused to obtain the spectral-spatial fusion feature. Then, the spectral-space fusion features are subjected to adaptive instance normalization (AdaIN), which can be represented as:

[0047]

[0048] in It is random noise sampled from a normal distribution; FC1 and FC2 represent fully connected layers. and These are the mean and standard deviation of the spectral-space fusion features, respectively. The features output by AdaIN are processed by a Leaky Rectified Linear Unit (LeakyReLU) nonlinear activation function and then decoded to obtain... Finally, through learnable weights and Adaptive fusion With the original spectral block Get S 2 DGN-generated domain data It can be formalized as:

[0049]

[0050] in

[0051] By using adversarial training and random noise perturbation to fuse the spectral and spatial features, the mean and standard deviation of S are obtained. 2DGN generates diverse generated domain data containing spatial and spectral information, providing a variety of generated domain data for the semantic consistency discriminator to capture essential class-invariant features from the generated domain.

[0052] Through adversarial training, S 2 DGN can generate diverse generation domains, however, if S... 2 DGN imposes constraints, S 2 The class-based semantic information of the generated domain in DGN deviates from that of the source domain, causing the semantic consistency discriminator to fail to capture class-invariant features from the generated domain. Therefore, a multi-scale mutual information regularization network is used to constrain the class-based semantic information between the generated and source domains, maximizing the mutual information between the generated domain data and the original spectral patches at different spatial scales. Generated domain data With the original spectral block The lower bound of mutual information between them can be estimated using the InfoNCE loss, which can be formalized as:

[0053]

[0054] Where T is the temperature parameter, and N(i) is the set of negative samples. The semantic consistency between the generated domain data and the source domain spectral blocks is maximized by maximizing the similarity between the original spectral blocks and the corresponding generated domain data. To more accurately estimate the mutual information between the generated domain data and the original spectral blocks, a multi-scale mutual information regularization network is used for multi-scale mutual information estimation. For example... Figure 3 The diagram shown illustrates the structure of a multi-scale mutual information regularization network. This network first processes the original spectral block through a ReflectionPad2D layer. and generate domain data The surrounding boundaries are filled with a symmetrical mirror image to avoid losing information from the boundary cells, resulting in... and Then, the original spectral block is extended using the Conv7×7-InstanceNorm2d-ReLU module. and generate domain data Downsampling is performed, and the first downsampled features are obtained by passing through a Conv3×3 layer with padding and a stride of 1. and Then, through a learnable downsample layer... and Downsampling is performed, and the features are obtained by passing through a Conv3×3 layer with padding and a stride of 1. and Finally, through the learnable Downsample layer... and Downsampling is performed, and the features obtained from the third downsampling are obtained through the residual module. and It is worth noting that the Downsample layer contains a ReflectionPad2D layer and a 2D convolutional layer with a stride of 2 and no padding. This is relevant for generating domain data. With the original spectral block The multi-scale mutual information after passing through the multi-scale mutual information estimation network can be estimated by minimizing the following formula:

[0055]

[0056] in Represents a multi-scale mutual information regularized network, θ MIN These are the parameters of the multi-scale mutual information regularization network, N=4.

[0057] Under the constraint of multi-scale mutual information regularization, domain generation networks can generate diverse generated domains with class-domain semantic consistency. To assist the semantic consistency discriminator in learning class-domain semantic consistency from these diverse generated domains, a multi-domain hybridization module is designed to generate hybrid domains to help the semantic consistency discriminator learn class-domain semantic consistency. Specifically, the multi-domain hybridization module generates more diverse hybrid domains by linearly mixing the source domain and the generated domain in a random proportion. These hybrid domains contain information from the source domain but are not entirely identical to it, thereby further increasing the diversity of the domains and enhancing their class-domain semantic consistency with the source domain. This process can be formalized as follows:

[0058]

[0059] Where β is a random number randomly sampled from a uniform distribution of 0-1. This indicates a mixed domain.

[0060] Thanks to the inclusion of more source domain information in the hybrid domain, the hybrid domain obtained through the multi-domain hybridization module is used as a template distribution to make the distributions of the generator domain and the hybrid domain more similar. This allows the semantic consistency discriminator to capture class-domain semantic consistency information from the generator domain and the hybrid domain to learn class-cross-domain invariant information. The distributional similarity between the generator domain and the hybrid domain is measured using KL divergence, which can be expressed as:

[0061]

[0062] in This represents the semantic consistency discriminator, where θ is a parameter in the semantic consistency discriminator.

[0063] A semantic consistency discriminator is used to capture essential class-domain semantic consistency information from the generated domain and the hybrid domain. Its structure is as follows: Figure 2The semantic consistency discriminator section is shown in the diagram. To capture spectral and spatial features from the generative and hybrid domains to adapt to the three-dimensional structure of hyperspectral data, the semantic consistency discriminator includes spectral feature branches and spatial feature branches. The spectral feature branch contains convolutional layers, Conv1×1 layers with a stride of 1 and no padding, batch normalization, and ReLU. Convolution is used to extract spectral features. Since convolution cannot change the number of channels in the output features, a Conv1×1 layer with a kernel size of 1 is used to change the number of channels in the convolutional output features. The spatial feature branch adopts a Conv3×3-BatchNorm-ReLU structure. After spectral and spatial feature extraction, learnable weights are used... and Adaptive fusion of spectral and spatial features, it is worth noting that Finally, the adaptively fused spectral spatial features are flattened and fed into the classification head to obtain the probability that the sample belongs to each category.

[0064] For the generator domain and the hybrid domain, in order for the semantic consistency discriminator to learn class-domain semantic consistency, the categories determined by the semantic consistency discriminator for the generator domain data and the hybrid domain data should be as consistent as possible with the categories of the corresponding source domain data. Cross-entropy (CE) loss is used to measure the similarity between the categories determined by the semantic consistency discriminator for the generator domain data and the categories of the corresponding source domain data, which can be formalized as follows:

[0065]

[0066]

[0067] in It is a one-hot encoding of the label information, and the labels of the generated domain data and the mixed domain data are the same as the labels of the corresponding source domain data.

[0068] Step 106: Perform adversarial training on the spectral-space adversarial multi-domain synthesis network based on the pre-constructed loss function and source domain data to obtain the trained spectral-space adversarial multi-domain synthesis network.

[0069] The proposed S2AMSnet is a cross-scene hyperspectral image classification task that generalizes to a single source domain. The source domain containing N samples is... The target domain containing M samples is Where x i With y i These represent the spectral blocks and their corresponding category labels. For the single-source-domain generalization setting, only source-domain data is used. The model is trained so that it can generalize well to the target domain. It is worth noting that the target domain data... It is inaccessible during training. To enable models trained solely on source domain data to achieve excellent classification performance when directly transferred to the target domain scene, S... 2 AMSnet employs a two-step min-max adversarial training process to generate diverse generative domains using two structurally identical but parameter-distributed multi-domain generators. A semantic consistency discriminator is trained to capture essential class-semantic information from this diverse generative domain data. The data generated by the domain generators is further amplified through a multi-domain fusion module. To ensure class-semantic consistency between the generated and source domains, mutual information regularization constraints are applied. Diverse generative domain data with class-semantic similarity to the source domain effectively improves the model's performance in unseen domains.

[0070] Specifically, the two-step min-max adversarial training process includes:

[0071] For the semantic consistency discriminator, minimizing the cross-entropy loss (CE Loss) of the generated domain and the mixed domain relative to the corresponding labels of the source domain ensures that the generated domain and the mixed domain have similar class semantic information to the source domain. This can be formalized as follows:

[0072]

[0073] in and These are the generated domains generated by two domain generators with the same structure but different parameters. and They are and The mixed domain obtained by mixing the source domain at a random ratio.

[0074] To enable the semantic consistency discriminator to learn class-domain semantic consistency information from the generator domain and the mixed domain, the similarity between the corresponding distributions obtained by the semantic consistency discriminator from the generator domain and the mixed domain is minimized, allowing the semantic consistency discriminator to capture class-domain semantic consistency information from diverse domains. This can be expressed as:

[0075]

[0076] The overall optimization objective of the semantic consistency discriminator is:

[0077]

[0078] For domain generation networks and multi-scale mutual information regularization networks, the similarity between the corresponding distributions obtained by the semantic consistency discriminator of the generated domain and the mixed domain is maximized to enable the domain generation network to generate diverse domains. To ensure that the class-semantic information of the generated domain does not deviate as much as possible from the class-semantic information of the source domain, a multi-scale mutual information regularization constraint is imposed on the domain generation network. The multi-scale mutual information regularization constraint maximizes the multi-scale mutual information between the generated domain and the source domain to constrain the domain generation network so that the class-semantic information of the generated domain is similar to that of the source domain. This process can be expressed as:

[0079]

[0080] The overall optimization objective for optimizing the domain generation network and the multi-scale mutual information regularization network is:

[0081]

[0082] Where θ a and θ b These are the parameters in the generation network of two domains with the same structure but different parameters.

[0083] Step 108: Input the target domain data into the trained spectral-space adversarial multi-domain synthesis network to obtain the classification results corresponding to the spectral blocks to be classified.

[0084] In the aforementioned cross-scene hyperspectral image classification method, the robustness of the model to unseen domains is improved by expanding the diverse distribution of the source domain. Specifically, to generate diverse generated domains spatially and spectrally, a spectral-spatial domain generation network (S2DGN) is designed. A multi-domain hybrid module is used to further expand the diversity of generated domains and enhance their class-domain semantic consistency information. In addition, a multi-scale mutual information regularization network is used to constrain the S2DGN so that the essential class-domain semantic information of the generated domains does not deviate from the source domain. A semantic consistency discriminator with spectral-spatial feature extraction capabilities is used to capture class-domain semantic consistency information from the diverse generated domains to obtain cross-domain invariant knowledge. This method can improve the classification accuracy of cross-scene hyperspectral image classification tasks that generalize to single-source-domain applications.

[0085] In one embodiment, the step of adversarially training a spectral-space adversarial multi-domain synthesis network based on a pre-constructed loss function and source domain data to obtain a trained spectral-space adversarial multi-domain synthesis network includes: inputting the original spectral blocks from the source domain data into a first branch and a second branch, perturbing the original spectral blocks for adversarial training; calculating a first loss function based on the similarity between the classification results of the generated domain data and the mixed domain data in the two branches of the first and second branches and the corresponding labels of the source domain data; and calculating a second loss function based on the distribution similarity between the generated domain data and the mixed domain data in the first and second branches. The loss function is calculated based on the multi-scale mutual information between the generated domain data and the source domain data in the first and second branches. The semantic consistency discrimination network is trained by minimizing the first and second loss functions to obtain the trained semantic consistency discrimination network. The domain generation network and the multi-scale mutual information regularization network are trained by maximizing the second and third loss functions to obtain the trained domain generation network and the multi-scale mutual information regularization network. Based on the trained semantic consistency discrimination network, domain generation network, and multi-scale mutual information regularization network, the trained spectral space adversarial multi-domain synthesis network is obtained.

[0086] In one embodiment, the first loss function is:

[0087]

[0088] in, For the first loss function, Represents cross-entropy loss, It is the one-hot encoding of the source domain data tag information. Generate the domain data output by the network for the domain in the first branch. Generate the domain data output by the network for the domain in the second branch. yes Mixed domain data obtained by mixing the source domain data at a random ratio yes Mixed domain data is obtained by mixing the source domain data at a random ratio.

[0089] In one embodiment, the second loss function is:

[0090]

[0091] in, For the second loss function, Denotes KL divergence, Generate the domain data output by the network for the domain in the first branch. Generate the domain data output by the network for the domain in the second branch. yes Mixed domain data obtained by mixing the source domain data at a random ratio yes Mixed domain data is obtained by mixing the source domain data at a random ratio.

[0092] In one embodiment, the third loss function is:

[0093]

[0094] in, For the third loss function, Indicates InfoNCE loss, Represents a multi-scale mutual information regularized network, θ MIN These are the parameters of a multi-scale mutual information regularization network. For the original spectral block, Generate the domain data output by the network for the domain in the first branch. Generate the domain data output by the network for the domain in the second branch.

[0095] In one embodiment, such as Figure 3 As shown, a schematic diagram of a multi-scale mutual information regularization network is provided. The multi-scale mutual information regularization network includes a mirror filling module, a first downsampling module, a second downsampling module, and a third downsampling module connected sequentially. The mirror filling module performs mirror filling on the original spectral block and the generated domain data respectively to obtain the corresponding extended features. The first downsampling module downsamples the extended features of the original spectral block and the generated domain data respectively to obtain the corresponding first downsampling features. The second downsampling module downsamples the first downsampling features of the original spectral block and the generated domain data respectively to obtain the corresponding second downsampling features. The third downsampling module downsamples the second downsampling features of the original spectral block and the generated domain data respectively to obtain the corresponding third downsampling features. Multi-scale mutual information estimation is performed based on the extended features, first downsampling features, second downsampling features, and third downsampling features of the original spectral block and the generated domain data to obtain the multi-scale mutual information of the original spectral block and the generated domain data.

[0096] In one embodiment, the first downsampling module includes a first convolutional layer, a regularization layer, an activation function layer, and a second convolutional layer; the second downsampling module includes a regularization layer, an activation function layer, a downsampling layer, and a second convolutional layer; the third downsampling module includes a regularization layer, an activation function layer, a downsampling layer, a residual block, and a concatenation layer, wherein the concatenation layer is used to concatenate the outputs of the downsampling layer and the residual block in the third downsampling module to obtain the third downsampling feature.

[0097] In one specific embodiment, in order to verify S 2The effectiveness of AMSnet for cross-scene hyperspectral image classification tasks generalizing to a single source domain was tested on the Pavia University and Pavia Center datasets. In the experiments, the Pavia University dataset was used as the source domain and the Pavia Center dataset as the target domain. Seven state-of-the-art transfer learning methods were used in conjunction with AMSnet. 2 AMSnet performance comparisons were conducted, including four domain-adaptive methods: Multi-Representation Adaptive Network (MRAN), Deep Subdomain Adaptive Network (DSAN), Domain Adaptive Network Based on Confidence Learning (CLDA), and Unsupervised Domain Adaptive Network Based on Supervised Contrastive Learning (SCLUDA). Three single-source domain generalization methods were also included: Learning Diversity Single-Source Domain Generalization (L2D), Progressive Domain Extension Network (PDEN), and Single-Source Domain Extension Network (SDEnet). To evaluate the performance of cross-scene classification in hyperspectral images, in addition to qualitative analysis based on classification maps, class accuracy (CA), overall accuracy (OA), and Kappa coefficient (κ) were used as quantitative evaluation metrics to quantitatively study the performance of the proposed methods compared to the contrasting methods.

[0098] For sample setup during model training, domain-adaptive methods such as MRAN, DSAN, CLDA, and SCLUDA select 80% of the labeled data from the source domain and their corresponding labels, along with all target domain data, for model training. The remaining labeled data from the source domain is used as a validation set to verify the model's performance. For L2D, PDEN, SDEnet, and S... (the last sentence appears to be incomplete and possibly refers to a different method or approach). 2 AMSnet uses only 80% of the labeled source domain data and their corresponding labels for model training, with the remaining 20% ​​serving as a validation set. For the parameter settings of the comparison methods, MRAN, DSAN, and L2D use ResNet18 as the feature extractor, employing 32×32 spectral patches as input to accommodate the ResNet18 input size. The parameter settings for CLDA, SCLUDA, PDEN, and SDEnet follow the optimal settings reported in the original paper. For the cross-scene hyperspectral image classification task in the experiments, S... 2 The learning rate in AMSnet is set to 1e-3. When using a multi-scale mutual information regularization network to estimate the mutual information between the source and generator domains by optimizing the InfoNCE loss function, the temperature coefficient in the InfoNCE loss function is set to 0.07, and the number of downsampling iterations in the multi-scale mutual information regularization network is set to 3. For the cross-scene hyperspectral image classification task at Pavia University and Pavia Center, S 2The input spectral block size for AMSnet is 9×9.

[0099] To reduce the data distribution differences caused by the different scales of data in the source and target domains and enhance the stability of the model, the data in both domains are first normalized during training. Furthermore, for reliable evaluation, both the proposed method and the comparison method are tested ten times on a cross-scene hyperspectral image classification task, and the results are averaged to evaluate the performance of the proposed method.

[0100] First, visually speaking, S 2 A qualitative evaluation was performed on the classification results obtained by AMSnet and seven state-of-the-art methods in the target scene. Figure 4 The image shows the classification results of S2AMSnet and the comparison method on the target scene Pavia Center. Unlabeled pixels are displayed as background, and labeled pixels are displayed as predicted categories by the model. 2 The classification results from AMSnet are closer to the actual distribution of ground features in the target scene compared to the comparative methods. For example, the third category (bricks) in Pavia Center shows a significant improvement over the comparative methods used in the experiment.

[0101] To further investigate S 2 AMSnet was quantitatively evaluated against competing methods. Table 1 summarizes the statistics of S. 2 AMSnet's CA, OA, and κ values ​​were compared with seven state-of-the-art methods on the target scene, Pavia Center:

[0102] Table 1 S 2 Classification results (mean ± standard deviation) of AMSnet and contrasting methods on the Pavia Center dataset for the target scene.

[0103]

[0104] Quantitative evaluation metrics show that, for DA-based methods, SCLUDA achieved the highest OA in the target scene. This is attributed to the domain similarity loss function based on a single classification task designed in SCLUDA to reduce inter-domain differences and the target domain sample selection strategy based on confidence learning for fine-tuning the DA model. For single-source domain generalization methods, in target scenes with more spectral bands, Pavia Center, L2D, PDEN, and SDEnet have weaker OA than SCLUDA. This indicates that L2D and PDEN, which only focus on the spatial diversity of images, cannot adapt well to the inter-domain spectral drift caused by high-dimensional hyperspectral data. While SDEnet, designed for hyperspectral images, expands the distribution of the source domain from the perspectives of spatial and spectral information, its lack of constraints on the spectral information of the expanded domain leads to deviations in class domain semantic information. Single-source domain generalization methods, which expand the diversity of the domain to enhance the model's generalization ability, show better performance on complex inter-domain shifts without using target domain data.

[0105] S is designed for cross-scene hyperspectral image classification tasks that generalize to single source domains. 2 AMSnet achieved competitive performance across all target scenarios in the experiments. Compared to the best-performing domain adaptation method, S... 2 AMSnet improves OA by 2.22% on the target scenario, Pavia Center. Compared to the best-performing single-source domain generalization method, S... 2 AMSnet improved OA by 2.43% in the target scenario. This indicates that S 2 The spectral spatial domain generation network designed in AMSnet can effectively generate diverse generation domains from spatial and spectral dimensions. The multi-scale mutual information regularization network can effectively constrain the class semantic information of the generation domain from the spatial-spectral feature dimension. The semantic consistency discriminator can successfully capture class domain semantic consistency information from diverse generation domains with essential class semantic information, thereby achieving more robust generalization performance.

[0106] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0107] In one embodiment, such as Figure 5 As shown, a cross-scene hyperspectral image classification device is provided, comprising:

[0108] The data acquisition module 502 is used to acquire source domain data and target domain data; the source domain data includes the original spectral blocks and their corresponding category labels; the target domain data includes the spectral blocks to be classified and their corresponding category labels.

[0109] Network construction module 504 is used to construct a spectral-spatial adversarial multi-domain synthesis network. The multi-domain synthesis network includes two branches with identical network structures and identical input spectral blocks: a first branch and a second branch. Each branch includes a domain generation network, a multi-scale mutual information regularization network, a multi-domain mixing module, and a semantic consistency discrimination network. The parameters of the domain generation network in the two branches of the multi-domain synthesis network are not shared, while the parameters of the semantic consistency discrimination network are shared. The domain generation network is used to extract the spectral and spatial information of the original spectral block to obtain spectral and spatial features, fuse the spectral and spatial features to obtain spectral-spatial fusion features, and perform adaptive instance normalization, activation, and decoding on the spectral-spatial fusion features to obtain output features. The output features and the original spectral block are adaptively fused to obtain generated domain data. The multi-scale mutual information regularization network is used to calculate the multi-scale mutual information of the original spectral block and the generated domain data. The multi-domain mixing module is used to mix the generated domain data and the source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network is used to extract spectral and spatial features, fuse features, flatten, and classify the mixed domain data and the generated domain data to obtain the corresponding classification results.

[0110] The network training module 506 is used to perform adversarial training on the spectral-space adversarial multi-domain synthesis network based on a pre-constructed loss function and source domain data, so as to obtain a trained spectral-space adversarial multi-domain synthesis network.

[0111] The target domain classification module 508 is used to input the target domain data into the trained spectral spatial adversarial multi-domain synthesis network to obtain the classification result corresponding to the spectral block to be classified.

[0112] Specific limitations regarding the cross-scene hyperspectral image classification device can be found in the limitations of the cross-scene hyperspectral image classification method above, and will not be repeated here. Each module in the aforementioned cross-scene hyperspectral image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0113] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a cross-scene hyperspectral image classification method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0114] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0116] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. A cross-scene hyperspectral image classification method, characterized in that, The method includes: Acquire source domain data and target domain data; the source domain data includes the original spectral blocks and their corresponding category labels; the target domain data includes the spectral blocks to be classified and their corresponding category labels. A spectral-spatial adversarial multi-domain synthesis network is constructed. The multi-domain synthesis network includes two branches with identical network structures and identical input spectral blocks: a first branch and a second branch. Each branch includes a domain generation network, a multi-scale mutual information regularization network, a multi-domain mixing module, and a semantic consistency discrimination network. The parameters of the domain generation network in the two branches of the multi-domain synthesis network are not shared, while the parameters of the semantic consistency discrimination network are shared. The domain generation network is used to extract the spectral and spatial information of the original spectral block to obtain spectral and spatial features. These spectral and spatial features are then fused to obtain spectral-spatial fusion features. Adaptive instance normalization, activation, and decoding are performed on the spectral-spatial fusion features sequentially to obtain output features. The output features and the original spectral block are then adaptively fused to obtain generated domain data. The multi-scale mutual information regularization network is used to calculate the multi-scale mutual information between the original spectral block and the generated domain data. The multi-domain mixing module is used to mix the generated domain data and the source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network is used to extract spectral and spatial features, fuse features, flatten, and classify the mixed domain data and the generated domain data to obtain corresponding classification results. The spectral-space adversarial multi-domain synthesis network is adversarially trained based on the pre-constructed loss function and the source domain data to obtain the trained spectral-space adversarial multi-domain synthesis network. The target domain data is input into a trained spectral-space adversarial multi-domain synthesis network to obtain the classification result corresponding to the spectral block to be classified.

2. The method according to claim 1, characterized in that, The steps of performing adversarial training on the spectral-space adversarial multi-domain synthesis network based on a pre-constructed loss function and the source domain data to obtain the trained spectral-space adversarial multi-domain synthesis network include: The original spectral block in the source domain data is input into the first branch and the second branch, and the original spectral block is perturbed to perform adversarial training. The first loss function is calculated based on the similarity between the classification results of the generated domain data and the mixed domain data in the two branches of the first branch and the second branch and the corresponding labels of the source domain data; The second loss function is calculated based on the distribution similarity between the generated domain data and the mixed domain data in the first branch and the second branch; The third loss function is calculated based on the multi-scale mutual information between the generated domain data and the source domain data in the first branch and the second branch; The semantic consistency discrimination network is trained by minimizing the first loss function and the second loss function to obtain the trained semantic consistency discrimination network. The domain generation network and the multi-scale mutual information regularization network are trained by maximizing the second loss function and the third loss function, resulting in a trained domain generation network and a multi-scale mutual information regularization network. Based on the trained semantic consistency discrimination network, domain generation network, and multi-scale mutual information regularization network, a trained spectral space adversarial multi-domain synthesis network is obtained.

3. The method according to claim 2, characterized in that, The first loss function is: in, For the first loss function, Represents cross-entropy loss, It is the one-hot encoding of the source domain data tag information. Generate the domain data output by the network for the domain in the first branch. Generate the domain data output by the domain generation network in the second branch. yes Mixed domain data obtained by mixing the source domain data at a random ratio yes Mixed domain data is obtained by mixing the source domain data at a random ratio.

4. The method according to claim 2, characterized in that, The second loss function is: in, For the second loss function, Denotes KL divergence, Generate the domain data output by the network for the domain in the first branch. Generate the domain data output by the domain generation network in the second branch. yes Mixed domain data obtained by mixing the source domain data at a random ratio yes Mixed domain data is obtained by mixing the source domain data at a random ratio.

5. The method according to claim 2, characterized in that, The third loss function is: in, For the third loss function, Indicates InfoNCE loss, Represents a multi-scale mutual information regularized network, θ MIN These are the parameters of a multi-scale mutual information regularization network. For the original spectral block, Generate the domain data output by the network for the domain in the first branch. Generate the domain data output by the network for the domain in the second branch.

6. The method according to claim 1, characterized in that, The multi-scale mutual information regularization network includes a mirror filling module, a first downsampling module, a second downsampling module, and a third downsampling module connected in sequence. The mirror filling module performs mirror filling on the original spectral block and the generated domain data respectively to obtain the corresponding extended features; The first downsampling module downsamples the extended features of the original spectral block and the generated domain data respectively to obtain the corresponding first downsampling features. The second downsampling module downsamples the first downsampling features of the original spectral block and the generated domain data respectively to obtain the corresponding second downsampling features. The third downsampling module downsamples the second downsampling features of the original spectral block and the generated domain data respectively to obtain the corresponding third downsampling features. Multi-scale mutual information estimation is performed based on the extended features, first downsampling features, second downsampling features, and third downsampling features of the original spectral block and the generated domain data to obtain the multi-scale mutual information of the original spectral block and the generated domain data.

7. The method according to claim 6, characterized in that, The first downsampling module includes a first convolutional layer, a regularization layer, an activation function layer, and a second convolutional layer; the second downsampling module includes a regularization layer, an activation function layer, a downsampling layer, and a second convolutional layer; the third downsampling module includes a regularization layer, an activation function layer, a downsampling layer, a residual block, and a concatenation layer, wherein the concatenation layer is used to concatenate the outputs of the downsampling layer and the residual block in the third downsampling module to obtain the third downsampling feature.

8. A cross-scene hyperspectral image classification device, characterized in that, The device includes: The data acquisition module is used to acquire source domain data and target domain data; the source domain data includes raw spectral blocks and their corresponding category labels; the target domain data includes spectral blocks to be classified and their corresponding category labels. The network construction module is used to construct a spectral-spatial adversarial multi-domain synthesis network. The multi-domain synthesis network includes a first branch and a second branch with identical network structures and input spectral blocks. Each branch includes a domain generation network, a multi-scale mutual information regularization network, a multi-domain hybridization module, and a semantic consistency discrimination network. The domain generation network parameters are not shared between the two branches, while the semantic consistency discrimination network parameters are shared. The domain generation network is used to extract the spectral and spatial information of the original spectral block to obtain spectral features and spatial features, and then fuses these features to obtain spectral-spatial fusion features. The spectral-spatial fusion features are sequentially subjected to adaptive instance normalization, activation, and decoding to obtain output features. The output features and the original spectral block are then adaptively fused to obtain the generated domain data. The multi-scale mutual information regularization network is used to calculate the multi-scale mutual information between the original spectral block and the generated domain data. The multi-domain mixing module is used to mix the generated domain data and the source domain data in a random ratio to obtain mixed domain data. The semantic consistency discrimination network is used to extract spectral and spatial features, fuse features, flatten, and classify the mixed domain data and the generated domain data respectively to obtain corresponding classification results. The network training module is used to perform adversarial training on the spectral-space adversarial multi-domain synthesis network based on a pre-constructed loss function and the source domain data, so as to obtain a trained spectral-space adversarial multi-domain synthesis network. The target domain classification module is used to input the target domain data into the trained spectral spatial adversarial multi-domain synthesis network to obtain the classification result corresponding to the spectral block to be classified.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images