Hyperspectral image cross-domain classification method and device suitable for complex ground feature scene

Abstract

The invention relates to a hyperspectral image cross-domain classification method and device suitable for a complex ground object scene, and the method comprises the steps: obtaining a hyperspectral image, and independently extracting features from the space, spectrum and time / scene dimensions of the hyperspectral image; performing interaction and fusion operation on the features of each dimension; matching the feature spaces of the source domain and the target domain; training a hyperspectral image cross-domain classification model, and performing joint optimization on the model through a multi-task loss function; and predicting the target domain data to obtain a final classification result. The method has the beneficial effects that the challenges caused by feature distribution differences under cross-time-phase and cross-scene conditions are successfully dealt with by performing deep adaptive optimization on multiple levels of domain-level and class-level feature alignment. The algorithm not only improves the global consistency of the feature space of the source domain and the target domain, but also ensures the accurate alignment of different types of features through a refined alignment strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image classification, and more specifically, to a hyperspectral image cross-domain classification method and device applicable to complex land object scenes. Background Art

[0002] In the field of remote sensing image classification, hyperspectral images provide rich information about land objects due to their high spatial and spectral resolution, and are widely used in land cover classification, environmental monitoring, and resource management. However, hyperspectral images taken at different times or in different scene environments often show obvious differences in feature distribution due to various factors such as shooting conditions, climate change, and seasonal differences. Especially in complex environments such as wetlands, these differences are more significant due to the high heterogeneity of land objects and the complexity of spectral features. This feature inconsistency greatly affects the generalization ability of traditional classification models, resulting in poor performance of the model when processing hyperspectral images across time or scenes.

[0003] Traditional hyperspectral image classification methods usually rely on source domain data for model training and assume that the trained model can be seamlessly applied to target domain data. However, due to the difference in feature distribution between the source domain and the target domain, this assumption is difficult to hold in practical applications. Especially in complex environments such as wetlands, this problem is more significant due to the diversity of surface cover types and drastic changes in environmental conditions (such as light and humidity), resulting in a significant decrease in classification accuracy. To address these challenges, domain adaptation (DA) technology has been introduced to improve the performance of the model on target domain data. Domain adaptation aims to improve the generalization ability of the model by reducing the difference in feature distribution between the source domain and the target domain. This usually involves techniques such as feature map alignment, recalibration, and sample selection to enable the trained model to better adapt to the feature distribution of the target domain.

[0004] With the rapid development of deep learning technology, many advanced domain adaptation methods have been proposed and applied to hyperspectral image classification tasks. These methods include generative adversarial networks (GANs), domain adversarial neural networks (DANNs), and adaptive feature selection techniques, which aim to align the feature distributions of the source domain and the target domain through different strategies. For example, generative adversarial networks generate pseudo source domain samples of target domain samples through adversarial training, thereby improving the model's adaptability to the target domain. This method generates pseudo samples with similar distribution to the target domain, enabling the classifier to better learn the characteristics of the target domain, thereby enhancing the generalization ability of the model. Domain adversarial neural networks directly align the feature distributions of the source domain and the target domain by introducing domain adversarial loss. The domain adversarial loss trains the network's feature extractor to make the features of the source domain and the target domain as similar as possible in the shared space, thereby reducing the difference between the domains. Adaptive feature selection techniques reduce the impact of feature differences on the classification effect by selecting the features that are most useful for the classification task. This method reduces the feature dimension and improves the classification performance by filtering out features that contribute less to the classification task. Although these technologies have made some progress in the domain adaptation task in hyperspectral image classification, there are still some challenges. Existing methods may have limited performance when dealing with complex environmental conditions and highly variable spectral features. For example, when dealing with complex environments such as wetlands, these methods may have difficulty coping with the high variability of spectral features. Although generative adversarial networks and domain adversarial neural networks can alleviate the difference in feature distribution between the source and target domains to a certain extent, their alignment effects may still be affected by actual environmental changes. In addition, although adaptive feature selection techniques can reduce feature differences, they usually require a lot of computing resources and time for feature screening, and their effectiveness is also limited by the quality of the selected features and the applicability of the screening method. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and to propose a hyperspectral image cross-domain classification method and device suitable for complex land object scenes.

[0006] First, a cross-domain classification method for hyperspectral images suitable for complex ground scenes is provided, including:

[0007] S1, obtain hyperspectral images and independently extract features from the spatial, spectral and temporal / scene dimensions of the hyperspectral images;

[0008] S2, interact and fuse the features of each dimension;

[0009] S3, by optimizing the feature distribution mapping and combining the dual constraints of domain alignment and class alignment, the feature space of the source domain and the target domain are matched;

[0010] S4. Use labeled source domain data and unlabeled target domain data to train a cross-domain classification model for hyperspectral images, and jointly optimize the model through a multi-task loss function;

[0011] S5. Use the trained model to predict the target domain data and obtain the final classification result.

[0012] Preferably, in S1, the mathematical expression for independently extracting features from the spatial, spectral and temporal / scene dimensions of the hyperspectral image is:

[0013]

[0014] Where X∈R H×W×B×T is the input hyperspectral image, H, W and B represent the height, width and number of bands of the image respectively, T represents the time or scene dimension, H ′ , W ′ , B ′ and T ′ They represent the dimensions corresponding to the feature extraction of hyperspectral images in different dimensions, L h , L s , L t are the number of feature extraction layers for spatial, spectral, and temporal / scene dimensions, respectively. represents the weight matrix of the lth layer corresponding to the spatial, spectral, and temporal / scene dimensions, are the bias terms corresponding to each layer, σ(·) is the activation function, * represents the convolution operation, and F spatial 、F spectral and F temporal Represent the extracted spatial, spectral and temporal / scene features, respectively.

[0015] Preferably, in S2, the mathematical expression for performing interaction and fusion operations on features of each dimension is:

[0016]

[0017] Where i, j represents the index position of the feature, σ(·) represents the activation function used to perform nonlinear transformation on the input feature interaction value, and W att is the attention weight matrix, represents the dot product operation of the vector, Represents the multidimensional concatenation vector of the i-th feature, A ij represents the interaction value between the i-th and j-th features, measuring their association strength under the attention mechanism; F fusion It represents the features obtained by interactively fusing spatial, spectral and temporal / scene features. The coefficients α, β and γ are used to control the weights of features in each dimension, satisfying α+β+γ=1.

[0018] Preferably, in S3, the mathematical expression for achieving the matching of the feature space of the source domain and the target domain is:

[0019] L domain_align =‖M(F s )-F T ‖ 2

[0020]

[0021] Among them, F s and F T are the source domain and target domain features, and represents the nth category feature of the source domain and the target domain, M represents the number of categories, M(·) is the feature mapping function, ‖·‖ 2 represents the Euclidean norm, L domain_align is the domain alignment loss between the source domain and the target domain, L class_align is the category alignment loss between the source domain and the target domain.

[0022] Preferably, in S4, the source domain classification loss is calculated, and the model parameters are optimized by combining the source domain classification loss and feature alignment loss. The mathematical expression is:

[0023]

[0024] L=L s +λ1L domain_align +λ2L class_align

[0025] Among them, L s is the source domain classification loss calculated using the cross entropy loss function, y i,c is the true label of the i-th sample in category c, is the prediction probability, L is the final comprehensive optimization objective function, λ1 and λ2 are adjustment parameters used to control the weights of different loss items.

[0026] Preferably, in S5, the target domain data is input into the trained classification model, and its mathematical expression is:

[0027]

[0028] in, is the prediction result of the target domain data, W and b are model parameters.

[0029] In a second aspect, a hyperspectral image cross-domain classification device applicable to complex ground scene is provided, which is used to execute any method described in the first aspect, including:

[0030] An acquisition module is used to acquire hyperspectral images and independently extract features from the spatial, spectral, and temporal / scene dimensions of the hyperspectral images;

[0031] The interactive fusion module is used to interact and fuse the features of each dimension;

[0032] The matching module is used to achieve the matching of the source domain and target domain feature spaces by optimizing the feature distribution mapping and combining the dual constraints of domain alignment and class alignment;

[0033] The training module is used to train the cross-domain classification model of hyperspectral images using labeled source domain data and unlabeled target domain data, and jointly optimize the model through a multi-task loss function;

[0034] The prediction module is used to use the trained model to predict the target domain data and obtain the final classification result.

[0035] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0036] In a fourth aspect, an electronic device is provided, including:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0039] The beneficial effects of the present invention are:

[0040] 1. This paper innovatively designs a multi-dimensional mutual feedback feature deep mining network (MRFDMNet) to achieve parallel feature extraction of the four dimensions of space, spectrum, and time / scene for hyperspectral images, greatly improving the efficient mining and expression of multi-dimensional information. The network combines deep convolutional layers and residual structures to capture the geometric features of objects in the spatial dimension, extract the spectral characteristics unique to materials in the spectral dimension, and capture the dynamic changes of objects in the time and scene dimensions, thereby enhancing the expression accuracy and detail perception ability of hyperspectral images in complex environments.

[0041] 2. This invention breaks through the limitations of traditional feature fusion methods by introducing an innovative multi-dimensional interactive fusion strategy, and significantly enhances the synergy and information integration capabilities between features of different dimensions. This mechanism optimizes the interactive expression of spatial, spectral, and temporal / scene features, improves global feature consistency, and enables the model to have stronger multi-dimensional information perception capabilities. In classification tasks in complex environments, the model accuracy and robustness have been significantly improved.

[0042] 3. The multi-level alignment consistency algorithm (MLACA) proposed in this paper successfully copes with the challenges brought by the differences in feature distribution under cross-temporal and cross-scenario conditions by performing deep adaptive optimization at multiple levels of domain-level and class-level feature alignment. This algorithm not only improves the global consistency of the feature space of the source domain and the target domain, but also ensures the accurate alignment of features of different categories through a refined alignment strategy, greatly enhancing the generalization ability and robustness of the model in complex land environments, and is suitable for remote sensing image classification tasks in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a cross-domain classification method for hyperspectral images applicable to complex land object scenes provided by the present invention;

[0044] Figure 2 Schematic diagram of two sets of hyperspectral cross-temporal domain adaptation data for Hangzhou Bay and Yancheng provided by the present invention;

[0045] Figure 3 Schematic diagram of the Yellow River Estuary-Yancheng hyperspectral cross-scene domain adaptation data provided by the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0047] Embodiment 1:

[0048] In order to solve the problems of the prior art, Example 1 of the present application provides a cross-domain classification method for hyperspectral images suitable for complex land object scenes, aiming to effectively solve the classification problems caused by differences in feature distribution of hyperspectral images under different time and scene environments. This technology aims to deal with the problem of inconsistency of hyperspectral image features caused by changes in the environment or conditions by improving feature extraction and domain alignment strategies. The experiment was carried out on a self-constructed wetland hyperspectral dataset, and the results showed that the method showed excellent performance and stability in a variety of classification tasks, demonstrating its broad application prospects in the fields of environmental monitoring and resource management. This research result is not only of great significance to the development of hyperspectral remote sensing image classification technology, but also provides an effective solution for environmental change analysis and surface cover monitoring in complex environments.

[0049] Specifically, Figure 1 As shown, the method includes:

[0050] S1. Acquire hyperspectral images and independently extract features from the spatial, spectral, and temporal / scene dimensions of the hyperspectral images.

[0051] In S1, a parallel four-dimensional feature extraction module is designed to fully explore the multi-dimensional information of the data. The purpose of this step is to ensure that the model can fully capture the spatial structure, spectral characteristics and time or scene change information of complex objects by performing multi-dimensional feature extraction on hyperspectral images, and provide accurate and comprehensive input for subsequent feature fusion and classification.

[0052] The mathematical expression for independently extracting features from the spatial, spectral, and temporal / scene dimensions of hyperspectral images is:

[0053]

[0054] Where X∈R H×W×B×T is the input hyperspectral image, H, W and B represent the height, width and number of bands of the image respectively, T represents the time or scene dimension, H ′ , W ′ , B ′ and T ′ They represent the dimensions corresponding to the feature extraction of hyperspectral images in different dimensions, L h , L s , L t are the number of feature extraction layers for spatial, spectral, and temporal / scene dimensions, respectively. represents the weight matrix of the lth layer corresponding to the spatial, spectral, and temporal / scene dimensions, are the bias terms corresponding to each layer, σ(·) is the activation function, * represents the convolution operation, F spatial 、F spectral and F temporal Represent the extracted spatial, spectral and temporal / scene features, respectively.

[0055] S2. Interact and fuse the features of each dimension.

[0056] In S2, an innovative interactive feature fusion mechanism is introduced to interactively operate the features of each dimension to achieve efficient fusion and enhancement of information. The purpose of this step is to enhance the complementarity and synergy between information by interactively fusing the features of each dimension, thereby enhancing the richness and robustness of feature representation and enabling the model to better cope with classification tasks in complex environments. The mathematical expression for the interactive and fusion operations on the features of each dimension is:

[0057]

[0058] Among them, i,j represents the index position of the feature, Watt is the attention weight matrix, represents the dot product operation of the vector, represents the multidimensional concatenated vector of the i-th feature, σ(·) represents the activation function used to perform nonlinear transformation on the input feature interaction value, and A ij represents the interaction value between the i-th and j-th features, measuring their association strength under the attention mechanism; F fusion It represents the features obtained by interactively fusing spatial, spectral and temporal / scene features. The coefficients α, β and γ are used to control the weights of features in each dimension, satisfying α+β+γ=1.

[0059] S3. Aiming at the differences in feature distribution between different times and scenes, an efficient adaptive alignment strategy is designed to achieve accurate matching of the feature space of the source domain and the target domain by optimizing the feature distribution mapping and combining the dual constraints of domain alignment and class alignment.

[0060] This step aims to reduce the feature distribution deviation under different time and scene conditions by adaptively aligning the source domain and target domain features, ensuring that the feature space can maintain global consistency and intra-class aggregation across time and scenes, thereby improving the model's feature expression ability and classification accuracy in a changing environment.

[0061] In S3, the mathematical expression for achieving the matching of the source domain and the target domain feature space is:

[0062] L domain_align =‖M(F s )-F T ‖ 2

[0063]

[0064] Among them, F s and F T are the source domain and target domain features, and represents the nth category feature of the source domain and the target domain, N represents the number of categories, M(·) is the feature mapping function, ‖·‖ 2 represents the Euclidean norm, L domain_align is the domain alignment loss between the source domain and the target domain, L class_align The category alignment loss S4 between the source domain and the target domain is used, the hyperspectral image cross-domain classification model is trained using labeled source domain data and unlabeled target domain data, and the model is jointly optimized through a multi-task loss function.

[0065] S4. Use labeled source domain data and unlabeled target domain data to train a cross-domain classification model for hyperspectral images, and jointly optimize the model through a multi-task loss function.

[0066] The purpose of this step is to use labeled source domain data for supervised learning, and unlabeled target domain data for unsupervised alignment, combined with multi-task loss functions, to optimize model parameters, thereby achieving comprehensive improvement in feature alignment and classification performance between the source and target domains.

[0067] S5. Use the trained model to predict the target domain data and obtain the final classification result.

[0068] The purpose of this step is to accurately classify the target domain data through the trained model, verify the generalization ability of the model under cross-temporal and cross-scenario conditions, and obtain the final high-precision classification results.

[0069] Embodiment 2:

[0070] Based on Example 1, Example 2 of the present application provides a more specific cross-domain classification method for hyperspectral images applicable to complex ground sceneries, including:

[0071] S1. Acquire hyperspectral images and independently extract features from the spatial, spectral, and temporal / scene dimensions of the hyperspectral images.

[0072] S2. Interact and fuse the features of each dimension.

[0073] S3. By optimizing the feature distribution mapping and combining the dual constraints of domain alignment and class alignment, the feature space matching of the source domain and the target domain is achieved.

[0074] S4. Use labeled source domain data and unlabeled target domain data to train a cross-domain classification model for hyperspectral images, and jointly optimize the model through a multi-task loss function.

[0075] In S4, the source domain classification loss is calculated, and the model parameters are optimized by combining the source domain classification loss and feature alignment loss. The mathematical expression is:

[0076]

[0077] L=L s +λ1L domain_align +λ2L class_align

[0078] Among them, L s is the source domain classification loss calculated using the cross entropy loss function, y i,c is the true label of the i-th sample in category c, is the prediction probability, L is the final comprehensive optimization objective function, which combines the source domain classification loss, domain alignment loss and category alignment loss, λ1 and λ2 are adjustment parameters used to control the weights of different loss items.

[0079] S5. Use the trained model to predict the target domain data and obtain the final classification result.

[0080] In S5, the target domain data is input into the trained classification model, and its mathematical expression is:

[0081]

[0082] in, is the prediction result of the target domain data, W and b are model parameters.

[0083] The classification results of the proposed method were compared with those of the other four mainstream methods by using a self-constructed hyperspectral dataset (images from the ZY1-02D ​​satellite). The specific results are shown in Tables 1-3. The results show that on the Hangzhou Bay cross-temporal data dataset, the overall accuracy (OA) of the proposed method is 87.40%, and the Kappa coefficient is 0.8324; on the Yancheng cross-temporal data dataset, the overall accuracy (OA) of the proposed method is 82.07%, and the Kappa coefficient is 0.7662; on the Yellow River Estuary-Yancheng cross-scene dataset, the overall accuracy (OA) of the proposed method is 67.82%, and the Kappa coefficient is 0.6053. The results show that on these three sets of hyperspectral domain adaptation datasets, the proposed method is significantly superior to other comparison methods, showing strong cross-temporal and cross-scene adaptability, further verifying its effectiveness in complex remote sensing image classification tasks.

[0084] Table 1 Experimental results of different domain adaptation methods on the cross-temporal data of Hangzhou Bay (2021.01.29→2022.01.03)

[0085]

[0086] Table 2 Experimental results of different domain adaptation methods on Yancheng cross-temporal data (2022.02.24→2023.01.20)

[0087]

[0088] Table 3 Experimental results of different domain adaptation methods on the Yellow River Estuary-Yancheng cross-scene data (2020.06.28→2020.09.06)

[0089]

[0090]

[0091] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0092] Embodiment 3:

[0093] On the basis of embodiments 1 and 2, embodiment 3 of the present application provides a hyperspectral image cross-domain classification device suitable for complex ground scene, including:

[0094] An acquisition module is used to acquire hyperspectral images and independently extract features from the spatial, spectral, and temporal / scene dimensions of the hyperspectral images;

[0095] The interactive fusion module is used to interact and fuse the features of each dimension;

[0096] The matching module is used to achieve the matching of the source domain and target domain feature spaces by optimizing the feature distribution mapping and combining the dual constraints of domain alignment and class alignment;

[0097] The training module is used to train the cross-domain classification model of hyperspectral images using labeled source domain data and unlabeled target domain data, and jointly optimize the model through a multi-task loss function;

[0098] The prediction module is used to use the trained model to predict the target domain data and obtain the final classification result.

[0099] Specifically, the device provided in this embodiment is a device corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A cross-domain classification method for hyperspectral images applicable to complex terrain scenes, characterized by: include: S1, obtain hyperspectral images and independently extract features from the spatial, spectral and temporal / scene dimensions of the hyperspectral images; S2, interact and fuse the features of each dimension; S3, by optimizing the feature distribution mapping and combining the dual constraints of domain alignment and class alignment, the feature space of the source domain and the target domain are matched; S4. Use labeled source domain data and unlabeled target domain data to train a cross-domain classification model for hyperspectral images, and jointly optimize the model through a multi-task loss function; S5. Use the trained model to predict the target domain data and obtain the final classification result.

2. The cross-domain classification method of hyperspectral images applicable to complex ground scenes according to claim 1 is characterized in that: In S1, the mathematical expression for independently extracting features from the spatial, spectral and temporal / scene dimensions of the hyperspectral image is: Where X∈R H×W×B×T is the input hyperspectral image, H, W and B represent the height, width and number of bands of the image respectively, T represents the time or scene dimension, H ′ , W ′ , B ′ and T ′ They represent the dimensions corresponding to the feature extraction of hyperspectral images in different dimensions, L h , L s , L t are the number of feature extraction layers for spatial, spectral, and temporal / scene dimensions, respectively. represents the weight matrix of the lth layer corresponding to the spatial, spectral, and temporal / scene dimensions, are the bias terms corresponding to each layer, σ(·) is the activation function, * represents the convolution operation, and F spatial 、F spectral and F temporal Represent the extracted spatial, spectral and temporal / scene features, respectively.

3. The cross-domain classification method of hyperspectral images applicable to complex ground scenes according to claim 2 is characterized in that: In S2, the mathematical expression for the interaction and fusion operation of the features of each dimension is: Among them, i,j represents the index position of the feature, W att is the attention weight matrix, represents the dot product operation of the vector, represents the multidimensional concatenated vector of the i-th feature, σ(·) represents the activation function used to perform nonlinear transformation on the input feature interaction value, and A ij represents the interaction value between the i-th and j-th features, measuring their association strength under the attention mechanism; F fusion It represents the features obtained by interactively fusing spatial, spectral and temporal / scene features. The coefficients α, β and γ are used to control the weights of features in each dimension, satisfying α+β+γ=1.

4. The cross-domain classification method of hyperspectral images applicable to complex ground scenes according to claim 3 is characterized in that: In S3, the mathematical expression for achieving the matching of the source domain and the target domain feature space is: L domain_align =‖M(F s )-F T ‖ 2 Among them, F s and F T are the source domain and target domain features, and represents the nth category feature of the source domain and the target domain, N represents the number of categories, M(·) is the feature mapping function, ‖·‖ 2 represents the Euclidean norm, L domain_align is the domain alignment loss between the source domain and the target domain, L class_align is the category alignment loss between the source domain and the target domain.

5. The cross-domain classification method of hyperspectral images applicable to complex ground scenes according to claim 4 is characterized in that: In S4, the source domain classification loss is calculated, and the model parameters are optimized by combining the source domain classification loss and feature alignment loss. The mathematical expression is: L=L s +λ1L domain_align +λ2L class_align Among them, L s is the source domain classification loss calculated using the cross entropy loss function, y i,c is the true label of the i-th sample in category c, is the prediction probability, L is the final comprehensive optimization objective function, λ1 and λ2 are adjustment parameters used to control the weights of different loss items.

6. The cross-domain classification method of hyperspectral images applicable to complex ground scenes according to claim 5 is characterized in that: In S5, the target domain data is input into the trained classification model, and its mathematical expression is: in, is the prediction result of the target domain data, W and b are model parameters.

7. A hyperspectral image cross-domain classification device suitable for complex ground scenes, characterized in that: Used to perform the method according to any one of claims 1 to 6, comprising: An acquisition module is used to acquire hyperspectral images and independently extract features from the spatial, spectral, and temporal / scene dimensions of the hyperspectral images; Interaction fusion module, used to interact and fuse features of each dimension; The matching module is used to achieve the matching of the source domain and target domain feature spaces by optimizing the feature distribution mapping and combining the dual constraints of domain alignment and class alignment; The training module is used to train the cross-domain classification model of hyperspectral images using labeled source domain data and unlabeled target domain data, and jointly optimize the model through a multi-task loss function; The prediction module is used to use the trained model to predict the target domain data and obtain the final classification result.

8. A computer storage medium, characterized in that The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 6.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6.

Images