Hyperspectral image cross-domain ground feature element extraction method based on block representation mechanism

Abstract

The application discloses a hyperspectral image cross-domain ground feature extraction method based on a block representation mechanism and belongs to the technical field of image processing technology and comprises the following steps: S1, constructing a space-spectrum consistent deep residual 3D convolution network, relieving the space-spectrum structure difference between scenes, and extracting space-spectrum embedded deep features; S2, constructing a relation mapping module, designing a block representation mechanism, and calculating prototype features by category; S3, designing a domain alignment strategy, and relieving the condition distribution difference between domains; S4, selecting a source scene and a target scene, respectively constructing a support set and a query set, and inputting the network for training; and S5, using the trained network for the target scene, performing panoramic inference, and extracting all ground features in the target scene. The application improves the ground feature extraction efficiency of heterogeneous data, strengthens the separability of features in a relation subspace, and realizes the migration of a model to different scenes.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for extracting cross-domain ground features from hyperspectral images based on a block representation mechanism. Background Technology

[0002] With the rapid development of remote sensing technology, hyperspectral remote sensing images are evolving towards big data, and the surge in the number of images urgently requires accurate classification of a large number of new scenes. In hyperspectral remote sensing classification tasks, both traditional modeling methods and existing deep learning methods rely on sufficient and accurate labeled samples to ensure classification accuracy. Given a sufficient amount of training samples, and when the training and testing data satisfy the independent and identically distributed assumption, they typically provide good performance. However, remote sensing observation scenes are large and exhibit strong landscape variability, often requiring the classification and interpretation of a large number of new scenes in practical applications.

[0003] Acquiring labeled samples for hyperspectral remote sensing data is challenging, requiring significant manpower and resources, resulting in high costs and long cycles. Furthermore, ensuring a rich sample pool is difficult when dealing with novel and variable scenarios. Constrained by the sample environment, current hyperspectral classification research often builds models based on predetermined scenarios (source domains) and performs direct interpretation for new scenarios (target domains). This fails to meet the independent and identically distributed assumptions required for traditional modeling, leading to extremely high generalization errors and poor or unclassifiable overall classification results. Therefore, the variable data distribution and insufficient labeled sample size are the core bottlenecks limiting the effectiveness of hyperspectral classification models.

[0004] Although the number of labeled samples in the target domain is insufficient, the related source domain scene often has enough labeled samples. Classifying the target domain data using source domain data is currently the most common learning method, known as hyperspectral image cross-domain classification. However, most existing cross-domain classification methods only focus on single-feature transfer tasks with the same sensor and the same land cover category, and cannot be applied to multi-feature transfer tasks where the source and target scenes are acquired from different sensors and have completely different land cover categories. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism, which combines a supervised comparison strategy to design a block representation mechanism and adopts cross-domain feature alignment technology and meta-task training principle. This method is applicable to cross-domain classification tasks with both single-feature differences and multi-feature differences, and maintains high extraction accuracy and low computational complexity in cross-scene prediction.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism includes the following steps:

[0008] S1: Construct a spatial spectrum consistent deep residual 3D convolutional network to alleviate the differences in spatial spectrum structure between scenes and extract spatial spectrum embedding depth features;

[0009] S2: Construct a relation mapping module, design a block representation mechanism, and calculate prototype features for each category;

[0010] S3: Design domain alignment strategies to mitigate differences in conditional distribution between domains;

[0011] S4: Select the source scene and the target scene, construct the support set and query set respectively, and input them into the network for training;

[0012] S5: The trained network is used in the target scene to perform panoramic reasoning and extract all ground features in the target scene.

[0013] Furthermore, step S1 includes the following sub-steps:

[0014] S11: Construct a mapping module that is consistent with spatial convolution and interspectral projection, and map the samples of the source scene and the target scene obtained from different sensors to the same dimension of the spatial-spectral subspace to obtain spatial-spectral consistent features.

[0015] S12: Construct a deep residual 3D convolutional network, which consists of multiple 3D convolution, pooling and 3D residual modules. Input the spatial-spectral consistent features in step S11 to obtain the spatial-spectral embedding deep features of the source scene and the target scene.

[0016] Furthermore, step S11 specifically includes: constructing a mapping module that is consistent with spatial convolution and interspectral projection. This mapping module contains two 2D convolution operations and two batch normalizations, wherein the convolution kernel size is 1×1 and is concatenated in the order of convolution-batch normalization; and constructing mapping modules for the source domain and the target domain respectively, with the input dimension being the data spectral dimension and the output dimension being consistently set to 100.

[0017] Furthermore, the deep residual 3D convolutional network in step S12 is based on ResNet and uses 3D convolution as the basic convolution operator, which takes into account the feature correlation extraction of both spatial and channel dimensions. The deep residual 3D convolutional network includes a 3D convolution module, a 3D residual module, and a 3D max pooling module, and their connection order is: 3D residual module - 3D max pooling module - 3D residual module - 3D max pooling module - 3D convolution module.

[0018] The 3D residual module consists of three concatenated modules: 3D convolution, 3D batch normalization, and activation function. The output of the first module is linked to the output of the third module. The kernel size of the 3D convolution is set to 3×3×3, and the activation function is ReLU.

[0019] A fully connected layer maps the feature map output by the deep residual 3D convolutional network to the label space. This fully connected layer consists of a fully connected layer and a 1D batch normalization layer, with the output dimension set to the number of classes. The predicted probabilities are obtained by inputting the softmax function, and the cross-entropy loss is calculated.

[0020]

[0021] in Represents the global features of the source domain samples. This represents the label corresponding to the source domain sample. Indicates the number of samples in the source domain. Represents a fully connected layer. This represents the softmax function.

[0022] Furthermore, step S2 includes the following sub-steps:

[0023] S21: Output the feature map before the last 3D convolution of the depth residual 3D convolutional network, and divide the feature map to obtain block depth features;

[0024] S22: Construct a relation mapping module, based on a convolutional neural network, to map the block depth features to the relation subspace for subsequent block representation mechanism;

[0025] S23: Calculate the mean of the support set block features for each category to obtain the prototype features, and combine them with the query set block features input relationship mapping module;

[0026] S24: Design a block representation mechanism, combined with a supervised contrast strategy, for the training process of the meta-tasks in the source and target domains respectively, to mine robust representations within samples that have been decomposed into multiple blocks.

[0027] Furthermore, in step S24, a block representation loss is constructed by combining a supervised contrastive strategy, which is used in the meta-task training process of the source and target domains respectively:

[0028]

[0029] For each relation feature in the mini-batch , It is its set of positive samples. It is the number of positive samples. It is one of the positive samples. It is a set of negative samples. It is one of the negative samples;

[0030] In block representation learning, features belonging to the same class in the relation space are placed into the positive sample set, and features from different classes are placed into the negative sample set. The network is optimized through supervised contrastive learning, enabling the network to learn cross-domain shared representations for each class from samples of the same class.

[0031] Furthermore, step S3 includes the following sub-steps:

[0032] S31: Design a local maximum mean difference domain alignment module. To improve the maximum mean difference, project the label source domain and target domain onto the Hilbert space and calculate the local mean difference for each category using the available labels.

[0033] S32: Input the support set and query set block depth features from step S21 and the prototype features from step S23, and use the local maximum mean difference domain alignment strategy to complete the inter-domain feature alignment.

[0034] Furthermore, step S4 includes the following sub-steps:

[0035] S41: Select two hyperspectral remote sensing images with different regions, resolutions, and spectral information;

[0036] S42: Use the image with the most land cover categories as the source domain and another image as the target domain; select all label samples in the source domain and a small number of label samples in the target domain to construct the support set and query set respectively;

[0037] S43: Set the number of training iterations for the network, the initial learning rate, the window size for selecting samples, the number of classes selected each time in the meta-task training, the number of supervised samples selected for each class in the support set, and the number of samples selected for each class in the query set.

[0038] S44: Input data is used to train the network. The weights are updated by backpropagating the gradient through the calculated loss function. The overall loss function of the network is as follows:

[0039]

[0040] in, The cross-entropy loss is calculated from the spatial spectral embedding depth features output in step S12 and the label. The block representation loss is obtained from the block representation losses of the source and target domains in step S24. The local maximum mean difference loss is calculated from the block depth features and prototype features of the source and target domains in step S32.

[0041] S43: After training is complete, save the network parameters that have been debugged.

[0042] Furthermore, step S5 includes the following sub-steps:

[0043] S51: All label samples in the target domain data divided in step S42 are used as the test dataset and input into the trained network to obtain the classification results of the hyperspectral image test dataset;

[0044] S52: Input the full target scene map into the trained network to obtain the complete target scene feature extraction results.

[0045] The beneficial effects of this invention are as follows:

[0046] (1) This invention fully considers the differences between single elements and multiple elements in cross-domain classification tasks, and proposes a complete model for extracting ground features from hyperspectral images across scenes. This method aims to alleviate the multidimensional differences between heterogeneous data and can extract ground features from remote sensing images of different regions, time periods and different sensors, which greatly improves the efficiency of extracting ground features from heterogeneous data.

[0047] (2) Compared with the traditional global sample representation mechanism, the block representation mechanism proposed in this invention can accurately predict the corresponding category using only local sample information, breaking through the upper limit of the traditional model's ability to learn the corresponding category representation, and further enhancing the separability of features in the relational subspace by combining supervised comparison strategy.

[0048] (3) The local maximum mean difference loss of the present invention makes full use of a small amount of labeled data in the source domain and the target domain compared with the maximum mean difference in the prior art. During the training process of the meta-task, the shared knowledge in the source domain is transferred to the target domain category by category, realizing the transfer of the model to different scenarios.

[0049] Other advantages, objectives, and features of the invention will be set forth in the following description and will be apparent to those skilled in the art in some respects, or may be learned by practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0050] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration:

[0051] Figure 1 This is a flowchart of the hyperspectral image cross-domain feature extraction method based on the block representation mechanism of the present invention;

[0052] Figure 2 This is a diagram of the deep residual 3D convolutional network structure constructed in this invention;

[0053] Figure 3 This is a schematic diagram of the block features designed in this invention;

[0054] Figure 4 This is a structural diagram of the relation mapping module constructed in this invention.

[0055] Figure 5 This is a flowchart of the block representation mechanism constructed in this invention;

[0056] Figure 6 This is a schematic diagram of the block representation mechanism constructed in this invention;

[0057] Figure 7 This is a flowchart of the domain alignment module constructed in this invention;

[0058] Figure 8 This is a flowchart of the cross-domain classification framework constructed by this invention;

[0059] Figure 9 The following are examples of Chikusei data in the embodiments: (a) pseudo-color representation, (b) ground truth map;

[0060] Figure 10 The following are examples of Pavia University data in the embodiments: (a) pseudo-color representation, and (b) the results of the trained network model extracting features from the data scene. Detailed Implementation

[0061] like Figure 1 As shown, a method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism includes the following steps:

[0062] Step 1: Construct a spatially consistent deep residual 3D convolutional network to alleviate the differences in spatial-spectral structure between scenes and extract spatial-spectral embedding depth features; specifically, Step 1 includes the following steps:

[0063] Step 1.1: Construct a spatially consistent and spectrally consistent mapping module. This module includes two 2D convolution operations and two batch normalization operations, where the convolution kernel size is 1×1, and they are concatenated in the order of convolution and batch normalization. Mapping modules are constructed separately for the source and target domains, with the input dimension being the data spectral dimension and the output dimension consistently set to 100. In the following embodiment, the sample size is set to 13×13.

[0064] Step 1.2: Construct a deep residual 3D convolutional network, the specific structure of which is as follows: Figure 2 As shown. This network is based on ResNet and, considering the spatial-spectral fusion characteristic of hyperspectral images, employs 3D convolution as the basic convolution operator to extract the correlation between spatial and channel dimensions. The network includes a 3D convolution module, a 3D residual module, and a 3D max-pooling module, following... Figure 2The module concatenation order is as follows: 3D residual module - 3D max pooling module - 3D residual module - 3D max pooling module - 3D convolution module, where the kernel sizes of the 3D max pooling module and the 3D convolution module are set to 2×2×1 and 3×3×1, respectively. Further, the 3D residual module consists of three concatenated modules of 3D convolution, 3D batch normalization, and activation function. The output of the first module is linked to the output of the third module. The kernel size of the 3D convolution is set to 3×3×3, and the activation function is ReLU. In the following embodiment, the feature map mapped in step 1.1 is input, with a size of 13×13×100, and the outputs are block features of 4×4×112 and global features of 32×2×2×5, respectively. A schematic diagram of the block features is shown below. Figure 3 As shown;

[0065] Step 1.3: Map the feature map output by the deep residual 3D convolutional network to the label space through a fully connected layer. This fully connected layer consists of a fully connected layer and 1D batch normalization, and the output dimension is set to the number of classes. Input the softmax function to obtain the predicted probability and calculate the cross-entropy loss.

[0066]

[0067] in Represents the global features of the source domain samples. This represents the label corresponding to the source domain sample. Indicates the number of samples in the source domain. Represents a fully connected layer. This represents the softmax function.

[0068] Step 2: Construct a relationship mapping module, design a block representation mechanism, and calculate prototype features for each category; specifically, such as... Figure 4 As shown, step 2 includes the following steps:

[0069] Step 2.1: Construct the relation mapping module, the specific structure of which is shown in Figure 4. This module consists of a 2D convolution, a 2D batch normalization, an activation function, and a fully connected layer connected in series. The convolution kernel size is 1×1, the input dimension of the fully connected layer is set to 512, and the output dimension is set to 256.

[0070] Step 2.2: Construct a block representation mechanism, the specific process is as follows: Figure 5 As shown. Input the block features from step 1.2, calculate the mean of the support set block features for each category to obtain the prototype features, and input them together with the query set block features into the relation mapping module. After passing through the relation mapping module, the relation features 4×4×256 are output.

[0071] Step 2.3: Construct a block representation loss using a supervised contrastive strategy, which is then used in the meta-task training process for both the source and target domains.

[0072]

[0073] For each relation feature in the mini-batch , It is its set of positive samples. It is the number of positive samples. It is one of the positive samples. It is a set of negative samples. These are negative samples. In block representation learning, features belonging to the same class in the relation space are placed in the positive sample set, and features from different classes are placed in the negative sample set. Supervised contrastive learning optimizes the network, promoting closer similarity between similar samples and greater distance between different classes, enabling the network to learn cross-domain shared representations from similar samples. This fully leverages the robust intra-block representations of the decomposed samples, improving the discriminative power of intra-class meta-knowledge. A schematic diagram of the block representation mechanism is shown below. Figure 6 As shown.

[0074] Step 3: Design a domain alignment strategy to mitigate differences in conditional distribution between domains; specifically, Step 3 includes the following sub-steps:

[0075] Step 3.1: Construct a local maximum mean difference domain alignment module. The specific process is as follows: Figure 7 As shown, the local maximum mean difference loss is calculated for the source and target domain block features and the prototype features of the input:

[0076]

[0077] in This represents the number of source domain samples of class c. The number of samples in the c-th target domain is represented by the average difference between the source and target domains for each class in the kernel Hilbert space (RKHS). The smaller this loss is, the smaller the difference in the conditional distributions of the two domains, thereby constraining the similarity of the conditional distributions of the source and target domain features in the hyperspectral data.

[0078] Step 3.2: Input the block features from Step 1.2 and the prototype features from Step 2.2, and calculate the local maximum mean difference loss.

[0079] Step 4: Select the source scene and target scene, construct the support set and query set respectively, and input them into the network for training; specifically, Step 4 includes the following sub-steps:

[0080] Step 4.1: Select two hyperspectral remote sensing images with different regions, resolutions, and spectral information;

[0081] Step 4.2: Use the image with the most land cover categories as the source domain and the other image as the target domain. Construct a support set and a query set by selecting all label samples from the source domain and a small number of label samples from the target domain, respectively. In the following embodiment, the source domain has 19 categories and the target domain has 9 categories;

[0082] Step 4.3: Set the number of training iterations, initial learning rate, window size for sample selection, number of classes selected in each meta-task training iteration, number of supervised samples selected for each class in the support set, and number of samples selected for each class in the query set. In the following embodiment, the number of training iterations is 300, the initial learning rate is 1e-3, the window size for sample selection is 13×13, the number of classes randomly selected in the meta-task is 9, the number of supervised samples selected for each class in the support set is 1, and the number of samples selected for each class in the query set is 19.

[0083] Step 4.4: Construct a cross-domain classification framework, such as... Figure 8 As shown. The overall loss function is as follows:

[0084]

[0085] in, The cross-entropy loss is calculated from the spatial spectral embedding deep features and the label. The block representation loss is represented by the block representation loss of the source domain and the target domain. The loss represents the local maximum mean difference, calculated from the block depth features and prototype features of the source and target domains;

[0086] Step 4.4: Begin training the network model. At the start of each training round, all training images are input into the network. The network adjusts its parameters according to the direction that minimizes the loss function value until the loss value is minimized, at which point the best training effect can be obtained.

[0087] Step 4.5: Save the network model parameters after training is complete.

[0088] Step 5: Apply the trained network to the target scene to perform panoramic inference and extract all terrain features in the target scene. Specifically, Step 5 includes the following steps:

[0089] Step 5.1: Load the network model parameters, take all the label samples in the target domain data divided in Step 4.2 as the test dataset, input them into the network to obtain the classification results of the hyperspectral image test dataset;

[0090] Step 5.2: Create 13×13 samples pixel by pixel from the entire target scene image, input them into the network to finally obtain the complete target scene feature extraction results.

[0091] In the following embodiments, the source scene uses Chikusei data, which has 128 bands, a spectral range from 363 nm to 1018 nm, an image size of 2517 × 2335 pixels, and a spatial resolution of 2.5 m. There are 19 categories, including urban and rural areas. The pseudo-color representation and ground truth map of the Chikusei data are shown below. Figure 9 As shown in (a) and (b) in the figure. The target scene uses Pavia University data, which has 103 spectral bands, covering a spectral range from 430 nm to 860 nm. The image size is 610 × 340 pixels, the spatial resolution is 1.3 m, and it contains 9 categories. The pseudo-color representation of the Pavia University data and the results of the trained network model for extracting ground features from the target scene are shown in the figure. Figure 10 As shown in (a) and (b) in the figure.

[0092] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism, characterized in that: Includes the following steps: S1: Construct a spatially consistent deep residual 3D convolutional network to alleviate the differences in spatially spectral structure between scenes and extract spatially embedded deep features; step S1 includes the following sub-steps: S11: Construct a spatial convolution-consistent and spectral projection-consistent mapping module to map samples from source and target scenes acquired from different sensors to the same-dimensional spatial-spectral subspace, obtaining spatial-spectral consistent features; Step S11 specifically includes: constructing a spatial convolution-consistent and spectral projection-consistent mapping module, which contains two 2D convolution operations and two batch normalizations, wherein the convolution kernel size is 1×1, and they are concatenated in the order of convolution and batch normalization; construct mapping modules for the source and target domains respectively, with the input dimension being the data spectral dimension and the output dimension consistently set to 100; S12: Construct a deep residual 3D convolutional network, which consists of multiple 3D convolution, pooling and 3D residual modules. Input the spatial-spectral consistent features in step S11 to obtain the spatial-spectral embedding depth features of the source scene and the target scene. S2: Construct a relation mapping module, design a block representation mechanism, and calculate prototype features for each category; step S2 includes the following sub-steps: S21: Output the feature map before the last 3D convolution of the depth residual 3D convolutional network, and divide the feature map to obtain block depth features; S22: Construct a relation mapping module, based on a convolutional neural network, to map the block depth features to the relation subspace for subsequent block representation mechanism; S23: Calculate the mean of the support set block features for each category to obtain the prototype features, and combine them with the query set block features input relationship mapping module; S24: Design a block representation mechanism, combined with a supervised contrast strategy for the training process of the meta-tasks in the source and target domains, to mine robust representations within samples that have been decomposed into multiple blocks. S3: Design a domain alignment strategy to mitigate differences in conditional distribution between domains; Step S3 includes the following sub-steps: S31: Design a local maximum mean difference domain alignment module. To improve the maximum mean difference, project the label source domain and target domain onto the Hilbert space and calculate the local mean difference for each category using the available labels. S32: Input the support set and query set block depth features from step S21 and the prototype features from step S23, and use the local maximum mean difference domain alignment strategy to complete the inter-domain feature alignment; S4: Select the source scene and the target scene, construct the support set and the query set respectively, and input them into the network for training; step S4 includes the following sub-steps: S41: Select two hyperspectral remote sensing images with different regions, resolutions, and spectral information; S42: Use the image with the most land cover categories as the source domain and another image as the target domain; select all label samples in the source domain and a small number of label samples in the target domain to construct the support set and query set respectively; S5: The trained network is used in the target scene to perform panoramic reasoning and extract all ground features in the target scene.

2. The method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism according to claim 1, characterized in that: The deep residual 3D convolutional network in step S12 is based on ResNet and uses 3D convolution as the basic convolution operator, which takes into account the feature correlation extraction of both spatial and channel dimensions. The deep residual 3D convolutional network includes a 3D convolution module, a 3D residual module, and a 3D max pooling module, and their connection order is: 3D residual module - 3D max pooling module - 3D residual module - 3D max pooling module - 3D convolution module. The 3D residual module consists of three concatenated modules: 3D convolution, 3D batch normalization, and activation function. The output of the first module is linked to the output of the third module. The kernel size of the 3D convolution is set to 3×3×3, and the activation function is ReLU. A fully connected layer maps the feature map output by the deep residual 3D convolutional network to the label space. This fully connected layer consists of a fully connected layer and a 1D batch normalization layer, with the output dimension set to the number of classes. The predicted probabilities are obtained by inputting the softmax function, and the cross-entropy loss is calculated. in Represents the global features of the source domain samples. This represents the label corresponding to the source domain sample. Indicates the number of samples in the source domain. Represents a fully connected layer. This represents the softmax function.

3. The method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism according to claim 1, characterized in that: In step S24, a block representation loss is constructed using a supervised contrastive strategy and used in the meta-task training process of both the source and target domains. For each relation feature in the mini-batch , It is its set of positive samples. It is the number of positive samples. It is one of the positive samples. It is a set of negative samples. It is one of the negative samples; In block representation learning, features belonging to the same class in the relation space are placed into the positive sample set, and features from different classes are placed into the negative sample set. The network is optimized through supervised contrastive learning, enabling the network to learn cross-domain shared representations for each class from samples of the same class.

4. The method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism according to claim 1, characterized in that: Step S4 also includes the following steps: S43: Set the number of training iterations for the network, the initial learning rate, the window size for selecting samples, the number of classes selected each time in the meta-task training, the number of supervised samples selected for each class in the support set, and the number of samples selected for each class in the query set. S44: Input data is used to train the network. The weights are updated by backpropagating the gradient through the calculated loss function. The overall loss function of the network is as follows: in, The cross-entropy loss is calculated from the spatial spectral embedding depth features output in step S12 and the label. The block representation loss is obtained from the block representation losses of the source and target domains in step S24. The local maximum mean difference loss is calculated from the block depth features and prototype features of the source and target domains in step S32. S45: After training is complete, save the network parameters that have been debugged.

5. The method for extracting cross-domain land cover features from hyperspectral images based on a block representation mechanism according to claim 1, characterized in that: Step S5 includes the following sub-steps: S51: All label samples in the target domain data divided in step S42 are used as the test dataset and input into the trained network to obtain the classification results of the hyperspectral image test dataset; S52: Input the full target scene map into the trained network to obtain the complete target scene feature extraction results.

Images