Generative self-supervised pre-training classification model construction method and device
Through the generative self-supervised pre-training classification model, the multi-scale masking and coding strategies of multi-source remote sensing images are used, combined with interactive comparison learning, and the problem of insufficient extraction of geographic discriminant features in hyperspectral image classification is solved, and the problem of insufficient extraction of geographic discriminant features in hyperspectral image classification is improved by a few Classification performance under sample conditions.
Patent Information
- Application Number
- CN202411094069.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-08
AI Technical Summary
In hyperspectral image classification task, generative learning methods are difficult to effectively utilize the complementarity of different types of remote sensing image features, resulting in insufficient extraction capabilities of ground objects discriminant features, especially under the condition of few samples.
The generative self-supervised pre-training classification model construction method is adopted. By collecting multi-source remote sensing images, using multi-scale masking strategies and spatial position encoding, multi-source features are extracted, and reconstruction and spectral consistency constraints are performed through the decoder, combined with interactive comparison Learn to optimize the encoder to improve the model's learning ability of the deep features of the ground objects.
It significantly improves the classification performance of hyperspectral images under the condition of few samples, enhances the separability and classification accuracy of deep features of the ground objects, and improves the model's discriminant feature extraction and fusion ability of multi-source remote sensing images.
Smart Images

Figure CN118918384A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image classification, and in particular to a method and device for constructing a generative self-supervised pre-training classification model. Background Art
[0002] At present, with the continuous expansion of hyperspectral remote sensing applications, the task of hyperspectral image classification has become more complex and challenging. Especially under the condition of few samples, effectively integrating and utilizing the advantages of different types of remote sensing images is an effective way to improve the performance of hyperspectral image classification. As a powerful self-supervised feature learning paradigm, generative learning has shown unique advantages in the field of hyperspectral image processing. It drives the model to learn the deep structural features of the data by designing reconstruction proxy tasks closely related to the characteristics of hyperspectral images, thereby mining the high-level feature information of the objects. However, in the task of hyperspectral image classification, generative learning often focuses on extracting rich and complex feature representations, which may ignore the discriminative differences between different objects, which may lead to insufficient separability between different types of objects at the feature level. In addition, due to the significant differences in the characteristics of different types of remote sensing images, the generative learning method for a single data source is difficult to effectively utilize the complementarity of the features of different types of remote sensing images, which limits the model's ability to extract discriminative features of objects. Summary of the invention
[0003] The present invention provides a method and device for constructing a generative self-supervised pre-training classification model, which can improve the model's ability to learn deep features of ground objects from multi-source remote sensing images under few-sample conditions, thereby improving the hyperspectral image classification performance under few-sample conditions.
[0004] To this end, the present invention provides the following technical solutions:
[0005] A method for constructing a generative self-supervised pre-trained classification model, the method comprising:
[0006] Collect hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object;
[0007] Based on the hyperspectral image, multispectral image, and panchromatic image, corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets are obtained by cutting, and a sample set is generated;
[0008] Using a multi-scale masking strategy to perform masking processing on the sample data in the sample set to obtain a masked image;
[0009] Performing spatial position encoding on the masked images, and embedding marking information according to the used masking strategy to obtain encoding data corresponding to each masked image;
[0010] Performing feature extraction on the encoded data to obtain multi-source features;
[0011] Fusing the multi-source features to obtain multi-source deep fusion features;
[0012] The multi-source deep fusion features are reconstructed using a decoder, and a spectral consistency constraint is introduced based on the reconstruction loss.
[0013] Optionally, the step of obtaining corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets based on the hyperspectral image, multispectral image, and panchromatic image clipping to generate sample data includes:
[0014] Taking any pixel in the hyperspectral image as a target pixel, cutting out a hyperspectral neighborhood image within a certain neighborhood range with the target pixel as the center, and generating a hyperspectral image subset according to the hyperspectral neighborhood image;
[0015] Cut out a multispectral neighborhood image corresponding to the hyperspectral image neighborhood image, and generate a multispectral image subset according to the multispectral neighborhood image;
[0016] Cut out a panchromatic neighborhood image corresponding to the hyperspectral image neighborhood image, and generate a panchromatic image subset based on the panchromatic neighborhood image;
[0017] A sample set is generated according to the hyperspectral image subset, the multispectral image subset, and the panchromatic image subset, wherein each sample in the sample set includes a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image corresponding to a target pixel.
[0018] Optionally, the multi-scale masking strategy includes any one or more of the following: small-size random masking, large-size random masking, small-size channel random masking, large-size channel random masking, random band masking, and continuous band masking.
[0019] Optionally, the using a multi-scale masking strategy to mask the sample data in the sample set to obtain a masked image includes: for different types of image data sets, randomly selecting different masking strategies to mask the sample data in the sample set to obtain a masked image.
[0020] Optionally, the method further comprises:
[0021] Before using an encoder to perform spatial position encoding on the mask image, the encoded data is mapped to a feature space of a predefined dimension.
[0022] Optionally, extracting features from the encoded data to obtain multi-source features includes: extracting features from the encoded data to obtain multi-source features using an asymmetric encoder-decoder structure based on Transformer.
[0023] Optionally, the fusing of the multi-source features to obtain multi-source deep fusion features includes: cross-attention fusing the multi-source features, and during the fusion process, taking the hyperspectral image as the main body, fusing the panchromatic and multispectral image features to obtain the multi-source deep fusion features.
[0024] Optionally, the reconstructing the multi-source deep fusion features by using a decoder includes:
[0025] The image blocks are randomly masked based on the set masking rate, and the different types of unmasked image areas are processed using linear projection and position encoding to obtain the corresponding feature embeddings. The different types of image feature embeddings output by the encoder are input into the cross-attention fusion process.
[0026] The decoder is used to reconstruct specific types of images based on the multi-source deep fusion features extracted by the encoder and the cross-attention fusion process combined with the position encoding information.
[0027] Optionally, the optimization objectives of the process of reconstructing a specific type of image include:
[0028] reconstruction loss and spectral loss;
[0029] The reconstruction loss refers to the pixel-by-pixel mean square error between the image reconstructed by the decoder of different tasks and the original image corresponding to the task;
[0030] The spectral loss is represented by a spectral loss function, and the spectral loss function uses spectral cosine to calculate the similarity between the reconstructed spectral features and the original spectral features.
[0031] Optionally, the method further comprises:
[0032] Perform interactive comparative learning on the multi-source features to optimize the encoder;
[0033] The multi-source feature interaction contrast loss includes: the contrast loss from the hyperspectral image feature to other image features Loss(z hsi ,z img ), and the contrast loss Loss(z img ,z hsi ).
[0034] A generative self-supervised pre-trained classification model construction device, the device comprising:
[0035] An image acquisition module, used to acquire hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object;
[0036] A sample set generation module, used for obtaining corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets based on the hyperspectral image, multispectral image, and panchromatic image clipping to generate a sample set;
[0037] A sample processing module, used for masking the sample data in the sample set using a multi-scale masking strategy to obtain a masked image;
[0038] A position encoding module, used to perform spatial position encoding on the masked images and embed marking information according to the masking strategy used to obtain encoding data corresponding to each masked image;
[0039] A feature extraction module, used to extract features from the encoded data to obtain multi-source features;
[0040] A fusion module, used to fuse the multi-source features to obtain deep fusion features;
[0041] A decoding module is used to reconstruct the deep fusion features by using a decoder, and introduce a spectral consistency constraint based on the reconstruction loss.
[0042] Optionally, the feature extraction module uses an asymmetric encoder-decoder structure based on Transformer to extract features from the encoded data to obtain multi-source features;
[0043] The device also includes:
[0044] The contrastive learning module is used to perform interactive contrastive learning on the multi-source features to optimize the encoder.
[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the generative self-supervised pre-training classification model construction method.
[0046] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the generative self-supervised pre-trained classification model construction method.
[0047] The generative self-supervised pre-training classification model construction method and device provided by the present invention combine the deep feature mining ability of generative learning with the feature discriminability enhancement effect of contrastive learning, and can extract the spatial spectral features of ground objects contained in multi-source remote sensing images and enhance the separability of these features, especially under the condition of few samples. By fusing multi-source features, multi-source deep fusion features are obtained, and the multi-source deep fusion features are reconstructed using a decoder, and spectral consistency constraints are introduced on the basis of reconstruction loss.
[0048] Furthermore, the encoder can be optimized by interactively comparing and learning multi-source features. Through the dual-branch parallel optimization mechanism, not only the ability of the generative self-supervised pre-trained classification model to learn deep features of ground objects from multi-source remote sensing images is enhanced, but also the discriminant features of different remote sensing images are further mined and aligned through interactive comparative learning, thereby greatly improving the hyperspectral image classification performance under few-sample conditions.
[0049] The classification model constructed using the scheme of the present invention enhances the model's ability to extract and fuse deep discriminant features through multi-source feature comparison optimization and cross-attention fusion. At the same time, a spectral consistency reconstruction loss is designed, and pre-training is completed through accurate reconstruction of a large number of unlabeled multi-source remote sensing images. The training results are used as the feature extractor of the classification model, which significantly improves the classification accuracy and generalization ability of few-sample hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0051] Figure 1 It is a flow chart of the method for constructing a generative self-supervised pre-trained classification model provided by the present invention;
[0052] Figure 2 It is a process schematic diagram of the method for constructing a generative self-supervised pre-training classification model provided by the present invention;
[0053] Figure 3 is a schematic diagram of the effect of masking sample data using various masking micro-strategies in an embodiment of the present invention;
[0054] Figure 4 Schematic diagram of the working mechanism of multi-source feature cross-attention fusion in an embodiment of the present invention;
[0055] Figure 5 Schematic diagram of a multi-source feature interactive comparison learning process in an embodiment of the present invention;
[0056] Figure 6 Schematic diagram of the structure of the loss function for accurate reconstruction of multi-source remote sensing images in an embodiment of the present invention;
[0057] Figure 7 It is a structural schematic diagram of a generative self-supervised pre-training classification model construction device provided by the present invention;
[0058] Figure 8It is another structural schematic diagram of the generative self-supervised pre-training classification model construction device provided by the present invention. DETAILED DESCRIPTION
[0059] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0060] In view of the problem that the existing generative learning methods for a single data source are difficult to effectively utilize the complementarity of features of different types of remote sensing images, thereby limiting the model's ability to extract discriminant features of ground objects, the present invention provides a generative self-supervised pre-training classification model construction method, a classification method and a device, which combine the deep feature mining capability of generative learning with the feature discriminability enhancement effect of contrastive learning, extract the spatial-spectral features of ground objects contained in multi-source remote sensing images, and enhance the separability of these features, thereby improving the model's ability to learn deep features of ground objects from multi-source remote sensing images, and at the same time further mine and align the discriminant features of different remote sensing images through interactive contrastive learning, thereby improving the classification performance of hyperspectral images under few-sample conditions.
[0061] like Figure 1 FIG. 1 is a flow chart of a method for constructing a generative self-supervised pre-training classification model provided by the present invention. Figure 2 It is a schematic diagram of the model building process.
[0062] Also refer to Figure 1 and Figure 2 , the method comprises the following steps:
[0063] Step 101 : Acquire a hyperspectral image, a multispectral image, and a panchromatic image corresponding to the same target object.
[0064] It should be noted that geometric registration has been performed between different images, and the method of geometric registration can adopt existing technology, which is not limited in the embodiment of the present invention.
[0065] Step 102: Based on the hyperspectral image, multispectral image, and panchromatic image, corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets are obtained by cropping to generate a sample set.
[0066] The specific process is as follows:
[0067] (1) Any pixel in the hyperspectral image is taken as a target pixel, and a hyperspectral neighborhood image of a certain neighborhood range is cut out with the target pixel as the center. In this way, hyperspectral neighborhood images of multiple pixels can be obtained, and a hyperspectral image subset can be obtained based on these hyperspectral neighborhood images;
[0068] (2) cutting out the multispectral neighborhood images corresponding to the neighborhood images of each hyperspectral image, and generating a multispectral image subset based on these multispectral neighborhood images;
[0069] (3) Similarly, the panchromatic neighborhood images corresponding to the neighborhood images of each hyperspectral image are cut out, and a panchromatic image subset is generated based on these panchromatic neighborhood images;
[0070] (4) generating a sample set according to the hyperspectral image subset, the multispectral image subset, and the panchromatic image subset, wherein the sample set includes a plurality of samples, and each sample includes a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image corresponding to a target pixel.
[0071] Step 103: mask the sample data in the sample set using a multi-scale masking strategy to obtain a masked image.
[0072] The masked image refers to an image obtained by performing masking processing on each image subset in the sample set, and includes masked image blocks and unmasked image blocks. The unmasked image blocks may also be referred to as patches.
[0073] In an embodiment of the present invention, the characteristics of panchromatic, multispectral and hyperspectral images can be combined to design a variety of masking micro-strategies of different scales, such as including but not limited to any one or more of the following: small-size random masking, large-size random masking, small-size channel random masking, large-size channel random masking, random band masking, and continuous band masking.
[0074] Reference Figure 3 , Figure 3 The diagram shows the effects of masking sample data using the above-mentioned various masking micro-strategies. (a) is small-size random masking, (b) is small-size channel random masking, (c) is large-size random masking, (d) is large-size channel random masking, (e) is random band masking, and (f) is edge band masking.
[0075] Figure 3 (a) and (c) show the masking methods of single-band images using different masking sizes. The masking in the spectral direction of these methods is randomly changed, which is consistent with the masking method of natural images. The different masking sizes in the figure are designed based on the rich spatial relationship characteristics of remote sensing images, aiming to enable the model to learn spatial context information at different spatial scales.
[0076] Figure 3 (b) and (d) also use different masking sizes to mask single-band images, but the masking method along the spectral direction is fixed. This design can guide the model to integrate and learn the global spatial and spectral information of the objects, and better learn and represent the high-level semantic information of the objects. The reason why hyperspectral images have higher recognition ability than multispectral images is that hyperspectral images contain more typical spectral features that can be represented, and these features are necessary to improve the separability between objects. Based on such a priori analysis, different bands are masked to guide the model to learn the typical spectral features of the objects, especially the diagnostic spectral features.
[0077] Figure 3 Middle (e) indicates random masking of different bands, but the masked bands are not continuous.
[0078] Figure 3 (f) in the middle represents masking of continuous bands. In this way, the model is guided to learn global spatial and spectral information, learn the spectral dependencies between cross-bands, and eliminate the influence of redundant information between different bands of hyperspectral images to a certain extent.
[0079] In a specific implementation, different masking strategies may be randomly selected for different types of image data sets to perform masking processing on the sample data in the sample set, which is not limited in this embodiment of the present invention.
[0080] Step 104 , spatially encoding the masked images, and embedding the marking information according to the used masking strategy to obtain the encoding data corresponding to each masked image.
[0081] When processing remote sensing images from different sources, if the masking strategy is randomly determined according to the corresponding type of image, it will lead to inconsistent label information embedded in the features of different images. Therefore, before encoding the masked image, the masked image can be mapped to a feature space of predefined dimensions by linear projection.
[0082] Since the spatial distribution of objects corresponding to image blocks from different sources is consistent, there is no need to embed any information specific to the object type during the linear projection process. Instead, we can use only an embedding vector of a certain dimension (such as 64 dimensions) to represent different masking strategies and use it as an image type feature.
[0083] It should be noted that, in order to enhance the understanding of the spatial structure of the image, both the masked image blocks and the unmasked image blocks in the masked image may be positionally encoded to ensure that the model can capture the relative spatial relationship between the image blocks.
[0084] Step 105: extract features from the encoded data to obtain multi-source features.
[0085] For example, an asymmetric encoder-decoder structure based on Transformer can be designed, a deep encoder network can be designed based on Transformer, and corresponding Transformer encoders can be designed for each of the three input images to extract the features of the unmasked image blocks in the three different masked images.
[0086] Step 106, fusing the multi-source features to obtain multi-source deep fusion features.
[0087] In an embodiment of the present invention, a multi-source feature cross-attention fusion mechanism is designed to achieve deep fusion of multi-source remote sensing image features; in the generative pre-training process of multi-source remote sensing images, a shallow Transformer decoder combined with position encoding is used to reconstruct the deep fusion features, and a spectral consistency constraint is introduced on the basis of the reconstruction loss to further improve the model feature extraction performance.
[0088] Furthermore, in some embodiments, interactive contrast learning may be performed on the multi-source features to optimize the encoder, that is, the encoder in the Transformer-based asymmetric encoder-decoder structure.
[0089] The multi-source feature fusion and interactive contrast learning processes in the embodiments of the present invention are described in detail below.
[0090] 1. Multi-source feature cross-attention fusion
[0091] Multi-source feature fusion is crucial for effectively learning the discriminant feature representation of multi-source remote sensing images. To this end, in the model provided by the present invention, the discriminant features in the spatial and spectral directions can be linearly jointly learned. Moreover, in order to give full play to the complementary advantages of spatial and spectral discriminant features and take into account the learning and extraction of deep features of different types of remote sensing images, a simple and effective multi-head cross-attention fusion mechanism is introduced in the scheme of the present invention to fuse multi-source remote sensing image information.
[0092] like Figure 4 , which is a schematic diagram of the working mechanism for performing cross-attention fusion of multi-source features in an embodiment of the present invention.
[0093] The traditional self-attention mechanism multiplies the input vector by three weight matrices to obtain the corresponding query, key, and key-value matrices.
[0094] The cross attention fusion mechanism in the embodiment of the present invention is to combine two input sources, such as Figure 4 Hyperspectral imagery in x hsi and the full-color image x pan (both are labeled image information) are used as query matrix and key-value matrix to calculate the attention map.
[0095] In the self-attention mechanism, label information plays an important role. Label information can be used to exchange information between feature data, thereby learning the high-level features of the unmasked image blocks of the entire image.
[0096] In addition, due to the large number of hyperspectral image bands, in order to avoid excessive computational consumption in the fusion process, the hyperspectral image is used as the main body in the cross-attention fusion process to fuse the panchromatic image features and multispectral image features. With full-color image information x added with marker information pan Taking the cross patch attention (CPA) fusion between the panchromatic image and the corresponding hyperspectral image as an example, the feature map from the panchromatic image is connected with the label information of the hyperspectral image. The process can be expressed as:
[0097]
[0098] x hp Represents the hyperspectral image x hsi and the full-color image x pan After fusion, the feature information Indicates the label information of the hyperspectral spectrum.
[0099] Then, in and x pan Cross attention fusion is performed between them, where:
[0100]
[0101] In formula (2), W q , W k and W v is the weight matrix, and then the query (q), key (k), and key value (v) matrices are normalized and randomly inactivated using the standard self-attention mechanism to obtain the labeled output corresponding to the two types of images. The whole process can be expressed as:
[0102]
[0103] In the formula, z is the intermediate variable; h d = d / h is the scale scaling factor, d represents the feature dimension, and h represents the number of attention heads; W lis the linear projection matrix; dropout(·) is the random dropout function, and the dropout probability is set to 0.1. When multiple types of image patches need to be fused (such as multi-spectral fusion), the multi-head cross patch attention (mCPA) mechanism can be used to further strengthen the fusion between different image patches and markers. When performing cross-attention fusion of multiple types of image patches, only the normalization layer (LN) and the residual connection mechanism are used instead of the multi-layer perceptron. This process can be described as:
[0104] y' cls =mCPA(LN(x hp ))
[0105]
[0106] Among them, y' cls represents the label information obtained by fusion of hyperspectral and panchromatic images; mCPA represents the multi-head cross attention mechanism; LN represents the normalization layer; x hp represents the fused features of hyperspectral and panchromatic patch embeddings;
[0107] Among them, y cls Represents the fused cross-attention information; Indicates the category label of the hyperspectral image; The patch embedding of the panchromatic image is equivalent to fusing the hyperspectral category label with the panchromatic patch embedding and then superimposing y' cls is the new vector.
[0108] In the cross-attention mechanism, the unmasked features of different remote sensing images are cross-fused, so that the model can not only extract the deep features of each type of image, but also fuse the features of different remote sensing images. After cross-attention fusion, the encoder combines the position embedding of the corresponding reconstruction task with the fused feature embedding, and calculates the loss at the masked position through the Transformer structure. The decoder uses both the fused feature information of multi-source remote sensing images and the specific information of the remote sensing images corresponding to the decoder, thereby realizing the learning and expression of high-level features of multi-source remote sensing images.
[0109] 2. Multi-source feature interactive comparison optimization
[0110] In an embodiment of the present invention, the cross-attention fusion mechanism can effectively realize the fusion of different types of remote sensing image features, so that the encoder has the ability to extract specific sources and fuse multi-source features. However, the encoder's ability to represent the ground object discrimination features in different types of remote sensing images is still insufficient. While completing feature fusion, enhancing the deep encoder's ability to extract ground object discrimination features can further enhance the encoder's learning and expression capabilities for ground object features. Therefore, in addition to accurately reconstructing different types of remote sensing images, a comparative learning task is also set in an embodiment of the present invention to explicitly align the spatial and spectral features of multi-source remote sensing images.
[0111] Panchromatic, multispectral and hyperspectral images all contain features of objects in both spatial and spectral dimensions. The generative self-supervised pre-trained classification model provided in the embodiment of the present invention does not perform band screening on multi-source remote sensing images, but takes the output of the encoder as the processing object. Therefore, the feature information of contrastive learning is richer, and the difficulty of feature comparison is also greater. In order to better utilize the high spatial resolution features contained in panchromatic and multispectral images, the feature interaction contrast loss between hyperspectral images and other multi-source remote sensing images is defined in the embodiment of the present invention. By aligning the features of multi-source remote sensing images, the deep encoder's ability to extract discriminative features is improved.
[0112] Ginseng Figure 5 , Figure 5 Schematic diagram of the multi-source feature interactive comparison learning process in an embodiment of the present invention.
[0113] Assuming that a batch has N samples, when only considering hyperspectral images and another high spatial resolution remote sensing image, the features obtained after encoder processing can be expressed as (z hsi ,z img ), when anchoring the hyperspectral image features in the i-th pair When the feature of another remote sensing image is As positive samples, the image features of other objects in the same batch is regarded as a negative sample, then its contrast loss is:
[0114]
[0115] Among them, the sim() function is a vector similarity function; τ is the temperature coefficient, which is the coefficient for adjusting the difficulty of feature distinction of the contrastive learning model; l is a rule function.
[0116] Similarly, when anchoring the features of other images When the hyperspectral image features It is a spatial feature Therefore, the multi-source remote sensing image interactive contrast loss includes the contrast loss Loss (zhsi ,z img ), and the contrast loss Loss(z img ,z hsi ) two parts, the interactive contrast loss function when there is only a single other type of remote sensing image can be defined as:
[0117] Loss(z hsi ,z img )=αLoss(z hsi ,z img )+(1-α)Loss(z img ,z hsi )(6)
[0118] Where α is a weight hyperparameter, which can be set to 0.7, for example. When a single type of remote sensing image is extended to multiple types of remote sensing images, the multi-source remote sensing image interactive contrast loss Loss (z hsi ,z msi ,z pan ), whose expression is:
[0119]
[0120] In formula (7), m represents the number of high spatial resolution image types.
[0121] Step 107: Reconstruct the multi-source deep fusion features using a decoder, and introduce a spectral consistency constraint based on the reconstruction loss.
[0122] For the input unlabeled multi-source remote sensing image data, taking hyperspectral image as an example, let represents the input unlabeled hyperspectral image block, (h, w, c) represents the number of rows, columns and bands of the image block respectively. The image block can be randomly masked based on the set masking rate. Let the masking data be The unmasked data is Linear projection and position encoding are used to process different types of unmasked remote sensing image areas to obtain corresponding label information (or embedded features), and the features of different types of remote sensing images output by the encoder are input into the cross-attention fusion process. This process can be expressed as:
[0123]
[0124] Among them, T e Represents the fusion results of three different types of remote sensing image features.
[0125] The encoder's learning of different types of remote sensing image features requires the Transformer structure's ability to learn long-range dependencies. The encoder can learn the features of a certain type of remote sensing image and provide key feature information for the reconstruction of that type of image. As mentioned earlier, the feature information between different types of remote sensing images has a rich complementary relationship. Therefore, the cross-attention fusion mechanism can drive the encoder to mine more general and advanced ground feature information, thereby establishing feature relationships between multi-source remote sensing images and providing advanced spatial-spectral feature information for the reconstruction of multi-source remote sensing images.
[0126] The decoder can be composed of shallow visual Transformers, and the decoders are designed for reconstruction tasks of different types of images. The decoder is based on the multi-source remote sensing image fusion features T extracted by the encoder and the cross-attention fusion process. e , combined with the position encoding information E pos , to reconstruct a specific remote sensing image, this process can be expressed as:
[0127]
[0128] In order to optimize the reconstruction process, the model combines the multi-task learning objectives to design Figure 6 The loss function shown.
[0129] Reference Figure 6 ,The optimization objective of this model includes two parts: reconstruction loss and spectral loss.
[0130] Reconstruction loss mainly refers to the pixel-by-pixel mean square error between the image reconstructed by the decoders of different tasks and the original image corresponding to the task. In order to avoid introducing additional system errors, the model only performs targeted loss calculation on the masked part, and the sum of the reconstruction losses of different types of images is the total reconstruction loss. The reconstruction loss can reflect the model's recovery of masked data of multi-source remote sensing images. The lower the reconstruction loss, the better the recovery, and thus the more high-level features learned. The reconstruction loss of the model can be expressed as:
[0131]
[0132] In the formula, Represent the reconstruction losses of hyperspectral, multispectral and panchromatic images respectively, and their calculation methods are:
[0133]
[0134] Hyperspectral images have important application value in the task of land object classification, thanks to the rich spatial and spectral information they contain, especially providing fine spectral feature information for effectively distinguishing different land objects. In the feature learning stage, the masked data is restored through the asymmetric encoder-decoder structure to learn the spatial spectrum features contained in the data. However, the reconstruction loss only plays a role of a global constraint. When processing hyperspectral images, the model's recovery of masked data cannot ensure that the spectral detail features are also fully restored. In order to meet this demand, a spectral loss function is designed in the scheme of the present invention to ensure that the model can learn the key spectral detail features. The spectral loss function uses the spectral cosine to calculate the similarity between the reconstructed spectral features and the original spectral features, and its expression is:
[0135]
[0136] The model reconstruction loss function can be expressed as:
[0137] Loss(x hsi ,x msi ,x pan )=βLoss mse +(1-β)Loss spectral (13)
[0138] In formula (13), β is a weight parameter, which can be set to 0.5 in this model.
[0139] So far, for the input unlabeled multi-source remote sensing image dataset (x hsi ,x msi ,x pan ), the model uses multi-source feature interaction contrast loss Loss(z hsi ,z msi ,z pan ) guides the model to mine the ground object discrimination features in multi-source remote sensing images, and uses the multi-source remote sensing image reconstruction loss Loss(x hsi ,x msi ,x pan ) guides the model to learn the deep feature information specific to the ground objects in multi-source remote sensing images, and the loss function of the model constructed in this way is:
[0140] Loss = λLoss(z hsi ,z msi ,z pan )+(1-λ)Loss(x hsi ,x msi ,x pan )(14)
[0141] In formula (14), λ is a weight hyperparameter, which can be set to 0.5 in this model.
[0142] The effect of using the method of the present invention to classify land objects is further illustrated by experiments below.
[0143] In the experiment, in order to simulate the few-sample scenario more realistically, the number of training samples for each category was set to 15. The classification results used OA (overall classification accuracy), AA (average classification accuracy) and Kappa coefficient to quantitatively evaluate the classification performance. The evaluation index range was standardized to 0-100%, and the higher the value, the better the classification performance. To ensure the reliability of the experimental results, the classification results of all methods were repeated 10 times under the same sample conditions and the average was taken. Tables 1 to 3 show the few-sample classification results of the Houston2018, GF14-1 and GF14-2 datasets, respectively.
[0144] Among them, Table 1 shows the classification results of different methods on the Houston2018 dataset.
[0145] Table 1
[0146] serial number TBCNN DBMA SSAN SepDG DMVL SSCLT S2FL GPMFCL 1 60.90 59.46 80.67 61.98 23.70 61.50 63.51 48.87 2 90.92 84.70 90.80 78.40 32.09 87.01 88.69 83.57 3 91.69 90.15 98.52 86.91 42.48 18.01 94.21 100.00 4 61.79 81.54 88.84 85.39 48.70 83.37 77.67 78.63 5 22.68 16.78 17.97 38.01 17.33 14.25 14.28 26.29 6 30.65 22.63 23.72 39.26 28.14 30.90 42.89 99.95 7 100.00 77.71 96.00 100.00 2.72 10.13 99.25 100.00 8 53.46 53.08 45.41 58.89 92.97 80.56 62.42 82.44 9 96.99 97.29 96.32 96.07 93.76 92.19 97.80 95.33 10 44.40 23.83 48.37 46.43 50.20 36.29 42.01 50.03 11 37.50 44.73 44.89 31.65 17.67 31.32 31.97 38.81 12 2.87 3.67 3.83 3.84 4.75 3.71 42.23 45.64 13 66.62 48.04 53.95 52.95 71.62 64.08 70.10 79.41 14 35.89 37.57 39.60 38.72 54.10 54.87 58.81 80.53 15 39.01 42.33 56.77 40.15 90.02 62.66 91.17 90.99 16 26.71 33.96 37.40 34.06 61.48 55.46 83.52 61.40 17 3.41 2.12 6.45 5.76 1.20 3.05 9.37 44.35 18 16.50 21.97 16.44 14.52 39.67 41.03 63.17 73.99 19 22.97 28.28 22.96 31.69 78.03 65.05 99.22 99.69 20 39.17 58.16 43.81 39.26 98.97 26.27 40.13 99.48 OA 51.91 53.99 53.81 55.65 56.45 58.60 63.57 74.66 AA 47.21 46.40 50.64 49.20 47.48 46.08 61.72 73.97 Kappa 45.54 46.98 47.37 48.67 48.62 51.33 57.07 68.59
[0147] Among them, Table 2 shows the classification results of different methods on the GF14-1 dataset
[0148] Table 2
[0149] serial number TBCNN DBMA SSAN SepDG DMVL SSCLT S2FL GPMFCL 1 30.31 20.09 24.69 14.25 37.32 8.77 26.14 51.30 2 66.44 85.16 62.09 60.01 6.15 55.87 98.79 91.17 3 15.82 12.53 17.06 33.36 76.98 14.67 28.63 48.37 4 99.44 99.75 99.01 98.60 95.76 89.16 98.75 99.25 5 7.94 25.76 12.79 8.58 94.91 23.21 25.97 92.22 6 13.65 16.71 13.08 19.12 20.4 13.08 24.97 50.51 7 27.71 16.83 19.87 24.28 63.95 18.98 84.51 70.45 8 94.98 94.93 90.91 90.61 69.00 82.11 96.71 96.84 9 3.46 5.45 7.07 5.01 33.91 1.24 8.34 57.54 10 22.03 37.45 33.78 33.86 83.92 24.21 64.39 89.64 11 99.70 98.94 100.00 100.00 98.48 99.44 98.51 99.53 12 82.24 86.14 91.75 90.56 92.91 56.10 82.19 91.14 13 29.58 49.40 26.28 42.11 32.91 61.43 82.36 71.49 14 45.80 37.40 48.04 55.71 68.65 25.79 75.11 93.46 15 87.88 68.32 73.03 82.14 55.74 17.80 87.61 91.41 16 18.03 24.58 15.14 14.34 19.36 20.52 60.57 85.12 17 46.09 24.62 41.24 57.77 54.62 29.42 83.44 96.88 18 58.30 47.58 59.30 77.23 70.77 29.74 76.03 90.12 OA 67.83 69.50 68.27 72.76 71.75 72.93 82.68 92.36 AA 47.19 47.31 46.40 50.42 59.76 37.31 66.83 81.47 Kappa 60.63 62.63 61.31 65.93 64.72 64.55 78.48 90.29
[0150] Among them, Table 3 shows the classification results of different methods on the GF14-2 dataset
[0151] Table 3
[0152]
[0153]
[0154] By observing and analyzing the classification results of different methods on the above three data sets, we can draw the following conclusions:
[0155] (1) On the three data sets, the classification model GPMFCL constructed by the embodiment of the present invention achieved the highest overall classification accuracy, which was 74.66%, 92.36% and 94.02% respectively, which was an average improvement of about 11.09% compared with S2FL. This achievement is mainly attributed to GPMFCL's full utilization of multi-source remote sensing image features and the advantages of model design. On the one hand, GPMFCL uses a generative self-supervised learning model to directly process the original multi-source remote sensing images, and can learn deeper ground features from multi-source remote sensing images. Through multi-source feature cross-attention fusion and interactive contrast optimization, GPMFCL can fully extract the spatial-spectral discriminant features of multi-source remote sensing images and their complementary relationships, and accurately reconstruct multi-source remote sensing images through a specific decoder, so that the encoder has the ability to mine the deep discriminant features of multi-source remote sensing data. On the other hand, the pre-training model significantly improves the discrimination ability of the classifier. The pre-training model relies on a large amount of unlabeled data for training. It can mine feature patterns of objects from multi-source remote sensing data and apply powerful feature extraction capabilities to the classification model, thereby improving the feature extraction ability of the classification model. With the support of a small number of samples, a higher classification accuracy can be obtained.
[0156] (2) The newly introduced DBMA classification method extracts and fuses multi-source remote sensing image features based on the attention mechanism and multi-branch network. Its classification accuracy is better than SSAN, which only uses hyperspectral images for classification, which also illustrates the important role of the attention mechanism. However, the classification accuracy of DBMA is still lower than that of the DMVL method, indicating that its ability to handle few-sample conditions is weak. Another new method, SepDG, is also based on multi-source remote sensing image classification. It uses grouped convolution and residual connection to extract and fuse multi-source remote sensing image features, and has achieved certain improvements in classification accuracy. However, CNN's ability to extract global feature dependencies is not as good as Transformer, so its classification accuracy has not been further improved.
[0157] (3) In the classification results of GF14-1 and GF14-2, it is observed that the classification accuracy of GPMFCL for crops such as rapeseed, peppers and broad beans is much higher than that of other comparison methods. The labeled samples of various supervised classification methods are only 15. It is difficult to establish the mapping relationship between the labeled samples and the data during the training process, resulting in insufficient learning of crop features, resulting in low classification accuracy. GPMFCL can achieve high classification accuracy thanks to its training based on a large amount of unlabeled data in the feature learning stage, which enables the model to learn the discriminant features of crops such as rapeseed that are different from other land features. Through the use of pre-trained models, the classifier has the ability to extract the discriminant features of crops, thereby obtaining more separable land features.
[0158] Accordingly, the present invention also provides a generative self-supervised pre-training classification model construction device, such as Figure 7 Shown is a structural schematic diagram of the device.
[0159] The generative self-supervised pre-trained classification model construction device 10 includes the following modules:
[0160] An image acquisition module 11 is used to acquire a hyperspectral image, a multispectral image, and a panchromatic image corresponding to the same target object;
[0161] A sample set generation module 12 is used to obtain corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets based on the hyperspectral image, multispectral image, and panchromatic image clipping to generate a sample set;
[0162] A sample processing module 13, configured to mask the sample data in the sample set using a multi-scale masking strategy to obtain a masked image;
[0163] The position encoding module 14 is used to perform spatial position encoding on the masked images and embed the marking information according to the masking strategy used to obtain the encoding data corresponding to each masked image;
[0164] A feature extraction module 15 is used to extract features from the coded data to obtain multi-source features;
[0165] A fusion module 16 is used to fuse the multi-source features to obtain deep fusion features;
[0166] The decoding module 17 is used to reconstruct the deep fusion features by using a decoder, and introduce a spectral consistency constraint based on the reconstruction loss.
[0167] In some embodiments, the feature extraction module 15 may utilize an asymmetric encoder-decoder structure based on a Transformer to perform feature extraction on the encoded data to obtain multi-source features.
[0168] like Figure 8 As shown, in another embodiment of a generative self-supervised pre-trained classification model construction device provided by the present invention, the generative self-supervised pre-trained classification model construction device 10 may further include:
[0169] The contrast learning module 18 is used to perform interactive contrast learning on the multi-source features to optimize the encoder.
[0170] For the specific implementation of each module in the device of the present invention, please refer to the description in the above embodiment of the method of the present invention, which will not be repeated here.
[0171] The generative self-supervised pre-training classification model construction method and device provided by the present invention combine the deep feature mining capability of generative learning with the feature discriminability enhancement effect of contrastive learning, which can extract the spatial-spectral features of ground objects contained in multi-source remote sensing images and enhance the separability of these features, effectively improving the classification performance of hyperspectral images under few-sample conditions and improving classification accuracy.
[0172] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0174] In several embodiments provided by the present invention, it should be understood that the disclosed device can be implemented in other ways.
[0175] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.
[0176] The “plurality” mentioned in the embodiments of the present invention refers to two or more than two.
[0177] The first, second, etc. descriptions appearing in the embodiments of the present invention are only used for illustration and distinction of the description objects. There is no order, nor does it indicate any special limitation on the number of devices in the embodiments of the present invention, and cannot constitute any limitation on the embodiments of the present invention.
[0178] The embodiment of the present invention further discloses a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is run, the steps of the method shown in the above embodiment can be executed. The storage medium may include a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. The storage medium may also include a non-volatile memory (non-volatile) or a non-transitory memory, etc.
[0179] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means.
[0180] The embodiments of the present invention are described in detail above. The present invention is described in detail using specific implementation methods herein. The description of the above embodiments is only used to help understand the method and system of the present invention. It is only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should belong to the scope of protection of the present invention, and the content of this specification should not be understood as limiting the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for constructing a generative self-supervised pre-trained classification model, characterized in that: The method comprises: Collect hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object; Based on the hyperspectral image, multispectral image, and panchromatic image, corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets are obtained by cutting, and a sample set is generated; Using a multi-scale masking strategy to perform masking processing on the sample data in the sample set to obtain a masked image; Performing spatial position encoding on the masked images, and embedding marking information according to the used masking strategy to obtain encoding data corresponding to each masked image; Performing feature extraction on the encoded data to obtain multi-source features; Fusing the multi-source features to obtain multi-source deep fusion features; The multi-source deep fusion features are reconstructed using a decoder, and a spectral consistency constraint is introduced based on the reconstruction loss.
2. The method for constructing a generative self-supervised pre-trained classification model according to claim 1, characterized in that: The step of obtaining corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets based on the hyperspectral image, multispectral image, and panchromatic image clipping to generate sample data includes: Taking any pixel in the hyperspectral image as a target pixel, cutting out a hyperspectral neighborhood image within a certain neighborhood range with the target pixel as the center, and generating a hyperspectral image subset according to the hyperspectral neighborhood image; Cut out a multispectral neighborhood image corresponding to the hyperspectral image neighborhood image, and generate a multispectral image subset according to the multispectral neighborhood image; Cut out a panchromatic neighborhood image corresponding to the hyperspectral image neighborhood image, and generate a panchromatic image subset based on the panchromatic neighborhood image; A sample set is generated according to the hyperspectral image subset, the multispectral image subset, and the panchromatic image subset, wherein each sample in the sample set includes a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image corresponding to a target pixel.
3. The method for constructing a generative self-supervised pre-trained classification model according to claim 1, characterized in that: The multi-scale masking strategy includes any one or more of the following: small-size random masking, large-size random masking, small-size channel random masking, large-size channel random masking, random band masking, and continuous band masking.
4. The method for constructing a generative self-supervised pre-trained classification model according to claim 1, characterized in that: The extracting features from the encoded data to obtain multi-source features comprises: Using an asymmetric encoder-decoder structure based on Transformer, feature extraction is performed on the encoded data to obtain multi-source features.
5. The method for constructing a generative self-supervised pre-trained classification model according to claim 1, characterized in that: The fusing of the multi-source features to obtain multi-source deep fusion features includes: The multi-source features are cross-attention fused. During the fusion process, the panchromatic and multi-spectral image features are fused with the hyperspectral image as the main body to obtain the multi-source deep fusion features.
6. The method for constructing a generative self-supervised pre-trained classification model according to claim 5, characterized in that: The reconstructing the multi-source deep fusion features by using a decoder includes: The image blocks are randomly masked based on the set masking rate, and the different types of unmasked image areas are processed using linear projection and position encoding to obtain the corresponding feature embeddings. The different types of image feature embeddings output by the encoder are input into the cross-attention fusion process. The decoder is used to reconstruct specific types of images based on the multi-source deep fusion features extracted by the encoder and the cross-attention fusion process combined with the position encoding information.
7. The method for constructing a generative self-supervised pre-trained classification model according to claim 6, characterized in that: The optimization goals of the reconstruction process for a particular type of image include: Reconstruction loss and spectral loss; The reconstruction loss refers to the pixel-by-pixel mean square error between the image reconstructed by the decoder of different tasks and the original image corresponding to the task; The spectral loss is represented by a spectral loss function, and the spectral loss function uses spectral cosine to calculate the similarity between the reconstructed spectral features and the original spectral features.
8. The method for constructing a generative self-supervised pre-trained classification model according to claim 4, characterized in that: The method further comprises: Perform interactive comparative learning on the multi-source features to optimize the encoder; The multi-source feature interaction contrast loss includes: the contrast loss from the hyperspectral image feature to other image features Loss(z hsi ,z img ), and the contrast loss Loss(z img ,z hsi ).
9. A generative self-supervised pre-trained classification model construction device, characterized in that: The device comprises: An image acquisition module, used for acquiring hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object; A sample set generation module, used for obtaining corresponding hyperspectral image subsets, multispectral image subsets, and panchromatic image subsets based on the hyperspectral image, multispectral image, and panchromatic image clipping to generate a sample set; A sample processing module, used for masking the sample data in the sample set using a multi-scale masking strategy to obtain a masked image; A position encoding module, used to perform spatial position encoding on the masked images and embed marking information according to the masking strategy used to obtain encoding data corresponding to each masked image; A feature extraction module, used to extract features from the encoded data to obtain multi-source features; A fusion module, used to fuse the multi-source features to obtain deep fusion features; A decoding module is used to reconstruct the deep fusion features by using a decoder, and introduce a spectral consistency constraint based on the reconstruction loss.
10. The generative self-supervised pre-training classification model construction device according to claim 9, characterized in that: The feature extraction module uses an asymmetric encoder-decoder structure based on Transformer to extract features from the encoded data to obtain multi-source features; The device also includes: The contrastive learning module is used to perform interactive contrastive learning on the multi-source features to optimize the encoder.
Citation Information
Patent Citations
Multi-source image combined urban land surface coverage classification method
CN113435253A
Construction method of deep mutual learning framework fusing spectral space information
CN113962913A
Hyperspectral remote sensing image classification method and device based on spectral reconstruction and medium
CN116486160A
Multi-modal remote sensing image classification method based on self-supervised pre-training
CN116503636A
Hyperspectral remote sensing image classification method based on mask spectral-space feature prediction
CN117115553A
Cited By
Construction method of remote sensing image recognition model based on self-supervised learning
CN120071164A
