Method and device for constructing multi-source heterogeneous feature contrast learning model
Through the multi-source heterogeneous feature comparison learning model, combining hyperspectral, multispectral and full-color images, the features are extracted and aligned, which solves the problem of insufficient spatial information of hyperspectral images, and improves the accuracy and separability of land objects classification. High-precision classification under the condition of few samples is realized.
Patent Information
- Application Number
- CN202411093361.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
AI Technical Summary
Under the condition of few samples, the separability of land feature of hyperspectral images is limited by insufficient spatial information, making it difficult to improve the performance of land feature classification model.
Using a multi-source heterogeneous feature comparison learning model, multi-source heterogeneous features are extracted by combining hyperspectral, multi-spectral and full-color images, and using a comparison learning method for interaction between multi-resolution spatial features and multi-source heterogeneous features to realize features Align to improve the accuracy of land objects classification.
Effectively utilize the rich feature information in multi-source remote sensing images, improve the accuracy and separability of land objects classification, overcome the influence of insufficient detailed information in the hyperspectral image space, and realize high-precision classification under the condition of few samples.
Smart Images

Figure CN118968305A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information security, and in particular to a method and device for constructing a multi-source heterogeneous feature comparison learning model. Background Art
[0002] With the widespread application of remote sensing technology in different fields, the scarcity of labeled samples of ground objects has increasingly become a key factor restricting the performance improvement of classification models. Under the condition of few samples, how to improve the separability of ground object features has become a hot topic in current research. Although hyperspectral images provide rich spectral information, their insufficient spatial information limits the further improvement of the separability of ground object features. Compared with hyperspectral images, panchromatic and multispectral images have higher spatial resolution and provide richer spatial information of ground objects. Therefore, effectively utilizing the spatial features of high spatial resolution images is of great significance to enhancing the separability of ground object features in hyperspectral images. Summary of the invention
[0003] The present invention provides a method and device for constructing a multi-source heterogeneous feature contrast learning model, so as to fully explore and utilize the rich feature information of land objects contained in multi-source remote sensing images, obtain richer and more discriminative representation of land object features, and improve the accuracy of land object classification.
[0004] To this end, the present invention provides the following technical solutions:
[0005] A method for constructing a multi-source heterogeneous feature contrast learning model, the method comprising:
[0006] Collect hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object;
[0007] Extracting a spectral curve of a target pixel based on the hyperspectral image, wherein the target pixel is any pixel in the hyperspectral image;
[0008] Acquire a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image of the target pixel respectively according to the hyperspectral image, the multispectral image, and the panchromatic image;
[0009] The spectral curve, the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image are used as input information, and features are extracted from each input information to obtain multi-source heterogeneous features;
[0010] Mapping the multi-source heterogeneous features into a unified feature space to obtain corresponding encoding information;
[0011] The coded information is subjected to comparative learning to achieve multi-source heterogeneous feature alignment and obtain a multi-source heterogeneous feature comparative learning model.
[0012] Optionally, the acquiring of a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image of the target pixel respectively according to the hyperspectral image, the multispectral image, and the panchromatic image comprises:
[0013] The information amount of each band of the hyperspectral image is evaluated by using image entropy to obtain the band with the highest image entropy, and a spatial image of a certain neighborhood range of the band is cut out with the target pixel as the center as the hyperspectral neighborhood image of the target pixel;
[0014] The information amount of each band of the multispectral image is evaluated by using image entropy to obtain the band with the highest image entropy, and the multispectral image corresponding to the band of the hyperspectral neighborhood image is cut out as the multispectral neighborhood image of the target pixel;
[0015] The panchromatic image corresponding to the hyperspectral neighborhood image band is cut out as the panchromatic neighborhood image of the target pixel.
[0016] Optionally, extracting features from each input information includes:
[0017] Extracting spectral features from the spectral curve using a spectral encoder to obtain corresponding spectral features;
[0018] The spatial encoder is used to extract spatial features of the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image to obtain corresponding spatial features.
[0019] Optionally, the spectral encoder and the spatial encoder are Transformer-based encoders.
[0020] Optionally, the extracting spectral features from the spectral curve using a spectral encoder to obtain corresponding spectral features includes:
[0021] Dividing the data corresponding to the spectral curve into a series of spectral blocks, and mapping the spectral blocks into an embedding feature space using linear projection;
[0022] Position encoding is performed on the mapped spectral blocks to obtain position encoding information of all spectral blocks;
[0023] A trainable label token is set in the embedded feature, and the embedded feature is input into the spectral encoder for feature learning to obtain the corresponding spectral feature.
[0024] Optionally, the spatial features of the neighborhood image are extracted using a spatial encoder, and the corresponding spatial features obtained include:
[0025] Converting the neighborhood image into a one-dimensional pixel sequence;
[0026] Performing linear projection on each pixel in the one-dimensional pixel sequence so as to map the pixel into a feature space;
[0027] Position encoding is performed on the mapped pixels to obtain position encoding information of all pixels;
[0028] The position encoding information is input into the spatial encoder for feature learning to obtain corresponding spatial features.
[0029] Optionally, mapping the multi-source heterogeneous features into a unified feature space includes: using a linear projection head to project the multi-source heterogeneous features into the unified feature space.
[0030] Optionally, the linear projection head includes: a fully connected layer, a batch normalization layer and a rectified linear unit.
[0031] Optionally, the contrastive learning includes any one or more of the following:
[0032] Contrastive learning based on multi-resolution spatial features;
[0033] Contrastive learning based on interaction of multi-source heterogeneous features.
[0034] Optionally, the contrastive learning based on multi-resolution spatial features includes:
[0035] Performing comparative learning based on multi-resolution spatial features on the coded information mapped based on the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image;
[0036] The multi-resolution spatial feature contrast loss function is:
[0037] Loss(z hsi ,z msi ,z pan )=Loss(z hsi ,z msi )+Loss(z pan ,z msi )+Loss(z hsi ,z pan );
[0038] Among them, z hsi ,z msi ,z pan Respectively represent the encoding information corresponding to the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image.
[0039] Optionally, the contrastive learning based on multi-source heterogeneous feature interaction includes:
[0040] Taking the coded information obtained based on the spectral mapping as the reference coded information, respectively comparing and learning the coded information obtained based on the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image mapping with the reference coded information;
[0041] The loss function of interactive contrast of multi-source heterogeneous features is:
[0042]
[0043] Among them, z spec ,z hsi ,z msi ,z pan Respectively represent the encoding information corresponding to the spectral curve, the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image.
[0044] A device for constructing a multi-source heterogeneous feature contrast learning model, the device comprising:
[0045] An image acquisition module, used to acquire hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object;
[0046] A spectral curve extraction module, used for extracting a spectral curve of a target pixel based on the hyperspectral image, wherein the target pixel is any pixel in the hyperspectral image;
[0047] A neighborhood image extraction module, used for respectively acquiring a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image of the target pixel according to the hyperspectral image, the multispectral image, and the panchromatic image;
[0048] A feature extraction module is used to take the spectral curve hyperspectral neighborhood image, multispectral neighborhood image, and panchromatic neighborhood image as input information, perform feature extraction on each input information, and obtain multi-source heterogeneous features;
[0049] A mapping module, used to map the multi-source heterogeneous features into a unified feature space to obtain corresponding encoding information;
[0050] The contrastive learning module is used to perform contrastive learning on the encoded information to achieve multi-source heterogeneous feature alignment and obtain a multi-source heterogeneous feature contrastive learning model.
[0051] Optionally, the device further comprises:
[0052] An image entropy screening module is used to evaluate the information content of each band of the hyperspectral image and the multispectral image to obtain the band with the highest image entropy;
[0053] The neighborhood image extraction module is specifically used to cut out a spatial image of a certain neighborhood range of the band with the highest image entropy in the hyperspectral image with the target pixel as the center as the hyperspectral neighborhood image of the target pixel, and to cut out a spatial image of a certain neighborhood range of the band with the highest image entropy in the multispectral image with the target pixel as the multispectral neighborhood image of the target pixel.
[0054] Optionally, the feature extraction module includes:
[0055] A spectral feature extraction unit, used for extracting spectral features from the spectral curve using a spectral encoder to obtain corresponding spectral features;
[0056] The spatial feature extraction unit is used to use a spatial encoder to extract spatial features from the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image to obtain corresponding spatial features.
[0057] Optionally, the comparative learning module includes any one or more of the following:
[0058] A multi-resolution spatial feature learning module, used for performing comparative learning on the coded information based on multi-resolution spatial features;
[0059] The multi-source heterogeneous feature interactive learning module is used to perform comparative learning on the encoded information based on the multi-source heterogeneous feature interaction.
[0060] A computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for constructing a multi-source heterogeneous feature contrast learning model.
[0061] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the method for constructing a multi-source heterogeneous feature contrast learning model.
[0062] The method and device for constructing a multi-source heterogeneous feature contrast learning model provided by the present invention utilize multi-resolution spatial feature contrast learning and multi-source heterogeneous feature interactive contrast learning to achieve automatic alignment of multi-source heterogeneous features, thereby overcoming the influence of insufficient spatial detail information of hyperspectral images on classification performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0064] Figure 1 It is a flow chart of the method for constructing a multi-source heterogeneous feature comparison learning model provided by the present invention;
[0065] Figure 2 is a schematic diagram of a process of extracting spectral features from a spectral curve in an embodiment of the present invention;
[0066] Figure 3 is a schematic diagram of a process of extracting spatial features from a neighborhood image in an embodiment of the present invention;
[0067] Figure 4 is a schematic diagram of a comparative learning process based on multi-resolution spatial features in an embodiment of the present invention;
[0068] Figure 5 Schematic diagram of a comparative learning process based on multi-source heterogeneous feature interaction in an embodiment of the present invention;
[0069] Figure 6 It is a structural schematic diagram of a device for constructing a multi-source heterogeneous feature comparison learning model provided by the present invention. DETAILED DESCRIPTION
[0070] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0071] In view of the problem that the classification model has poor performance and inaccurate classification due to the scarcity of ground object labeled samples in ground object feature classification, the present invention provides a method for constructing a multi-source heterogeneous feature contrast learning model, and a classification method and device based on the model, which fully mines and utilizes the rich ground object feature information contained in multi-source remote sensing images, comprehensively utilizes hyperspectral images, multispectral images, and panchromatic image information, and establishes a multi-source heterogeneous feature contrast learning model, so that the model has better performance. By using the model to classify ground objects, more accurate classification results can be obtained.
[0072] like Figure 1 FIG. 1 is a flow chart of a method for constructing a multi-source heterogeneous feature comparison learning model provided by the present invention, comprising the following steps:
[0073] Step 101 : Acquire a hyperspectral image, a multispectral image, and a panchromatic image corresponding to the same target object.
[0074] It should be noted that geometric registration has been performed between different images, and the method of geometric registration can adopt existing technology, which is not limited in the embodiment of the present invention.
[0075] Step 102: extracting a spectral curve of a target pixel based on the hyperspectral image, where the target pixel is any pixel in the hyperspectral image.
[0076] Step 103 , respectively obtaining a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image of the target pixel according to the hyperspectral image, the multispectral image, and the panchromatic image.
[0077] Entropy is a core concept in the field of thermodynamics and information theory. It can represent order or disorder in a physical sense, and can also measure the predictability or uncertainty of information. These characteristics of entropy provide important tools for understanding and describing various phenomena in nature. In information theory, Shannon entropy is one of the most representative entropies, which can quantitatively describe the uncertainty of information. Its expression is as follows:
[0078] E=-∑ i p i log2p i (1)
[0079] Among them, p i is the probability of an event, and E is the information entropy.
[0080] As a metric, entropy can also be used to measure the randomness or disorder of an image. For example, for a grayscale image, p i represents the probability of a certain gray level. i ∈[0,1], entropy is a positive interval measure. Therefore, the larger the entropy value, the higher the complexity of the image and the richer the information contained in the image.
[0081] For hyperspectral images, due to the different reflectivity of ground objects, the scene information will be expressed differently in different bands. For bands with larger entropy values, the scene information they contain is richer.
[0082] To this end, in the embodiment of the present invention, the extraction of each neighborhood image can be carried out in the following manner:
[0083] Firstly, the information amount of each band of the hyperspectral image is evaluated by using image entropy to obtain the band with the highest image entropy. Taking the target pixel as the center, a spatial image of a certain neighborhood range of the band is cut out as the hyperspectral neighborhood image of the target pixel.
[0084] Then, the information amount of each band of the multispectral image is evaluated using image entropy to obtain the band with the highest image entropy, and the multispectral image corresponding to the hyperspectral neighborhood image band is cut out as the multispectral neighborhood image of the target pixel.
[0085] Finally, the panchromatic image corresponding to the hyperspectral neighborhood image band is cut out as the panchromatic neighborhood image of the target pixel.
[0086] At this point, the four input information of the model are obtained, namely the spectral curve of the target pixel and its corresponding multi-resolution spatial image information, namely the above-mentioned hyperspectral neighborhood image, multispectral neighborhood image, and panchromatic neighborhood image.
[0087] Step 104 , taking the spectral curve, the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image as input information, extracting features from each input information to obtain multi-source heterogeneous features.
[0088] Specifically, a corresponding encoder may be set for each input information, and the encoder is used to extract features from the input information. The encoder may be, but is not limited to, an encoder based on a Transformer.
[0089] 1. For the spectral curve, a corresponding Transformer-based spectral encoder can be set, and the spectral encoder is used to extract spectral features of the spectral curve to obtain corresponding spectral features.
[0090] like Figure 2 FIG. 2 is a schematic diagram of a process of extracting spectral features from a spectral curve in an embodiment of the present invention. The specific process includes the following steps (21) to (23):
[0091] (21) Divide the data corresponding to the spectral curve into a series of spectral blocks, and use linear projection to map the spectral blocks into the embedded feature space. Each spectral block contains certain spectral information of the ground object, which corresponds to the local detail spectral feature.
[0092] (22) Position encoding is performed on the mapped spectral blocks to obtain position encoding information of all spectral blocks.
[0093] The positional relationship between different spectral features in the spectral curve is crucial for mining the diagnostic spectral features of ground objects. In order to extract the position-based spectral feature dependency, a random one-dimensional Gaussian distribution N(μ,σ 2 ) is used as a position encoder to assign a position code to each spectral block and merge the position codes of all spectral blocks into a complete position code information.
[0094] For example, the following formula (2) can be used for position encoding:
[0095]
[0096] Among them, E pos is the position encoding result, x i represents a spectral block, Represents x i The dimension of the feature space.
[0097] (23) A trainable tag token is set in the embedded feature, and the embedded feature is input into a spectral encoder for feature learning to obtain a corresponding spectral feature.
[0098] After completing linear projection and position encoding, a trainable tag token is set in the embedded feature. Then, the embedded feature is input into the spectral encoder for feature learning to obtain the corresponding spectral feature.
[0099] In an embodiment of the present invention, the spectral encoder may use a multi-head attention layer to extract fine spectral features of the spectral block itself and feature dependencies between spectral blocks to obtain a discriminative feature representation describing the spectrum of the object.
[0100] 2. For hyperspectral neighborhood images, multispectral neighborhood images, and panchromatic neighborhood images, a Transformer-based spatial encoder corresponding to each neighborhood image can be set to extract spatial features of the corresponding neighborhood image to obtain the corresponding spatial features.
[0101] It should be noted that for the above three types of neighborhood images, the extraction process of their spatial features is the same, the specific process is as follows: Figure 3 As shown, the process comprises the following steps (31) to (34):
[0102] (31) Convert the neighborhood image into a one-dimensional pixel sequence.
[0103] Each neighborhood image is a two-dimensional image, which is converted into a one-dimensional pixel sequence.
[0104] (32) Performing linear projection on each pixel in the one-dimensional pixel sequence so that the pixel is mapped into the feature space.
[0105] (33) Position encoding is performed on the mapped pixels to obtain position encoding information of all pixels.
[0106] The same position encoding method as that for position encoding the spectral block in the above step (22) can be adopted to ensure that the subsequent spatial encoder can effectively identify and extract the spatial dependency between the target pixel and other pixels in its neighborhood, thereby improving the spatial encoder's ability to learn the spatial features of the hyperspectral image.
[0107] (34) The position encoding information is input into a spatial encoder for feature learning to obtain corresponding spatial features.
[0108] Through the above design, spatial feature learning can fully explore and utilize the spatial information of hyperspectral images, and accurately extract the complex spatial relationship between the target pixels and their surrounding environment, thereby improving the separability of the spatial-spectral features of the ground objects and providing more accurate and rich spatial context information for hyperspectral image classification.
[0109] After the above feature extraction process, multi-source heterogeneous features can be obtained, namely, spectral features extracted from spectral curves, and spatial features extracted from hyperspectral neighborhood images, multispectral neighborhood images, and panchromatic neighborhood images respectively.
[0110] Step 105, mapping the multi-source heterogeneous features into a unified feature space to obtain corresponding encoding information.
[0111] Due to the multi-source heterogeneous characteristics of different input information, the extracted spectral features and spatial features are not features of the same feature space, and these features need to be unified into the same feature space.
[0112] In some embodiments of the present invention, a linear projection head can be used to process the output of each encoder (i.e., the above-mentioned spectral encoder and each spatial encoder) so that different information is projected into a unified feature space, that is, the multi-source heterogeneous features are projected into a unified feature space.
[0113] In a non-limiting embodiment, the linear projection head may adopt existing technologies, for example, it may include: a fully connected layer, a batch normalization layer, a rectified linear unit and other parts.
[0114] Step 106 , performing comparative learning on the encoded information to achieve multi-source heterogeneous feature alignment and obtain a multi-source heterogeneous feature comparative learning model.
[0115] The encoded information after linear projection can be used in the contrast feature space through contrast learning in a variety of different ways to achieve multi-source heterogeneous feature alignment.
[0116] The contrastive learning may include any one or more of the following: contrastive learning based on multi-resolution spatial features, contrastive learning based on interaction of multi-source heterogeneous features.
[0117] After contrast learning processing, its feature output is based on hyperspectral image pixels. Therefore, the features obtained after multi-source heterogeneous feature alignment can be directly used for hyperspectral image object classification. At this time, high-precision classification can be achieved using only a small number of labeled samples.
[0118] The comparative learning methods of the above-mentioned various methods are described in detail below.
[0119] 1. Contrastive learning based on multi-resolution spatial features
[0120] Reference Figure 4 , Figure 4 It is a schematic diagram of a comparative learning process based on multi-resolution spatial features in an embodiment of the present invention.
[0121] When processing multi-resolution spatial information from different remote sensing data sources (such as panchromatic, multispectral and hyperspectral images), each data source provides spatial information of objects at different scales. Although these images differ in spatial resolution, their expression of the spatial relationship of the same object is consistent within a certain neighborhood. This feature enables multi-resolution spatial features from different sources to be effectively aligned in feature space.
[0122] In order to achieve multi-scale spatial information alignment, the multi-source heterogeneous feature contrast learning model (MHFCL) provided in an embodiment of the present invention adopts the InfoNCE contrast loss function to optimize the similarity between features of different spatial resolutions, thereby reducing the distance between positive samples in the feature space and increasing the distance between negative samples. Among them, the positive samples are composed of the spatial features corresponding to the same object in images of different resolutions. For example, the low-resolution spatial features of a certain object in a hyperspectral image and the high-resolution spatial features of its corresponding area in a panchromatic image constitute a pair of positive samples; while the negative samples are composed of feature pairs that do not belong to the same object. Taking the spatial information contrast learning of hyperspectral and panchromatic images as an example, in order to close the coding information z corresponding to the hyperspectral image, hsi The coded information z corresponding to the full-color image pan The contrast loss function of the distance in the feature space is:
[0123]
[0124] In formula (3), sim is the vector similarity measurement function, and the superscripts i, j, and k represent different samples in the deep learning model.
[0125] in, Represents the similarity measure between two spatial feature samples; τ is the temperature coefficient used to adjust the attention of the loss function to distinguish difficult sample pairs, which can be set to τ = 0.5 in multi-resolution spatial contrastive learning.
[0126] It should be noted that in single-source feature contrast learning, the model enhances N sample data to obtain 2N sample data, and then constructs the single-source feature contrast loss. However, in the MHFCL model, in order to construct the contrast loss between multi-resolution spatial information, the multi-resolution spatial samples can be enhanced, and only the enhanced spatial data is used to construct the contrast loss. For samples of two types of objects, the number of samples is still 2N.
[0127] Since the spatial resolution of panchromatic images and hyperspectral images is quite different, and the spatial resolution of multispectral images is between the two, in order to more easily align the three types of spatial information, MHFCL is designed as follows: Figure 4 The multi-resolution spatial feature contrast learning strategy shown.
[0128] Therefore, combined with the following formula (4), the complete multi-resolution spatial feature contrast loss function can be obtained:
[0129] Loss(z hsi ,z msi ,z pan )=Loss(z hsi ,z msi )+Loss(z pan ,z msi )+Loss(z hsi ,z pan )(4)
[0130] 2. Contrastive learning based on interaction of multi-source heterogeneous features
[0131] For a certain object, its corresponding spectral curve can be obtained through hyperspectral images, and its corresponding multi-resolution spatial images can be obtained through panchromatic, multispectral and hyperspectral images, which together describe the spatial-spectral feature information of the object. Images of different spatial resolutions express the spatial contextual relationship of objects at different scales, and multi-resolution spatial information can be regarded as a multi-view expression of spectral curves. Therefore, based on the analysis that there is the same semantic information between multi-source spatial and spectral features, MHFCL maps the spatial and spectral information from different data sources (such as panchromatic, multispectral and hyperspectral images) into the same feature space, and realizes the alignment of multi-source heterogeneous features through interactive comparative learning between spectral features and multi-resolution spatial features, so that these features remain consistent and complementary in this space.
[0132] Reference Figure 5 , Figure 5 It is a schematic diagram of a comparative learning process based on the interaction of multi-source heterogeneous features in an embodiment of the present invention.
[0133] Figure 5In the example shown, the spectral curve is anchored, and then a certain spatial neighborhood image is interactively compared with the spectral curve for learning. spec Represents the spectrum curve information, which is extracted by the encoder to obtain the encoded information Z spec Similarly, X hsi , X msi , X pan Respectively represent the hyperspectral neighborhood image, multispectral neighborhood image, and panchromatic neighborhood image information, Z hsi , Z msi , Z pan They are the coded signals of the hyperspectral neighborhood image, multispectral neighborhood image, and panchromatic neighborhood image extracted by the encoder respectively.
[0134] When considering only the interactive contrast learning of spectral curves and one kind of spatial information, suppose there are N pairs of spectral curves (s, x img ) and its coded information (z spec ,z img ), when anchoring the spectral curve encoding information in the i-th pair of samples When MHFCL encodes the spatial features is regarded as a positive sample, and the encoding information of the spatial features of other objects in the same batch is regarded as a negative sample, then its contrast loss is:
[0135]
[0136] Similarly, when the encoding information of the anchor space feature When , the spectral feature encodes information is the encoding information of spatial features Therefore, the multi-source heterogeneous feature interactive contrast loss includes the contrast loss Loss(z spec ,z img ), and the contrast loss from spatial features to spectral features Loss(z img ,z spec ) two parts, the multi-source heterogeneous feature interactive contrast loss function when there is only a single spatial scale sample can be defined as:
[0137] Loss(z spec ,z img )=αLoss(z spec ,z img )+(1-α)Loss(z img ,z spec )(6)
[0138] In formula (6), α is a weight hyperparameter. In the training process of the MHFCL model, it can be set according to the principle of ensuring the important role of spectral curve features in the task of ground object classification and making full use of high spatial resolution feature information, such as setting α = 0.7.
[0139] By expanding the single spatial information to multi-resolution spatial information, the interactive contrast loss Loss(z spec ,z hsi ,z msi ,z pan ), whose expression is:
[0140]
[0141] In formula (7), m represents the number of spatial features. The reason for this setting is to provide
[0142] Improve the generalization performance of the model. When the model input has fewer spatial features or fewer image types, such as only hyperspectral and RGB images, m=2 can be set. The model can still construct the contrast loss between the spectral curve and the spatial image.
[0143] In order to make the multi-source heterogeneous features of the same object close in the feature space, MHFCL established a contrast loss function from two branches: multi-resolution spatial feature contrast learning and multi-source heterogeneous feature interactive contrast learning. Through the contrast loss function, the model is encouraged to embed the spectral information of the same pixel and its multi-resolution spatial information closer in the feature space, while at the same time, embedding the spatial and spectral information away from other pixels.
[0144] Through multi-resolution spatial feature contrast loss Loss(z hsi ,z msi ,z pan ) can guide the model to explore the multi-scale spatial contextual relationship of objects, through the interactive contrast loss Loss(z spec ,z hsi ,z msi ,z pan ) can guide the model to learn the multi-source spatial spectrum discriminant information of the ground object, and the contrast loss of the model constructed in this way is:
[0145] Loss = λLoss(z hsi ,z msi ,z pan )+(1-λ)Loss(z spec ,z hsi ,z msi ,z pan )(8)
[0146] In formula (8), λ is a weight hyperparameter, for example, λ=0.4 can be set.
[0147] During the training process of the MHFCL model, the multi-resolution spatial feature contrast loss mainly realizes the alignment of spatial features, while the multi-source heterogeneous feature interaction contrast loss strengthens the alignment and fusion of spatial and spectral features, playing a more important role.
[0148] The following examples illustrate the effect of using the method of the present invention to classify land objects.
[0149] The public dataset Houston2018 was selected, which contains hyperspectral images and high-resolution RGB images, which can fully demonstrate the performance of MHFCL in processing multi-source remote sensing images. At the same time, the panchromatic, multispectral and hyperspectral images of two Gaofen-14 satellite hyperspectral image datasets GF14-1 and GF14-2 were used as experimental data to further verify the accuracy and generalization ability of MHFCL. The experiment selected seven classification methods that performed well in the field of spatial and spectral feature extraction as comparison objects to deeply explore the ability of MHFCL in learning and extracting spatial and spectral discriminant features of multi-source remote sensing images.
[0150] First, SVM is selected to classify hyperspectral images to more intuitively compare the feature extraction capabilities of MHFCL. Secondly, SSRN and SSAN are used as supervised classification comparison methods. In addition, TBCNN and S2FL are selected as multi-source remote sensing image feature extraction comparison methods. TBCNN uses a dual-branch convolutional neural network to extract the spatial-spectral features of objects in hyperspectral images and other remote sensing data respectively, and then stacks and fuses the features of each branch and uses them for classification tasks, thereby combining hyperspectral images with other remote sensing data to improve the accuracy of object classification; S2FL is a generative self-supervised learning model based on multiple views to extract the spatial-spectral feature representation of multi-source remote sensing images. Finally, DMVL and SSCLT are selected as self-supervised learning comparison methods. In order to facilitate comparison, SSCLT also uses SVM as a classifier in the classification stage.
[0151] According to the experimental data, in order to simulate the few-sample scenario more realistically, the number of training samples for each category in the Houston2018, GF14-1 and GF14-2 datasets is 15. The classification results use OA, AA and Kappa coefficients to quantitatively evaluate the classification performance. The evaluation index range is standardized to 0-100%, and the higher the value, the better the classification performance. In order to ensure the reliability of the experimental results, the classification results of all methods are repeated 10 times under the same sample conditions and the average is taken. The few-sample classification results of the Houston2018, GF14-1 and GF14-2 datasets are shown respectively.
[0152] Table 1 below shows the classification results of different methods on the Houston 2018 dataset.
[0153] Table 1
[0154]
[0155]
[0156] Table 2 below shows the classification results of different methods on the GF14-1 dataset.
[0157] Table 2
[0158]
[0159]
[0160] Table 3 shows the classification results of different methods on the GF14-2 dataset.
[0161] Table 3
[0162]
[0163]
[0164] By observing and analyzing the classification results of different methods on the above three data sets, it can be seen that the classification performance of different types of classification methods varies greatly, and the difference between the highest and lowest overall classification accuracy methods is 26.18%, 25.73% and 20.64% respectively. This also shows that the classification of objects under the condition of few samples is a common problem in remote sensing applications. The analysis of the experimental results can also draw the following conclusions:
[0165] (1) From the perspective of the use of labeled samples, the classification methods based on supervised learning (such as SVM, SSRN, SSAN and TBCNN) require a large number of labeled samples for training during the classification process. When the number of samples is small, these methods are prone to overfitting, resulting in low classification accuracy. The highest overall classification accuracy on the three datasets is only 53.81%, 68.27% and 72.19%. In contrast, the classification methods based on self-supervised learning (such as DMVL, SSCLT, S2FL and MHFCL) do not rely on labeled samples in the feature extraction stage and can learn more separable feature representations of objects, thus achieving better classification accuracy under the condition of few samples. MHFCL achieved the highest overall classification accuracy on all three datasets.
[0166] (2) Classification methods based on attention mechanisms can better extract spatial-spectral feature information of ground objects. For example, SSAN uses attention mechanisms in both spatial and spectral dimensions to extract key feature information, and therefore achieves the highest classification accuracy among supervised learning methods. This shows that the attention mechanism plays an important role in the feature extraction of ground objects in multi-source remote sensing images. The multi-head self-attention mechanism in the Transformer structure can capture more feature dependencies, which is crucial for improving the separability of ground object features. From the results of the self-supervised classification method, it can be seen that both DMVL and S2FL methods are constructed based on CNN networks, and their overall classification accuracy on the three data sets is 1.97% and 6% lower than that of SSCLT and MHFCL on average, respectively. Among them, DMVL and SSCLT are both based on contrastive learning methods to classify hyperspectral images, and the difference in classification accuracy can better show the advantages of Transformer.
[0167] (3) TBCNN uses a dual-branch network to utilize the features of multi-source remote sensing data, but due to the limitation of the number of samples, it does not fully utilize the advantages of multi-source data features. In contrast, the S2FL and MHFCL methods based on multi-source remote sensing images achieved better classification results. MHFCL achieved the highest overall classification accuracy on the three datasets, which were 70.76%, 84.72%, and 86.79%, respectively. This is due to the model's full mining of the feature information of multi-source remote sensing data, especially the use of rich spatial feature information in high spatial resolution images, which effectively compensates for the limitation of insufficient spatial information of hyperspectral images, thereby improving the classification performance of objects.
[0168] (4) The classification accuracy of various classification methods for the Houston 2018 dataset is generally not high, which is related to the complexity of the objects in the dataset. The Houston 2018 dataset contains 20 object categories, and the distinction between some categories is not large, so the model needs to be able to extract feature representations with diagnostic characteristics. The classification accuracy of MHFCL on the GF14-1 and GF14-2 datasets is greatly improved compared with the Houston 2018 dataset. On the one hand, the GF14-1 and GF14-2 datasets have more detailed spectral feature information of objects, which significantly improves the separability of object features; on the other hand, in addition to hyperspectral images, these two datasets also contain panchromatic, multispectral and hyperspectral images, which provide more comprehensive and detailed spatial and spectral feature information for object classification, thereby helping to improve classification performance.
[0169] Accordingly, an embodiment of the present invention also provides a device for constructing a multi-source heterogeneous feature contrast learning model, such as Figure 6FIG. 1 is a schematic diagram of the structure of the device. The device 900 for constructing a multi-source heterogeneous feature comparison learning model includes the following modules:
[0170] An image acquisition module 901 is used to acquire a hyperspectral image, a multispectral image, and a panchromatic image corresponding to the same target object;
[0171] A spectral curve extraction module 902 is used to extract a spectral curve of a target pixel based on the hyperspectral image, where the target pixel is any pixel in the hyperspectral image;
[0172] A neighborhood image extraction module 903 is used to respectively obtain a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image of the target pixel according to the hyperspectral image, the multispectral image, and the panchromatic image;
[0173] The feature extraction module 904 is used to use the spectral curve hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image as input information, perform feature extraction on each input information, and obtain multi-source heterogeneous features;
[0174] A mapping module 905 is used to map the multi-source heterogeneous features into a unified feature space to obtain corresponding encoding information;
[0175] The contrast learning module 906 is used to perform contrast learning on the encoded information to achieve multi-source heterogeneous feature alignment and obtain a multi-source heterogeneous feature contrast learning model.
[0176] In some embodiments, the multi-source heterogeneous feature contrast learning model construction device 900 may also include: an image entropy screening module (not shown), which is used to evaluate the information content of each band of the hyperspectral image and the multispectral image to obtain the band with the highest image entropy. Accordingly, the neighborhood image extraction module 903 can cut out a spatial image of a certain neighborhood range of the band with the highest image entropy in the hyperspectral image with the target pixel as the hyperspectral neighborhood image of the target pixel, and cut out a spatial image of a certain neighborhood range of the band with the highest image entropy in the multispectral image with the target pixel as the multispectral neighborhood image of the target pixel.
[0177] In some embodiments, the feature extraction module 904 may include the following units:
[0178] A spectral feature extraction unit, used for extracting spectral features from the spectral curve using a spectral encoder to obtain corresponding spectral features;
[0179] The spatial feature extraction unit is used to use a spatial encoder to extract spatial features from the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image to obtain corresponding spatial features.
[0180] In some embodiments, the contrast learning module 906 may include a multi-resolution spatial feature learning module and / or a multi-source heterogeneous feature interactive learning module.
[0181] The multi-resolution spatial feature learning module is used to perform contrast learning on the coded information based on multi-resolution spatial features;
[0182] The multi-source heterogeneous feature interactive learning module is used to perform comparative learning on the encoded information based on multi-source heterogeneous feature interaction.
[0183] For more details on the above modules, please refer to the description in the previous embodiments of the present invention, which will not be repeated here.
[0184] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0185] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0186] In several embodiments provided by the present invention, it should be understood that the disclosed device can be implemented in other ways.
[0187] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.
[0188] The “plurality” mentioned in the embodiments of the present invention refers to two or more than two.
[0189] The first, second, etc. descriptions appearing in the embodiments of the present invention are only used for illustration and distinction of the description objects. There is no order, nor does it indicate any special limitation on the number of devices in the embodiments of the present invention, and cannot constitute any limitation on the embodiments of the present invention.
[0190] The embodiment of the present invention further discloses a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is run, the steps of the method shown in the above embodiment can be executed. The storage medium may include a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. The storage medium may also include a non-volatile memory (non-volatile) or a non-transitory memory, etc.
[0191] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means.
[0192] The embodiments of the present invention are described in detail above. The present invention is described in detail using specific implementation methods herein. The description of the above embodiments is only used to help understand the method and system of the present invention. It is only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should belong to the scope of protection of the present invention, and the content of this specification should not be understood as limiting the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for constructing a multi-source heterogeneous feature contrast learning model, characterized in that: The method comprises: Collect hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object; Extracting a spectral curve of a target pixel based on the hyperspectral image, wherein the target pixel is any pixel in the hyperspectral image; Acquire a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image of the target pixel respectively according to the hyperspectral image, the multispectral image, and the panchromatic image; The spectral curve, the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image are used as input information, and features are extracted from each input information to obtain multi-source heterogeneous features; Mapping the multi-source heterogeneous features into a unified feature space to obtain corresponding encoding information; The coded information is subjected to comparative learning to achieve multi-source heterogeneous feature alignment and obtain a multi-source heterogeneous feature comparative learning model.
2. The method for constructing a multi-source heterogeneous feature contrast learning model according to claim 1, characterized in that: The step of respectively acquiring the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image of the target pixel according to the hyperspectral image, the multispectral image, and the panchromatic image comprises: The information amount of each band of the hyperspectral image is evaluated by using image entropy to obtain the band with the highest image entropy, and a spatial image of a certain neighborhood range of the band is cut out with the target pixel as the center as the hyperspectral neighborhood image of the target pixel; The information amount of each band of the multispectral image is evaluated by using image entropy to obtain the band with the highest image entropy, and the multispectral image corresponding to the band of the hyperspectral neighborhood image is cut out as the multispectral neighborhood image of the target pixel; The panchromatic image corresponding to the hyperspectral neighborhood image band is cut out as the panchromatic neighborhood image of the target pixel.
3. The method for constructing a multi-source heterogeneous feature contrast learning model according to claim 1, characterized in that: The feature extraction of each input information comprises: Extracting spectral features from the spectral curve using a Transformer-based spectral encoder to obtain corresponding spectral features; The spatial features of the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image are extracted using a Transformer-based spatial encoder to obtain corresponding spatial features.
4. The method for constructing a multi-source heterogeneous feature contrast learning model according to claim 3, characterized in that: The spectral feature extraction of the spectral curve by using a spectral encoder to obtain the corresponding spectral feature includes: Dividing the data corresponding to the spectral curve into a series of spectral blocks, and mapping the spectral blocks into an embedding feature space using linear projection; Position encoding is performed on the mapped spectral blocks to obtain position encoding information of all spectral blocks; A trainable label token is set in the embedded feature, and the embedded feature is input into the spectral encoder for feature learning to obtain the corresponding spectral feature.
5. The method for constructing a multi-source heterogeneous feature contrast learning model according to claim 3, characterized in that: The spatial encoder is used to extract spatial features from the neighborhood image, and the corresponding spatial features include: Converting the neighborhood image into a one-dimensional pixel sequence; Performing linear projection on each pixel in the one-dimensional pixel sequence so as to map the pixel into a feature space; Position encoding is performed on the mapped pixels to obtain position encoding information of all pixels; The position encoding information is input into the spatial encoder for feature learning to obtain corresponding spatial features.
6. The method for constructing a multi-source heterogeneous feature contrast learning model according to claim 1, characterized in that: The contrastive learning based on multi-resolution spatial features includes: Performing comparative learning based on multi-resolution spatial features on the coded information mapped based on the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image; The multi-resolution spatial feature contrast loss function is: Loss(z hsi ,z msi ,z pan )=Loss(z hsi ,z msi )+Loss(z pan ,z msi )+Loss(z hsi ,z pan ); Among them, z hsi ,z msi ,z pan Respectively represent the encoding information corresponding to the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image.
7. The method for constructing a multi-source heterogeneous feature contrast learning model according to claim 1, characterized in that: The contrastive learning based on multi-source heterogeneous feature interaction includes: Taking the coded information obtained based on the spectral mapping as the reference coded information, respectively comparing and learning the coded information obtained based on the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image mapping with the reference coded information; The loss function of interactive contrast of multi-source heterogeneous features is: Among them, z spec ,z hsi ,z msi ,z pan Respectively represent the encoding information corresponding to the spectral curve, the hyperspectral neighborhood image, the multispectral neighborhood image, and the panchromatic neighborhood image.
8. A device for constructing a multi-source heterogeneous feature contrast learning model, characterized in that: The device comprises: An image acquisition module, used to acquire hyperspectral images, multispectral images, and panchromatic images corresponding to the same target object; A spectral curve extraction module, used for extracting a spectral curve of a target pixel based on the hyperspectral image, wherein the target pixel is any pixel in the hyperspectral image; A neighborhood image extraction module, used for respectively acquiring a hyperspectral neighborhood image, a multispectral neighborhood image, and a panchromatic neighborhood image of the target pixel according to the hyperspectral image, the multispectral image, and the panchromatic image; A feature extraction module is used to take the spectral curve hyperspectral neighborhood image, multispectral neighborhood image, and panchromatic neighborhood image as input information, perform feature extraction on each input information, and obtain multi-source heterogeneous features; A mapping module, used to map the multi-source heterogeneous features into a unified feature space to obtain corresponding encoding information; The contrastive learning module is used to perform contrastive learning on the encoded information to achieve multi-source heterogeneous feature alignment and obtain a multi-source heterogeneous feature contrastive learning model.
9. The device for constructing a multi-source heterogeneous feature contrast learning model according to claim 8, characterized in that: The device also includes: An image entropy screening module is used to evaluate the information content of each band of the hyperspectral image and the multispectral image to obtain the band with the highest image entropy; The neighborhood image extraction module is specifically used to cut out a spatial image of a certain neighborhood range of the band with the highest image entropy in the hyperspectral image with the target pixel as the center as the hyperspectral neighborhood image of the target pixel, and to cut out a spatial image of a certain neighborhood range of the band with the highest image entropy in the multispectral image with the target pixel as the multispectral neighborhood image of the target pixel.
10. The device for constructing a multi-source heterogeneous feature contrast learning model according to claim 8 or 9, characterized in that: The comparative learning module includes any one or more of the following: A multi-resolution spatial feature learning module, used for performing comparative learning on the coded information based on multi-resolution spatial features; The multi-source heterogeneous feature interactive learning module is used to perform comparative learning on the encoded information based on the multi-source heterogeneous feature interaction.
