Multi-level contrastive few-shot learning method for cross-domain hyperspectral image classification

Abstract

The present invention discloses a multi-level contrast small sample learning method for cross-domain hyperspectral image classification, which obtains a source domain hyperspectral image dataset and a target domain hyperspectral image dataset; establishes a source domain meta-task based on the source domain hyperspectral image dataset, and establishes a target domain meta-task based on the target domain hyperspectral image dataset; trains an image classifier using the source domain meta-task, the target domain meta-task and unlabeled hyperspectral image data in the target domain; the present invention can make the class distribution of the source domain and the target domain closer by adding class distribution loss to the loss function, and at the same time expand the class difference in each domain; by adding category loss to the loss function, the category prior knowledge can be fully utilized, and the potential semantic connection between categories can be mined, thereby increasing the classification accuracy of the image classifier for hyperspectral images.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image processing, and in particular relates to a multi-level contrast small sample learning method for cross-domain hyperspectral image classification. Background Art

[0002] Hyperspectral images are taken by carrying a hyperspectral imager on a satellite. It combines imaging technology and spectral technology to obtain one-dimensional spectral information while detecting the two-dimensional space of ground objects, so as to accurately identify the land cover category of hyperspectral images.

[0003] With the development of deep learning and the improvement of computing power, deep learning-based methods have been widely used in hyperspectral image classification. However, due to the limited number of labeled hyperspectral images and the insufficient explanatory power of existing models, in order to solve these problems, the small sample learning method uses the source domain dataset to help the target domain dataset train the classifier, where the hyperspectral image data in the source domain dataset and the hyperspectral image data in the target domain dataset come from different sources, the source domain dataset has enough labeled hyperspectral image data, and the target domain dataset has a small amount of labeled hyperspectral image data and hyperspectral image data that needs to be classified.

[0004] However, in practical applications, there is usually a problem of large distribution differences between the source domain dataset and the target domain dataset, which seriously affects the classification performance of the image classifier in the target domain. In order to overcome the problem of domain distribution differences, many studies have introduced various domain alignment methods in small sample learning. Typical domain alignment methods include methods based on adversarial learning, which use domain discriminators to make the features of hyperspectral image data in the source domain dataset and the features of hyperspectral image data in the target domain dataset indistinguishable. However, this method is prone to class conflict problems. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-level contrast small sample learning method for cross-domain hyperspectral image classification, which trains the classifier by adding category loss and class distribution loss to improve the image classification accuracy.

[0006] The present invention adopts the following technical solution: a multi-level contrast small sample learning method for cross-domain hyperspectral image classification, comprising the following steps:

[0007] A source domain hyperspectral image dataset and a target domain hyperspectral image dataset are obtained; wherein the number of hyperspectral image data in the source domain hyperspectral image dataset is greater than the number of hyperspectral image data in the target domain hyperspectral image dataset, the hyperspectral image data in the source domain hyperspectral image dataset and the hyperspectral image data in the target domain hyperspectral image dataset are both labeled, and the number of label categories in the source domain hyperspectral image dataset is greater than the number of label categories in the target domain hyperspectral image dataset;

[0008] A source domain meta-task is established based on a source domain hyperspectral image dataset, and a target domain meta-task is established based on a target domain hyperspectral image dataset. The meta-task parameters in the source domain meta-task and the target domain meta-task are the same.

[0009] The image classifier is trained using source domain meta-task, target domain meta-task and unlabeled hyperspectral image data in the target domain; the loss functions used for training include category loss and class distribution loss.

[0010] Furthermore, the class distribution loss calculation method includes:

[0011] Generate class distribution of label categories in source domain meta-task;

[0012] Distribution fitting is performed on several unlabeled hyperspectral image data in the target domain to obtain the class distribution of label categories in the target domain; wherein the number of label categories in the target domain is equal to the number of label categories in the source domain meta-task;

[0013] The mean distribution alignment loss is calculated based on the class distribution mean of the label category in the source domain meta-task and the class distribution mean of the label category in the target domain;

[0014] Calculate the variance distribution alignment loss based on the class distribution variance of the label category in the source domain meta-task and the class distribution variance of the label category in the target domain;

[0015] The class distribution loss is obtained by summing the mean distribution alignment loss and the variance distribution alignment loss.

[0016] Furthermore, the mean distribution alignment loss includes:

[0017] ,

[0018] in, represents the mean distribution alignment loss, Indicates the source domain s In the source domain meta-task The class distribution mean of the label categories, Indicates the source domain s The class distribution mean of the k-th label category in the source domain meta-task, Indicates the target domain tThe class distribution mean of the j-th label category, Indicates the target domain t The class distribution mean of the kth label category, N represents the number of label categories, and the number of label categories in the target domain is equal to the number of label categories in the source domain meta-task; when The corresponding label categories and When the corresponding label category is a positive sample pair, and ,otherwise and ;when The corresponding label categories and When the corresponding label category is a negative sample pair, ,otherwise ;when The corresponding label categories and When the corresponding label category is a negative sample pair, ,otherwise ; Represents a constant.

[0019] Furthermore, the variance distribution alignment loss includes:

[0020] ,

[0021] in, represents the variance distribution alignment loss, Indicates the source domain s In the source domain meta-task The class distribution variance of the label categories, Indicates the target domain t The class distribution variance of the j-th label category, express and The Euclidean distance of The corresponding label categories and When the corresponding label category is a positive sample pair, ,otherwise .

[0022] Furthermore, the category loss is composed of the source domain meta-task category loss and the target domain meta-task category loss.

[0023] Furthermore, the calculation method of the source domain meta-task category loss includes:

[0024] Extracting a second feature of each support sample in the source domain meta-task based on a second feature extractor;

[0025] Establishing a source domain hybrid dictionary based on the second feature;

[0026] The first feature of each query sample in the source domain meta-task is key-matched with each element in the source domain mixed dictionary, and the source domain meta-task category loss is calculated based on the matching results.

[0027] Furthermore, the source domain meta-task category loss is calculated based on the matching results, including:

[0028] ,

[0029] in, represents the source domain meta-task category loss, js represents the ordinal number of the query sample in the source domain meta-task, represents the number of query samples in each label category in the source domain meta-task, h represents the number of rows in the source domain mixed dictionary, C S represents the number of label categories in the source domain hyperspectral image dataset, w represents the number of columns of the source domain hybrid dictionary, and DL represents the total number of columns of the source domain hybrid dictionary; Indicates the label category of the jsth query sample, Represents the element in the hth row and wth column in the source domain hybrid dictionary The label category, when and At the same time, ,otherwise ; Indicates that the query sample in the source domain meta-task is extracted by the first feature extractor The first characteristic of and different, ,otherwise .

[0030] Furthermore, the calculation method of the target domain meta-task category loss is the same as that of the source domain meta-task category loss.

[0031] Furthermore, the first feature extractor is a feature extractor in an image classifier, the second feature extractor has the same structure as the first feature extractor, and the parameter optimization method of the second feature extractor is different from that of the first feature extractor.

[0032] Furthermore, the loss function also includes small sample learning loss;

[0033] The few-shot learning loss consists of the cross-entropy loss of the query sample in the source domain meta-task and the cross-entropy loss of the query sample in the target domain meta-task.

[0034] The beneficial effects of the present invention are as follows: by adding class distribution loss to the loss function, the present invention can make the class distribution of the source domain and the target domain closer, while expanding the inter-class differences in each domain; moreover, by adding category loss to the loss function, the category prior knowledge can be fully utilized to explore the potential semantic connections between categories, thereby increasing the classification accuracy of the image classifier for hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of a multi-level contrast small sample learning method for cross-domain hyperspectral image classification according to an embodiment of the present invention;

[0036] Figure 2 Graph showing classification results of an image classifier obtained using different domain adaptation methods in an embodiment of the present invention on an Indian Pines dataset;

[0037] Figure 3 This is a diagram showing the classification results of the image classifier obtained using different domain adaptation methods in an embodiment of the present invention on the Salinas dataset. DETAILED DESCRIPTION

[0038] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] In image classifier training for hyperspectral images, the source and target domains usually have the same semantic categories, such as water, road, mountains, etc. The same or similar semantic categories should have similar feature representations to achieve effective domain alignment, but current methods often ignore the semantic category consistency between the source and target domains.

[0040] The present invention combines small sample learning technology with domain adaptation methods to solve the problem of large differences in data distribution between source domain and target domain in small sample learning, thereby further improving the generalization ability and classification performance of image classifiers.

[0041] Specifically, the present invention discloses a multi-level contrast small sample learning method for cross-domain hyperspectral image classification, such as Figure 1As shown, the method comprises the following steps: step S110, obtaining a source domain hyperspectral image dataset and a target domain hyperspectral image dataset; wherein, the number of hyperspectral image data in the source domain hyperspectral image dataset is greater than the number of hyperspectral image data in the target domain hyperspectral image dataset, the hyperspectral image data in the source domain hyperspectral image dataset and the hyperspectral image data in the target domain hyperspectral image dataset are both labeled, and the number of label categories in the source domain hyperspectral image dataset is greater than the number of label categories in the target domain hyperspectral image dataset; step S120, establishing a source domain meta-task based on the source domain hyperspectral image dataset, and establishing a target domain meta-task based on the target domain hyperspectral image dataset, and the meta-task parameters in the source domain meta-task and the target domain meta-task are the same; step S130, training an image classifier using the source domain meta-task, the target domain meta-task and the unlabeled hyperspectral image data in the target domain; wherein the loss function used for training comprises a category loss and a class distribution loss.

[0042] By adding class distribution loss to the loss function, the present invention can make the class distribution of the source domain and the target domain closer, while expanding the inter-class differences in each domain; by adding category loss to the loss function, the category prior knowledge can be fully utilized to explore the potential semantic connections between categories, thereby increasing the classification accuracy of the image classifier for hyperspectral images.

[0043] In the present invention, the source domain hyperspectral image dataset and the target domain hyperspectral image dataset refer to hyperspectral image datasets from different sources. For example, the source domain hyperspectral image dataset is derived from hyperspectral images taken by a hyperspectral camera carried by an unmanned aerial vehicle, and the target domain hyperspectral image dataset is derived from hyperspectral images taken by a hyperspectral camera carried by a satellite. When it is necessary to classify the target domain hyperspectral image data, since only a small number of hyperspectral images taken by a hyperspectral camera carried by a satellite have label categories, the target domain hyperspectral image dataset is not sufficient to train an image classifier that meets the expected requirements. At this point, the source domain hyperspectral image dataset can be used to assist the target domain hyperspectral image dataset to train the image classifier, thereby training an image classifier that meets the expected requirements.

[0044] In the method of the present invention, after obtaining the above-mentioned hyperspectral image, it is first necessary to unify the dimensions of the source domain hyperspectral image and the target domain hyperspectral image, such as unifying both into a hyperspectral image with 100 bands. Specifically, the dimension of the hyperspectral image can be mapped using a mapping layer in the prior art.

[0045] Hyperspectral images are usually in the form of H × W × C A three-dimensional data cube representation of H and WRepresents the height and width of the image respectively. C It represents the number of spectral bands. When performing image processing and classification tasks, relying solely on the information of a single pixel may not be enough to capture rich context and spatial features. Therefore, the hyperspectral image is divided into image blocks to make full use of the local information around each pixel.

[0046] Specifically, in the embodiment of the present invention, the hyperspectral image is divided into 9×9× C 1-sized image patch, C 1 represents the number of bands after mapping, which means that for each pixel in the hyperspectral image, four pixels are extended upward, downward, leftward, and rightward with the pixel as the center, thus forming an image block with a cross section of 9 rows and 9 columns. It should be noted that if the edge pixels are not enough to form a complete image block, the image boundary is extended by zero padding.

[0047] This image block not only contains the spectral features of the central pixel, but also incorporates the spatial information of its surrounding neighborhood, thus providing richer and more useful features for the classification task. Ultimately, the label category of the central pixel of the image block is used as the label category of the image block, thereby achieving more accurate hyperspectral image classification.

[0048] That is to say, each element in the constructed source domain hyperspectral image dataset and the target domain hyperspectral image dataset is an image block, and the image block serves as a training sample in the dataset.

[0049] In meta-learning, the image classifier is not trained on the entire dataset, but the learning method is updated through training on multiple meta-tasks. Each meta-task is a small training task, which includes a support set and a query set. The support set is used for model learning, and the query set is used to test the generalization ability of the model.

[0050] For example, a 5-1-15 meta-task means that the meta-task has 5 label categories ( N =5), there is 1 support sample for each label category in the support set ( K =1), there are 15 query samples for each label category in the query set ( ).

[0051] When constructing the meta-task, we randomly sample from the source domain hyperspectral image dataset and the target domain hyperspectral image dataset to construct N Class, each class K Support samples, The source domain meta-task with query samples and N Class, each class K Support samples, The target domain meta-task for each query sample.

[0052] The support set of the source domain meta-task is , the query set of the source domain meta-task is In the support set, It is Support samples, , It is The label category number of the supporting samples, , Support sample The label category, C S is the number of label categories in the source domain hyperspectral image dataset. In the query set, For the js A sample of queries, , It is js Query samples The label category number, y js ∈{1,…, N}, Is a query sample The label category.

[0053] The support set and query set of the target domain meta-task are respectively expressed as and In the support focus It is Support samples, , It is The label category number of the supporting samples, , Support sample The label category, C t is the number of categories in the target domain hyperspectral image dataset. In the query set, For the jt A sample of queries, , y jt It is jt The label category number of the query sample, y jt ∈{1,…, N}, The target domain jt The category labels of the query samples.

[0054] After the source domain meta-task and the target domain meta-task are obtained, the image classifier can be trained. In the present invention, there is no specific requirement for the form of the image classifier, and any image classifier suitable for hyperspectral images in the prior art can be used, or an image classifier formed by combining multiple image classifiers suitable for hyperspectral images in the prior art can be used.

[0055] The present invention is mainly aimed at optimizing the loss function in the training process. Regarding the class distribution loss, the calculation method includes: generating the class distribution of the label category in the source domain meta-task; performing distribution fitting on a number of unlabeled hyperspectral image data in the target domain to obtain the class distribution of the label category in the fitted target domain; wherein the number of label categories in the target domain is equal to the number of label categories in the source domain meta-task; calculating the mean distribution alignment loss based on the class distribution mean of the label category in the source domain meta-task and the class distribution mean of the label category in the target domain; calculating the variance distribution alignment loss based on the class distribution variance of the label category in the source domain meta-task and the class distribution variance of the label category in the target domain; summing the mean distribution alignment loss and the variance distribution alignment loss to obtain the class distribution loss.

[0056] First, the first feature of the training sample in the source domain meta-task (which can be a support sample or a query sample, or a combination of the two) is used to perform Gaussian fitting on each label category to obtain the class distribution of each label category in the source domain meta-task.

[0057] Specifically, in this embodiment, the class distribution selects Gaussian distribution. Taking the query sample in the source domain meta-task as an example, the query sample in the source domain meta-task n The class distribution of the label categories corresponding to the label category numbers is as follows:

[0058] (1)

[0059] in, Indicates the source domain s In the source domain meta-task n The class distribution of label categories corresponding to label category numbers, is a supervised Gaussian distribution function, Represents a query sample The first characteristic, Represents a query sample The label category number is n , Indicates the source domain s No. n The mean of the class distribution, Indicates the source domain s No. n The variance of the class distribution.

[0060] Next, a number of unlabeled hyperspectral image data (such as 500, 1000 or 1200) are selected from the target domain hyperspectral image dataset, and Gaussian fitting is performed on these hyperspectral image data using an unsupervised Gaussian mixture model. The number of fitted label categories is also designed to be N , thus obtaining N The class distribution in the target domain.

[0061] Assumptions Represents several unlabeled hyperspectral image data in the target domain hyperspectral image dataset, For the unlabeled hyperspectral image data, the unsupervised Gaussian mixture model can be defined as:

[0062] (2)

[0063] in, and Respectively represent the target domain t No. n The mean and variance of the corresponding class distribution of the label categories, Indicates n The mixing coefficient of the label categories, represents the first feature of several unlabeled hyperspectral image data in the target domain hyperspectral image dataset. Assign to n The posterior probability of the corresponding class distribution of the label category is:

[0064] (3)

[0065] in, Indicates that Assign to n The posterior probability of the corresponding class distribution of the label categories, N Indicates the number of label categories, Represents unlabeled hyperspectral image data The first characteristic, represents the unsupervised Gaussian distribution function, c ∈{1,…, N}, Indicates c The mixing coefficient of the label categories, and Respectively represent the target domain t No. c The mean and variance of the corresponding class distribution for each label category.

[0066] The mean distribution alignment loss includes:

[0067] (4)

[0068] in, Indicates the source domain s In the source domain meta-task The class distribution mean of the label categories, Indicates the source domain s In the source domain meta-task k The class distribution mean of the label categories, Indicates the target domain t No. j The class distribution mean of the label categories, Indicates the target domain t No. k The class distribution mean of the label categories, N represents the number of label categories. The number of label categories in the target domain is equal to the number of label categories in the source domain meta-task. The corresponding label categories and When the corresponding label category is a positive sample pair, and ,otherwise and ;when The corresponding label categories and When the corresponding label category is a negative sample pair, ,otherwise ;when The corresponding label categories and When the corresponding label category is a negative sample pair, ,otherwise ; Represents a constant.

[0069] Regarding positive sample pairs and negative sample pairs, since the label categories in the source domain are known and the label categories in the target domain are unknown, after Gaussian fitting, the Euclidean distance between the label category of the target domain and each label category in the source domain is calculated. When the Euclidean distance is less than a predetermined threshold, the label category of the target domain is considered to be the same as the corresponding label category in the source domain. In addition, when a certain Euclidean distance is greater than all predetermined thresholds, it is considered to be a new label category. When a label category in the source domain matches a label category in the target domain and the label category of the target domain is the same as other label categories in the source domain, the two are considered to be a negative sample pair, otherwise they are considered to be a positive sample pair.

[0070] The variance distribution alignment loss includes:

[0071] (5)

[0072] in, represents the variance distribution alignment loss, Indicates the source domain s In the source domain meta-task The class distribution variance of the label categories, Indicates the target domain t No. j The class distribution variance of the label categories, express and The Euclidean distance of The corresponding label categories and When the corresponding label category is a positive sample pair, ,otherwise .

[0073] Finally, the mean distribution alignment loss and the variance distribution alignment loss are summed to obtain the class distribution loss:

[0074] (6)

[0075] in, represents the class distribution loss, represents the mean distribution alignment loss, represents the variance distribution alignment loss.

[0076] The above process makes the label category distribution of the source domain and the target domain closer, while the inter-class differences within each domain are enlarged to maintain category discriminability. This method ensures that the alignment process is both effective and semantically meaningful, even when the semantic categories may not completely overlap. The variance distribution alignment loss ensures that the feature distribution of each label category has a small variance, thereby reducing the number of confusing samples near the classification boundary after domain alignment. Therefore, the class distribution loss achieves discriminative domain alignment from the perspective of class distribution.

[0077] Regarding the category loss, it consists of the source domain meta-task category loss and the target domain meta-task category loss.

[0078] Specifically, the method for calculating the source domain meta-task category loss includes: extracting the second feature of each support sample in the source domain meta-task based on the second feature extractor; establishing a source domain mixed dictionary based on the second feature; key matching the first feature of each query sample in the source domain meta-task with each element in the source domain mixed dictionary, and calculating the source domain meta-task category loss based on the matching results.

[0079] More specifically, the first feature extractor is a feature extractor in an image classifier, the second feature extractor has the same structure as the first feature extractor, and the parameter optimization method of the second feature extractor is different from that of the first feature extractor.

[0080] First, the momentum encoder (i.e., the second feature extractor) is used to obtain the support sample features (i.e., the second features) in the source domain meta-task and the target domain meta-task. The momentum encoder has the same structure as the feature extractor (i.e., the first feature extractor) in the image classifier, and is updated according to the parameters of the first feature extractor. The specific update method is:

[0081] (7)

[0082] in, Indicates +1 is the parameter of the second feature extractor in iteration, b is the momentum coefficient, Indicates The parameters of the second feature extractor of the iteration, Indicates Parameters of the first feature extractor of the iteration.

[0083] Next, a source domain mixed dictionary guided by category information is constructed for the source domain using the second feature in the source domain meta-task, and a target domain mixed dictionary guided by category information is constructed for the target domain using the second feature in the target domain meta-task; wherein the second feature is a feature of the support sample extracted by the second feature extractor.

[0084] In this embodiment, the source domain is taken as an example. s The source domain mixed dictionary is represented as ,in, h Indicates the number of rows in the source domain mixed dictionary, C S represents the number of label categories in the source domain hyperspectral image dataset, w Indicates the number of columns of the source domain hybrid dictionary, DL Indicates the total number of columns of the source domain mixed dictionary, Indicates the source domain s The source domain mixed dictionary h Line w The elements of the column, , Is the source domain s The second feature extractor.

[0085] In the source domain hybrid dictionary, the number of rows is the number of label categories in the source domain, and the number of columns is the total number of columns in the dictionary that is predetermined. In each source domain meta-task, the second feature is stored at the end of the corresponding label category queue according to its label category. In the subsequent training process, the dictionary is dynamically maintained, that is, the second feature of the current source domain meta-task is added, and the second feature that enters the earliest is taken out.

[0086] Build the target domain in the same way t The domain mixed dictionary can be expressed as , where the number of rows is the number of label categories in the target domain, and the number of columns is the same as the source domain mixed dictionary. Indicates the target domain t The target domain hybrid dictionary h Line w Elements of a column.

[0087] Then, the source domain meta-task category loss is calculated based on the key matching between the label category of the query sample and each element in the source domain mixed dictionary.

[0088] The source domain meta-task category loss is:

[0089] (8)

[0090] in, represents the source domain meta-task category loss, js represents the ordinal number of the query sample in the source domain meta-task, represents the number of query samples in each label category in the source domain meta-task, h Indicates the number of rows in the source domain mixed dictionary, C S represents the number of label categories in the source domain hyperspectral image dataset, w Indicates the number of columns of the source domain hybrid dictionary, DL Indicates the total number of columns of the source domain hybrid dictionary; Indicates js The label category of the query sample, Indicates the source domain mixed dictionary h Line w Elements of a column The label category, when and At the same time, ,otherwise ; Indicates that the query sample in the source domain meta-task is extracted by the first feature extractor The first characteristic of and different, ,otherwise .

[0091] The calculation method of the target domain meta-task category loss is the same as that of the source domain meta-task category loss, so the target domain meta-task category loss is:

[0092] (9)

[0093] in, represents the target domain meta-task category loss, jt represents the ordinal number of the query sample in the target domain meta-task, Ct Represents the number of label categories in the target domain hyperspectral image dataset; Indicates jt The label category of the query sample, Indicates the first h Line w Elements of a column The label category, when and At the same time, ,otherwise ; Indicates that the query sample in the target domain meta-task is extracted by the first feature extractor The first characteristic of and different, ,otherwise .

[0094] Therefore, the class loss is:

[0095] (10)

[0096] in, represents the class loss.

[0097] It should be noted that this method carefully constructs a domain hybrid dictionary guided by label categories for the source domain and the target domain, making full use of the prior knowledge of label categories and mining the potential semantic connections between label categories. Designing class-level loss can not only significantly improve the compactness of features within each label category, make similar features tightly clustered, and enhance the model's ability to capture unique features of each category, but also enhance the feature similarity of semantically similar label categories between different domains, effectively avoid category confusion, improve label category discriminability, learn more discriminative domain feature representations, and make image classifiers perform better in complex tasks.

[0098] In addition, the loss function of the present invention also includes a small sample learning loss, which is composed of the cross entropy loss of the query sample in the source domain meta-task and the cross entropy loss of the query sample in the target domain.

[0099] Exemplarily, when calculating the cross entropy loss, the first features of the support samples in each meta-task are averaged by label category to obtain the class prototype of each label category.

[0100] For example, in the source domain meta-task, the first feature of the support sample in the support set is averaged to construct the class prototype, specifically:

[0101] (11)

[0102] in, Pns For the n The class prototype of the tag category corresponding to the tag category number, is the first feature extractor, Represents the support samples extracted by the first feature extractor The first feature.

[0103] Then, the Euclidean distance between the query sample and each class prototype in the source domain meta-task is calculated, and the corresponding probability is calculated based on the Euclidean distance, and the label category with the highest probability is selected as the class label. The category distribution of the query sample is:

[0104] (12)

[0105] in, is the Euclidean distance, Represents a query sample Corresponding probability.

[0106] It can be seen that the cross entropy loss of the query sample in the source domain meta-task is:

[0107] (13)

[0108] in, represents the cross entropy loss of the query sample in the source domain meta-task, Represents the calculation of cross entropy loss.

[0109] Similarly, the cross entropy loss of the query sample in the target domain meta-task is:

[0110] (14)

[0111] in, represents the cross entropy loss of the query sample in the target domain meta-task, Represents a query sample Corresponding probability.

[0112] Thus, the small sample learning loss is:

[0113] (15)

[0114] in, represents the few-shot learning loss.

[0115] As described above, the calculation process of each loss is described. In this embodiment, different source domain meta-tasks, target domain meta-tasks, and unlabeled hyperspectral image data in the target domain are used for training until a preset number of training times is reached to obtain a trained image classifier. Regarding the image classifier, in the embodiment of the present invention, a 3D residual convolutional neural network is used as the first feature extractor in the image classifier, and KNN is used as the classifier in the image classifier. In addition, a suitable hyperspectral image classifier can also be selected according to actual conditions.

[0116] Finally, the trained image classifier can be used to classify the unlabeled hyperspectral image data in the target domain. After classification, the classification results are arranged according to the spatial position of the pixels in the original image to generate a complete classification map.

[0117] In summary, the present invention constructs the intra-domain category loss based on the domain mixing dictionary at the category level, which can ensure the feature compactness within each label category in the two domains, as well as the similar feature representation of semantically similar label categories across domains. At the class distribution level, a more representative class distribution is obtained by using supervised Gaussian mixture models and unsupervised Gaussian mixture models in the source domain and the target domain. On this basis, the domain alignment of the source domain and the target domain is promoted and label category conflicts are avoided through the inter-domain class distribution loss.

[0118] In order to verify the effectiveness of the method of the present invention, the following simulation experiments were also carried out.

[0119] Simulation conditions:

[0120] The hardware conditions for the simulation of the present invention are: windows XP, SPI, CPU Pentium(R)4, basic frequency is 2.4GHZ; the software platform is: Pycharm, pytorch.

[0121] The source domain hyperspectral image data used in the simulation is the Chikusei hyperspectral dataset, which contains 20 types of ground objects; the target domain hyperspectral image data used in the simulation are the Indian Pines hyperspectral image dataset, the Salinas hyperspectral image dataset, and the HoustonU 2013 hyperspectral image dataset, among which the Indian Pines hyperspectral image dataset contains 16 types of ground objects, the Salinas hyperspectral image dataset contains 16 types of ground objects, and the HoustonU 2013 hyperspectral image dataset contains 15 types of ground objects.

[0122] Simulation content and results:

[0123] Simulation: The method of the present invention is used to perform classification simulation on the three target domain data sets. For the source domain data, the categories with less than 200 labeled samples are first deleted. Then, 200 labeled samples are selected from each remaining category as the training set. For the target domain, 5 labeled samples are randomly selected from each category, and Gaussian noise is used to increase these samples to 200 to form a training set, and the remaining samples are used as a test set.

[0124] The size of the input spatial image patch is set to 9 × 9 × 100, that is, the spectral dimension of all datasets is uniformly reduced to 100 dimensions.

[0125] The simulation results are as follows Figure 2 and Figure 3 As shown, in Figure 2 In the figure, (a), (b), (c), (d) and (e) are respectively the original image, the classification map of the real object, the classification result map of the Indian Pines hyperspectral dataset using the DFSL cross-domain hyperspectral image classification method, the classification result map of the Indian Pines hyperspectral dataset using the Gia-CFSL cross-domain hyperspectral image classification method and the classification result map of the Indian Pines hyperspectral dataset using the method of the present invention. Figure 3 In the figure, (a), (b), (c), (d) and (e) are respectively the original image, the real object classification map, the classification result map of the Salinas hyperspectral dataset using the DFSL cross-domain hyperspectral image classification method, the classification result map of the Salinas hyperspectral dataset using the Gia-CFSL cross-domain hyperspectral image classification method and the classification result map of the Salinas hyperspectral dataset using the method of the present invention.

[0126] pass Figure 2 and Figure 3 By comparison, it can be seen that the classification graph of the cross-domain hyperspectral image classification method of the present invention is smoother and has fewer misclassified areas.

[0127] In addition, in order to further quantify the performance advantages of the method in the present invention, Tables 1, 2 and 3 list the classification index values ​​of the method in the present invention and other five cross-domain hyperspectral image classification methods on the above three data sets.

[0128] Table 1

[0129]

[0130] Table 1 shows the numerical comparison of classification accuracy of the existing DFSL, DCFSL, STBDIP, Gia-CFSL, ADAFSL methods and the method of the present invention (CDCML) on the Indian Pines dataset. It can be seen from the content of Table 1 that the classification accuracy of the method of the present invention for different categories is mostly the optimal value and suboptimal value.

[0131] Table 2 is a numerical comparison of the classification accuracy of the DFSL, DCFSL, STBDIP, Gia-CFSL, ADAFSL methods in the prior art and the method of the present invention on the Salinas dataset.

[0132] Table 3 shows the numerical comparison of the classification accuracy of the existing technologies DFSL, DCFSL, STBDIP, Gia-CFSL, ADAFSL and the method of the present invention on the Houtson U2013 dataset. The table uses three indicators commonly used in hyperspectral image classification, namely OA, AA and kappa coefficient to evaluate the performance of different methods. The higher the value of these indicators, the better the effect of the method.

[0133] Table 2

[0134]

[0135] The experimental results show that in the three data sets, the proposed method achieved the highest or second highest values ​​in these three indicators. These results show that the proposed method has superior performance in cross-domain hyperspectral image classification tasks and is significantly better than other existing methods.

[0136] From the results of the above simulation experiments, it can be concluded that the method of the present invention performs well in cross-domain hyperspectral image classification, can more effectively improve the generalization ability and classification accuracy of the model, and has high practical value and technical advantages.

[0137] In summary, the present invention adds class distribution loss to the loss function to make the class distribution of the source domain and the target domain closer, and expand the difference between classes in each domain. The difference in image blocks or pixel features of different label categories is made more obvious, and the difference in image blocks or pixel features of the same label category is made closer, thereby achieving the effect of expanding the difference in heterogeneous features and reducing the difference in homogeneous features. By adding category loss to the loss function, the category prior knowledge can be fully utilized to explore the potential semantic connection between categories, thereby increasing the classification accuracy of the image classifier for hyperspectral images.

[0138] Table 3

[0139]

[0140] The present invention also discloses a multi-level contrast small sample learning device for cross-domain hyperspectral image classification, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above method is implemented when the processor executes the computer program.

[0141] The present invention also discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0142] The present invention also provides a computer program product. When the computer program product runs on a data storage device, the data storage device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0143] It is known that if the integrated unit module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to a storage device, a recording medium, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0144] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Claims

1. A multi-level contrastive small sample learning method for cross-domain hyperspectral image classification, characterized by: The following steps are involved: A source domain hyperspectral image dataset and a target domain hyperspectral image dataset are obtained; wherein the number of hyperspectral image data in the source domain hyperspectral image dataset is greater than the number of hyperspectral image data in the target domain hyperspectral image dataset, the hyperspectral image data in the source domain hyperspectral image dataset and the hyperspectral image data in the target domain hyperspectral image dataset are both labeled, and the number of label categories in the source domain hyperspectral image dataset is greater than the number of label categories in the target domain hyperspectral image dataset; Establishing a source domain meta-task based on the source domain hyperspectral image dataset, and establishing a target domain meta-task based on the target domain hyperspectral image dataset, wherein the meta-task parameters in the source domain meta-task and the target domain meta-task are the same; The image classifier is trained using the source domain meta-task, the target domain meta-task and the unlabeled hyperspectral image data in the target domain; wherein the loss function used for training includes category loss and class distribution loss; The category loss is composed of source domain meta-task category loss and target domain meta-task category loss; The method for calculating the source domain meta-task category loss includes: Extracting a second feature of each support sample in the source domain meta-task based on a second feature extractor; Establishing a source domain hybrid dictionary according to the second feature; Key-matching the first feature of each query sample in the source domain meta-task with each element in the source domain mixed dictionary, and calculating the source domain meta-task category loss according to the matching result; Calculating the source domain meta-task category loss according to the matching result includes: ， in, represents the source domain meta-task category loss, js represents the ordinal number of the query sample in the source domain meta-task, represents the number of query samples in each label category in the source domain meta-task, h represents the number of rows in the source domain hybrid dictionary, and C S represents the number of label categories in the source domain hyperspectral image dataset, w represents the number of columns of the source domain hybrid dictionary, and DL represents the total number of columns of the source domain hybrid dictionary; Indicates the label category of the jsth query sample, Represents the element in the hth row and wth column of the source domain hybrid dictionary The label category, when and At the same time, ,otherwise ; Indicates that the query sample in the source domain meta-task is extracted by the first feature extractor The first characteristic of and different, ,otherwise .

2. The multi-level contrast small sample learning method for cross-domain hyperspectral image classification according to claim 1, characterized in that: The class distribution loss calculation method includes: Generate a class distribution of label categories in the source domain meta-task; Performing distribution fitting on a number of unlabeled hyperspectral image data in the target domain to obtain the class distribution of label categories in the target domain; wherein the number of label categories in the target domain is equal to the number of label categories in the source domain meta-task; Calculating a mean distribution alignment loss based on the class distribution mean of the label category in the source domain meta-task and the class distribution mean of the label category in the target domain; Calculating a variance distribution alignment loss based on the class distribution variance of the label category in the source domain meta-task and the class distribution variance of the label category in the target domain; The mean distribution alignment loss and the variance distribution alignment loss are summed to obtain the class distribution loss.

3. The multi-level contrast small sample learning method for cross-domain hyperspectral image classification according to claim 2, characterized in that: The mean distribution alignment loss includes: ， in, represents the mean distribution alignment loss, Indicates the source domain s In the source domain meta-task The class distribution mean of the label categories, Indicates the source domain s The class distribution mean of the k-th label category in the source domain meta-task, Indicates the target domain t The class distribution mean of the j-th label category, Indicates the target domain t The class distribution mean of the kth label category, N represents the number of label categories, and the number of label categories in the target domain is equal to the number of label categories in the source domain meta-task; when The corresponding label categories and When the corresponding label category is a positive sample pair, and ,otherwise and ;when The corresponding label categories and When the corresponding label category is a negative sample pair, ,otherwise ;when The corresponding label categories and When the corresponding label category is a negative sample pair, ,otherwise ; Represents a constant.

4. The multi-level contrast small sample learning method for cross-domain hyperspectral image classification according to claim 3, characterized in that: The variance distribution alignment loss includes: ， in, represents the variance distribution alignment loss, Indicates the source domain s In the source domain meta-task The class distribution variance of the label categories, Indicates the target domain t The class distribution variance of the j-th label category, express and The Euclidean distance of The corresponding label categories and When the corresponding label category is a positive sample pair, ,otherwise .

5. The multi-level contrast small sample learning method for cross-domain hyperspectral image classification according to claim 1, characterized in that: The calculation method of the target domain meta-task category loss is the same as the calculation method of the source domain meta-task category loss.

6. The multi-level contrast small sample learning method for cross-domain hyperspectral image classification according to claim 5, characterized in that: The first feature extractor is a feature extractor in the image classifier, the second feature extractor has the same structure as the first feature extractor, and the parameter optimization method of the second feature extractor is different from that of the first feature extractor.

7. The multi-level contrast small sample learning method for cross-domain hyperspectral image classification according to claim 5 or 6, characterized in that: The loss function also includes small sample learning loss; The few-shot learning loss is composed of the cross-entropy loss of the query sample in the source domain meta-task and the cross-entropy loss of the query sample in the target domain meta-task.

Images