Remote sensing image multi-label classification method, device, equipment, medium and product

Abstract

The application discloses a remote sensing image multi-label classification method, device, equipment, medium and product, and relates to the technical field of computer vision. The method comprises the following steps: processing remote sensing images by using weak transformation and strong transformation respectively to obtain weak transformation images and strong transformation images; inputting the weak transformation images into a first branch to obtain output vectors and hidden features of the weak transformation images; inputting the strong transformation images into a second branch to obtain output vectors and hidden features of the strong transformation images, and obtaining a predicted label vector according to the output vectors; correcting label vectors according to the two output vectors; calculating the value of a loss function according to the corrected label vectors, the predicted label vector and the hidden features, and training a sample model; and performing multi-label classification by using the trained model. The application can improve the precision, generalization ability and robustness of a multi-label remote sensing image classification model, and further improve the accuracy of multi-label classification results.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device, medium and product for multi-label classification of remote sensing images. Background Technology

[0002] With the rapid development of remote sensing technology, AI-based remote sensing image classification systems have been widely applied in fields such as land detection, urban planning, and vegetation cover detection. Compared with conventional images, remote sensing images are rich in various semantic information, with significant differences in the scale and number of objects. Using traditional single-label classification methods often results in the loss of semantic information. Multi-label remote sensing image classification algorithms aim to assign multiple labels to a single remote sensing image. Using multi-labels for remote sensing image classification can fully utilize the semantics of remote sensing images, which is of great significance for further interpretation of remote sensing images.

[0003] In related technologies, research on multi-label remote sensing image classification algorithms can be broadly divided into traditional machine learning and deep learning-based methods. Early multi-label classification algorithms typically extracted image features, such as texture and spectral features, from remote sensing images, and then used classifiers and optimized algorithms for label classification. However, traditional manual features are insensitive to changes in illumination and struggle to capture complex information. In contrast, deep learning, by automatically learning hierarchical feature representations in images, overcomes the shortcomings of traditional methods and possesses stronger feature capture capabilities. However, deep learning-based image classification algorithms are highly sensitive to data, usually requiring large amounts of labeled datasets for training. This data labeling work is influenced by the labeler's subjective biases, making mislabeling and omissions unavoidable, resulting in noisy labels. Training multi-label remote sensing image classification models with noisy labels significantly impacts the model's generalization ability. Therefore, under noise interference, the accuracy and generalization ability of existing deep learning-based multi-label classification models are largely affected by noisy labels, making it difficult to train robust multi-label remote sensing image classification models, leading to inaccurate multi-label classification results. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, medium, and product for multi-label classification of remote sensing images, which can improve the accuracy, generalization ability, and robustness of multi-label remote sensing image classification models, thereby improving the accuracy of multi-label classification results.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a multi-label classification method for remote sensing images, including:

[0007] Obtain a training sample set, which includes multiple training samples, each training sample including a remote sensing image and a label vector of the remote sensing image; the label vector of the remote sensing image is obtained based on a clean label and a noise label representing the category of the remote sensing image;

[0008] For any remote sensing image in the training sample set, weak transformation and strong transformation are applied to the remote sensing image respectively to obtain the weakly transformed image and the strongly transformed image corresponding to the remote sensing image.

[0009] In the current training iteration, the weakly transformed image corresponding to the remote sensing image is input into the first branch of the sample model from the previous training iteration to obtain the output vector and latent features of the classifier in the first branch in the current training iteration; the predicted label vector of the weakly transformed image corresponding to the remote sensing image in the current training iteration is obtained based on the output vector of the classifier in the first branch in the current training iteration; the sample model includes: a first branch and a second branch set in parallel; both the first branch and the second branch include a multi-label remote sensing image classification model and a multilayer perceptron; the multi-label remote sensing image classification model includes a feature encoder and a classifier connected in sequence; the multilayer perceptron in the first branch is connected to the output of the feature encoder in the first branch; the multilayer perceptron in the second branch is connected to the output of the feature encoder in the second branch;

[0010] Input the strongly transformed image corresponding to the remote sensing image into the second branch of the sample model of the previous training iteration to obtain the output vector and latent features of the classifier in the second branch of the current training iteration; and obtain the predicted label vector of the strongly transformed image corresponding to the remote sensing image in the current training iteration based on the output vector of the classifier in the second branch of the current training iteration.

[0011] The label vector of the remote sensing image is corrected based on the output vector of the classifier in the first branch and the output vector of the classifier in the second branch under the current training iterations to obtain the corrected label vector of the remote sensing image under the current training iterations.

[0012] The loss function value for the current training iteration is calculated based on the corrected label vectors of each remote sensing image in the training sample set for the current training iteration, the predicted label vectors and latent features of the weakly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration, and the predicted label vectors and latent features of the strongly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration.

[0013] The sample model for the previous training iteration is trained based on the loss function value for the current training iteration. The process is then determined whether the set number of training iterations has been reached. If not, the process proceeds to the next training iteration until the set number of training iterations is reached, resulting in a well-trained sample model. The multi-label remote sensing image classification model in the first or second branch of the well-trained sample model is then used to perform multi-label classification on the target remote sensing image.

[0014] Secondly, this application provides a remote sensing image multi-label classification device, comprising:

[0015] The acquisition module is used to acquire a training sample set, which includes multiple training samples. Each training sample includes a remote sensing image and a label vector of the remote sensing image. The label vector of the remote sensing image is obtained based on a clean label and a noise label representing the category of the remote sensing image.

[0016] The data augmentation module is used to process the remote sensing image in any training sample in the training sample set using weak transformation and strong transformation respectively, to obtain the weakly transformed image and the strongly transformed image corresponding to the remote sensing image.

[0017] The first branch calculation module is used to input the weakly transformed image corresponding to the remote sensing image into the first branch of the sample model from the previous training iteration to obtain the output vector and latent features of the classifier in the first branch for the current training iteration; and to obtain the predicted label vector of the weakly transformed image corresponding to the remote sensing image for the current training iteration based on the output vector of the classifier in the first branch for the current training iteration; the sample model includes: a first branch and a second branch set in parallel; both the first branch and the second branch include a multi-label remote sensing image classification model and a multilayer perceptron; the multi-label remote sensing image classification model includes a feature encoder and a classifier connected in sequence; the multilayer perceptron in the first branch is connected to the output of the feature encoder in the first branch; the multilayer perceptron in the second branch is connected to the output of the feature encoder in the second branch;

[0018] The second branch calculation module is used to input the strongly transformed image corresponding to the remote sensing image into the second branch of the sample model of the previous training iteration to obtain the output vector and latent features of the classifier in the second branch of the current training iteration; and to obtain the predicted label vector of the strongly transformed image corresponding to the remote sensing image in the current training iteration based on the output vector of the classifier in the second branch of the current training iteration.

[0019] The correction module is used to correct the label vector of the remote sensing image based on the output vector of the classifier in the first branch and the output vector of the classifier in the second branch under the current training number to obtain the corrected label vector of the remote sensing image under the current training number.

[0020] The loss function calculation module is used to calculate the value of the loss function for the current training iteration based on the corrected label vector of each remote sensing image in the training sample set for the current training iteration, the predicted label vector and latent features of the weakly transformed image corresponding to each remote sensing image in the training sample set for the current training iteration, and the predicted label vector and latent features of the strongly transformed image corresponding to each remote sensing image in the training sample set for the current training iteration.

[0021] The training module is used to train the sample model from the previous training iteration based on the loss function value at the current training iteration to obtain the sample model at the current training iteration. It also determines whether the set number of training iterations has been reached. If the set number of training iterations has not been reached, it proceeds to the next training iteration until the set number of training iterations is reached, and a trained sample model is obtained. The multi-label remote sensing image classification model in the first or second branch of the trained sample model is used to perform multi-label classification on the target remote sensing image.

[0022] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the remote sensing image multi-label classification method described in any one of the above.

[0023] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing image multi-label classification method described above.

[0024] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the remote sensing image multi-label classification method described above.

[0025] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0026] This application provides a method, apparatus, device, medium, and product for multi-label classification of remote sensing images. By processing the remote sensing image using weak and strong transforms respectively, weakly transformed and strongly transformed images are obtained. The label vector of the remote sensing image is then corrected based on the output vectors of the classifiers in the first and second branches at the current training iterations to obtain the corrected label vector for the current training iterations. This approach corrects noisy labels as much as possible during training with complex noise labels, thereby improving the performance of multi-label classification of remote sensing images in weakly supervised environments. It solves the problem that existing multi-label remote sensing image classification methods are easily affected by noisy labels and struggle to train robust multi-label remote sensing image classification models. This achieves the technical effect of improving the accuracy, generalization ability, and robustness of multi-label remote sensing image classification models, thereby improving the accuracy of multi-label classification results. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a remote sensing image multi-label classification method in one embodiment of this application;

[0029] Figure 2 This is a schematic diagram of a multi-label classification method for remote sensing images in one embodiment of this application;

[0030] Figure 3 This is a flowchart of sample model data processing for a remote sensing image multi-label classification method according to an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] In one exemplary embodiment, a multi-label classification method for remote sensing images is provided, specifically a robust weakly supervised multi-label classification method for remote sensing images, such as... Figure 2 As shown, firstly, training set images (including multiple remote sensing images) are acquired. Two transformations, strong and weak, are applied to the remote sensing images to obtain two classes of views with significant differences. Depth features of the two views are extracted using two backbone networks with shared weights. Next, classifiers predict class scores (predicted label vectors) based on the two image features. Simultaneously, a multilayer perceptron projects the two image features into the latent space to obtain hidden features. Then, internal consistency constraints are defined based on the hidden features of the two classes of views, and structural consistency constraints are defined based on the hidden features and predicted label vectors of different images. These two constraints are incorporated into the binary cross-entropy loss function. Finally, ranking errors are defined based on the predicted label vectors of the two enhanced images to identify and correct noisy labels, and the network is trained. Figure 1 As shown, the specific steps include:

[0035] Step 201: Obtain a training sample set, which includes multiple training samples. Each training sample includes a remote sensing image and its label vector. To train a robust multi-label remote sensing image classification model, remote sensing images with noisy labels need to be obtained as training samples. Therefore, the label vector of the remote sensing image is obtained based on clean labels and noisy labels representing the remote sensing image category. The noisy labels should include two categories: pseudo-positive labels and pseudo-negative labels. The training sample set is denoted by D, where D = {x} i ,y i x | 1 ≤ i ≤ N}, where N represents the total number of training samples, x i y is the remote sensing image in the i-th training sample. i ={y i1 ,y i2 ,...,y iq}∈{0,1} n×q Let y be the label vector of the remote sensing image in the i-th training sample, where y iq =1 indicates that the q-th label is the true category of the remote sensing image in the i-th training sample, y iq=0 indicates that the q-th label is not the true category of the remote sensing image in the ith training sample. For example, {0, 1, 0, 1, 0, 0} means that the image set has a total of 6 label categories, and the remote sensing image does not have the 1st, 3rd, 5th, and 6th category labels, which can be called the negative label of the sample; while the 2nd and 4th category labels belong to the sample, which can be called the positive label of the sample. Before training the samples, in order to ensure that noisy labels are introduced into the training samples, the clean labels need to be noise-added. The noise-adding process is as follows: select a certain noise ratio, and add additive noise and subtractive noise randomly according to the ratio. Among them, the addition of additive noise is achieved by flipping the 0 in the selected label to 1, while the addition of subtractive noise is achieved by flipping the 1 to 0, ensuring that pseudo-positive labels and pseudo-negative labels are introduced into the training set as introduced noise labels.

[0036] Step 202: For any remote sensing image in the training sample set, perform weak transformation (using simple geometric transformations such as cropping, rotation, and scaling) and strong transformation (performing geometric transformations first, followed by non-geometric transformations such as brightness, contrast, and saturation) on the remote sensing image to obtain the weakly transformed image and the strongly transformed image corresponding to the remote sensing image. Through weak and strong transformations, two images with significant differences are obtained. To ensure the consistency of the training sample image size and meet the input image requirements of the multi-label remote sensing image classification model, weak transformation processing is performed on the training sample images.

[0037] Step 203: In the current training iteration, input the weakly transformed image corresponding to the remote sensing image into the first branch of the sample model from the previous training iteration to obtain the output vector and latent features of the classifier in the first branch in the current training iteration; obtain the predicted label vector of the weakly transformed image corresponding to the remote sensing image in the current training iteration based on the output vector of the classifier in the first branch in the current training iteration. The sample model includes: a first branch and a second branch set in parallel; both the first branch and the second branch include a multi-label remote sensing image classification model and a multilayer perceptron; the multi-label remote sensing image classification model includes a feature encoder and a classifier connected in sequence; the multilayer perceptron in the first branch is connected to the output of the feature encoder in the first branch; the multilayer perceptron in the second branch is connected to the output of the feature encoder in the second branch. The classifier includes a flattening layer and a fully connected layer, wherein the fully connected layer obtains an output vector z, then z is passed through softmax to obtain a posterior probability, and then the predicted label vector is obtained based on the posterior probability and a set threshold.

[0038] Step 204: Input the strongly transformed image corresponding to the remote sensing image into the second branch of the sample model in the previous training iteration to obtain the output vector and latent features of the classifier in the second branch in the current training iteration; obtain the predicted label vector of the strongly transformed image corresponding to the remote sensing image in the current training iteration based on the output vector of the classifier in the second branch in the current training iteration.

[0039] Step 205: Correct the label vector of the remote sensing image based on the output vector of the classifier in the first branch under the current training iteration and the output vector of the classifier in the second branch under the current training iteration to obtain the corrected label vector of the remote sensing image under the current training iteration.

[0040] Step 206: Calculate the value of the loss function for the current training iteration based on the corrected label vectors of each remote sensing image in the training sample set for the current training iteration, the predicted label vectors and latent features of the weakly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration, and the predicted label vectors and latent features of the strongly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration.

[0041] Step 207: Train the sample model from the previous training iteration based on the loss function value at the current training iteration to obtain the sample model for the current training iteration. Determine if the set number of training iterations has been reached. If not, proceed to the next training iteration until the set number of training iterations is reached, resulting in a trained sample model. Use the multi-label remote sensing image classification model from the first or second branch of the trained sample model to perform multi-label classification on the target remote sensing image. Specifically, input the target remote sensing image into the trained multi-label remote sensing image classification model to obtain the output vector. Based on the output vector and the softmax function, obtain the posterior probability of each category of the target remote sensing image. If the posterior probability of a certain category is greater than a set threshold, this category is the label of the target remote sensing image, i.e., a positive label; otherwise, it is determined to be an irrelevant label, i.e., a negative label.

[0042] In another exemplary embodiment of this application, steps 203 and 204 specifically involve:

[0043] For the weakly transformed and strongly transformed images corresponding to remote sensing images, a shared-weight feature encoder (ResNet18 network) can be used as the backbone framework to extract depth features from the input images (weakly transformed and strongly transformed images) to obtain the depth features of the weakly transformed and strongly transformed images corresponding to remote sensing images.

[0044] Then, the depth features of the weakly transformed image corresponding to the remote sensing image are input into the classifier and multilayer perceptron in the first branch for further extraction and transformation, yielding the output vector and latent features of the weakly transformed image. The multilayer perceptron further projects the representation vector into the latent space to obtain the latent features. The classifier makes predictions based on the encoder output to obtain the output vector. Similarly, the depth features of the strongly transformed image corresponding to the remote sensing image are input into the classifier and multilayer perceptron in the second branch to obtain the output vector and latent features of the weakly transformed image corresponding to the remote sensing image. The predicted label vector is obtained based on the output vector.

[0045] This application adopts a collaborative learning mode, which extracts deep features from the image through a feature encoder to obtain the deep features of the image. The high-level features of the network often contain rich semantic information. In order to extract rich semantic information in noisy scenes, a classifier and a multilayer perceptron are used on the basis of the feature encoder to output output vectors and latent features respectively, which are used for subsequent supervised training and similarity constraint training.

[0046] In another exemplary embodiment of this application, step 205 is replaced by the following:

[0047] Construct the positive label set L of the remote sensing image (the remote sensing image in the i-th training sample set). i and negative label set The positive label set includes the true categories of the remote sensing images; the negative label set includes categories to which the remote sensing images do not belong. Both sets may contain pseudo-positive and pseudo-negative labels.

[0048] Based on the output vector of the classifier in the first branch at the current training iterations, we obtain the posterior probabilities of the classifier in the first branch for each positive label in the positive label set and the posterior probabilities of the classifier in the first branch for each negative label in the negative label set. The posterior probability of the classifier in the c-th branch for the i-th label is the posterior probability obtained from softmax mentioned above, according to the formula... Calculate, where z i (c) and z j (c) Let i and j represent the i-th and j-th elements in the output vector z of the classifier in the c-th branch, respectively.

[0049] For any positive label in the set of positive labels, the probability that the positive label is a pseudo-positive label from the perspective of the classifier in the first branch is obtained based on the posterior probability of the positive label by the classifier in the first branch and the posterior probability of each negative label in the set of negative labels by the classifier in the first branch. Specifically: based on Calculate the probability that the positive label l is a false positive label from the perspective of the classifier in the c-th branch, where c takes the value of 1 or 2. and These represent the classifier in the c-th branch for the positive label l and the negative label l, respectively. The posterior probability.

[0050] For any negative label in the set of negative labels, the probability that the negative label is a pseudo-negative label from the perspective of the classifier in the first branch is obtained based on the posterior probability of the negative label by the classifier in the first branch and the posterior probability of each positive label in the set of positive labels by the classifier in the first branch. Specifically, this is done according to the formula... Calculate the negative label from the perspective of the classifier in the c-th branch. The probability of being a pseudo-negative label.

[0051] Based on the output vector of the classifier in the second branch at the current training iterations, the posterior probabilities of the classifier in the second branch for each positive label in the positive label set and the posterior probabilities of the classifier in the second branch for each negative label in the negative label set are obtained.

[0052] and The values ​​of l and l were measured from the perspective of the classifier in the c-th branch, respectively. The probability of a positive label being a pseudo-positive or pseudo-negative label is determined by the classifier in the second branch. Therefore, for any positive label in the set of positive labels, the probability that the positive label is a pseudo-positive label from the perspective of the classifier in the second branch is obtained based on the posterior probability of the positive label from the classifier in the second branch and the posterior probability of each negative label from the set of negative labels.

[0053] For any negative label in the set of negative labels, the probability that the negative label is a pseudo-negative label is obtained from the perspective of the classifier in the second branch, based on the posterior probability of the negative label by the classifier in the second branch and the posterior probability of each positive label in the set of positive labels by the classifier in the second branch.

[0054] The probability of each positive label in the positive label set being a pseudo-positive label is determined from the perspective of the classifier in the first branch, and the positive label corresponding to the highest probability among the probabilities of each positive label in the positive label set being a pseudo-positive label is determined from the perspective of the classifier in the second branch. Specifically, this is done according to the formula... Determine the maximum positive label l * .

[0055] The probability of each negative label in the negative label set being a pseudo-negative label is determined from the perspective of the classifier in the first branch, and the negative label corresponding to the highest probability among the probabilities of each negative label in the negative label set being a pseudo-negative label is determined from the perspective of the classifier in the second branch. Specifically, this is done according to the formula... Determine the maximum negative label

[0056] The values ​​corresponding to the largest positive label and the largest negative label in the label vector of the remote sensing image are corrected (by flipping them from 0 to 1 or 1 to 0) to obtain the corrected label vector of the remote sensing image for the current training iterations. When using a supervised loss function for network training, noisy labels as the expected output can mislead the model; therefore, it is necessary to correct the expected output.

[0057] In another exemplary embodiment of this application, step 206 is replaced by the following:

[0058] Based on the implicit features of the weakly transformed images corresponding to each remote sensing image in the training sample set under the current training iterations and the implicit features of the strongly transformed images corresponding to each remote sensing image in the training sample set under the current training iterations, calculate the internal consistency function value under the current training iterations.

[0059] The structural consistency function value for the current training iteration is calculated based on the predicted label vectors and latent features of the weakly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration, as well as the predicted label vectors and latent features of the strongly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration.

[0060] The supervised loss function value for the current training iteration is calculated based on the corrected label vectors of each remote sensing image in the training sample set for the current training iteration, the predicted label vectors of the weakly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration, and the predicted label vectors of the strongly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration.

[0061] Calculate the value of the loss function for the current training iteration based on the internal consistency function value, the structural consistency function value, and the supervised loss function value for the current training iteration.

[0062] In another exemplary embodiment of this application, since weakly transformed and strongly transformed images of the same remote sensing image originate from the same remote sensing image, they should obtain similar features after passing through different branch networks. Internal similarity constraints are used on the latent features of the two transformations, and an internal consistency function is defined based on these two latent features. Internal similarity represents that the same data from different transformations has the same mapping, and its similarity measure, i.e., the internal consistency function, is defined as:

[0063]

[0064] and The consistency measure between the two is defined as: express and Pairwise similarity.

[0065] Therefore, based on the latent features of the weakly transformed images corresponding to each remote sensing image in the training sample set under the current training iterations and the latent features of the strongly transformed images corresponding to each remote sensing image in the training sample set under the current training iterations, the internal consistency function value under the current training iterations is calculated as follows:

[0066] According to the formula Calculate the internal consistency function value, where C int This represents the internal consistency function value, where N represents the total number of remote sensing images in the training sample set. This represents the latent features of the weakly transformed image corresponding to the remote sensing image in the i-th training sample. This represents the latent feature of the strongly transformed image corresponding to the remote sensing image in the i-th training sample. Let represent the latent features of the strongly transformed image corresponding to the remote sensing image in the j-th training sample, τ represent the temperature parameter, and exp() represent an exponential function with the natural constant e as the base. express The transpose of .

[0067] In another exemplary embodiment of this application, the classifier makes predictions based on the output of the feature encoder. Since the outputs of the classifier and the multilayer perceptron come from the same enhanced image (weakly transformed image or strongly transformed image), the prediction result of the classifier should match the key structural information in the latent features, that is, there is a point-to-point structural similarity between the classifier output and the latent features. A structural consistency constraint is defined based on the latent features and the predicted label vector. This constraint enhances the representational ability of the features to a certain extent, improves the accuracy of the predicted label vector, and thus reduces the interference of noisy labels. Specifically, the structural consistency constraint is the KL divergence constraint between the classifier output and the multilayer perceptron output: in, Let $c$ represent the structural consistency loss between the classifier and the perceptron in the $c$-th branch (c = 1, 2). express and The Euclidean distance between them express and The Euclidean distance between them Parameters μ and σ represent the mean and variance of sample x, respectively, and apply point-to-point structural similarity constraints between the classifier output and latent features in the two branches of the network. The structural similarity measure, i.e., the structural consistency function, is defined as follows:

[0068] Therefore, the structure consistency function value for the current training iteration is calculated based on the predicted label vectors and latent features of the weakly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration, as well as the predicted label vectors and latent features of the strongly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration. Specifically:

[0069] According to the formula Calculate the structural consistency function value, where C str This represents the structural consistency function value. express and The Euclidean distance between them, where μ represents the mean and σ represents the variance. express and The Euclidean distance between them express and The Euclidean distance between them express and The Euclidean distance between them This represents the latent features of the weakly transformed image corresponding to the remote sensing image in the i-th training sample. This represents the latent features of the weakly transformed image corresponding to the remote sensing image in the j-th training sample. This represents the latent feature of the strongly transformed image corresponding to the remote sensing image in the i-th training sample. This represents the latent feature of the strongly transformed image corresponding to the remote sensing image in the j-th training sample. Let represent the predicted label vector of the weakly transformed image corresponding to the remote sensing image in the i-th training sample. Let represent the predicted label vector of the weakly transformed image corresponding to the remote sensing image in the j-th training sample. Let represent the predicted label vector of the strongly transformed image corresponding to the remote sensing image in the i-th training sample. This represents the predicted label vector of the strongly transformed image corresponding to the remote sensing image in the j-th training sample.

[0070] In another exemplary embodiment of this application, the supervised loss L commonly used in network training is described. sup That is, BCELoss performs supervised learning on the outputs of the two classifiers respectively, defined as: Let represent the loss function based on the classifier output in the c-th branch (c = 1, 2). The final supervised loss function is defined as: Therefore, the supervised loss function value for the current training iteration is calculated based on the corrected label vectors of each remote sensing image in the training sample set, the predicted label vectors of the weakly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration, and the predicted label vectors of the strongly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration. Specifically:

[0071] Among them, L sup This represents the supervised loss function value, where N represents the total number of remote sensing images in the training sample set, and y' i This represents the corrected label vector of the remote sensing image in the i-th training sample set. Indicates y' i transpose, Let (1-y') represent the predicted label vector of the weakly transformed image corresponding to the remote sensing image in the i-th training sample. i ) T It represents (1-y') i The transpose of ) This represents the predicted label vector of the strongly transformed image corresponding to the remote sensing image in the i-th training sample.

[0072] In another exemplary embodiment of this application, two consistency constraints are introduced into the objective function to guide the network to learn more discriminative features and reduce the negative impact of noisy labels in the BCELoss of the objective function. That is, supervised training of the network is performed by combining supervised binary cross-entropy loss with internal consistency loss and structural consistency loss. The loss function formula is: L = L sup +C str +C int .

[0073] In practical applications, the training process incorporates a loss function and uses the Adam optimizer algorithm, setting the learning rate to 0.001 and minimizing the loss function to update the model parameters until the model converges.

[0074] The following is combined Figure 3 The data processing procedure of the sample model provided in this application is illustrated with an example:

[0075] like Figure 3 As shown, the training image (remote sensing image) x was obtained. i The same training image undergoes both strong and weak transform processing. Specifically, the training image is uniformly cropped, flipped, rotated, and normalized to adjust the input image size to 3×256×256, where 3 represents the three channels of the sample. The strong transform operation adjusts the brightness, contrast, and saturation of the image, with adjustment probabilities of 0.8 and adjustment intensities of {0.4, 0.4, 0.4}, respectively. After preprocessing, the training set sample images are randomly shuffled and input into the multi-label remote sensing image classification model. Deep feature extraction is performed through a feature encoder, using convolutional blocks of a ResNet18 network as the backbone framework. This module has an input dimension of 3×224×224 and an output dimension of 512×1×1. Further feature processing is performed using a classifier and a multilayer perceptron. The classifier consists of a flattening layer and a fully connected layer, while the multilayer perceptron is a three-layer perceptron that nonlinearizes deep features, mainly consisting of a linear layer, a ReLU activation layer, and another linear layer. This application can maintain high classification accuracy even when the noise ratio is 50%, and the model runs efficiently, which can meet the requirements of speed and accuracy in real-world applications.

[0076] This application combines two data augmentation methods of different intensities (weak and strong transforms) to generate diverse training data, enabling the network to learn different image features of the same image. A co-learning model is employed in feature extraction, utilizing two similarity constraints for training the two augmented images. Finally, a ranking loss is introduced to select and correct noisy labels. The final loss function uses the commonly used multi-label binary cross-entropy loss for supervision, weighted with the two similarity constraints to guide the model in robust learning. It not only provides end-to-end sample modeling and uses a deep feature network to extract deep features, but also corrects noise labels as much as possible during training with complex noise labels, thereby improving the performance of multi-label classification of remote sensing images in weakly supervised environments. In practical applications, it exhibits excellent classification accuracy and generalization ability.

[0077] Based on the same inventive concept, this application also provides a remote sensing image multi-label classification device for implementing the aforementioned remote sensing image multi-label classification method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the remote sensing image multi-label classification device provided below can be found in the limitations of the remote sensing image multi-label classification method described above, and will not be repeated here.

[0078] In one exemplary embodiment, a remote sensing image multi-label classification device is provided, comprising:

[0079] The acquisition module is used to acquire a training sample set, which includes multiple training samples. Each training sample includes a remote sensing image and a label vector of the remote sensing image. The label vector of the remote sensing image is obtained based on clean labels and noise labels representing the category of the remote sensing image.

[0080] The data augmentation module is used to process the remote sensing image in any training sample in the training sample set using weak transformation and strong transformation respectively, to obtain the weakly transformed image and the strongly transformed image corresponding to the remote sensing image.

[0081] The first branch calculation module is used to, under the current training iteration, input the weakly transformed image corresponding to the remote sensing image into the first branch of the sample model under the previous training iteration to obtain the output vector of the classifier in the first branch under the current training iteration and the latent features of the weakly transformed image corresponding to the remote sensing image under the current training iteration; and obtain the predicted label vector of the weakly transformed image corresponding to the remote sensing image under the current training iteration based on the output vector of the classifier in the first branch under the current training iteration. The sample model includes: a first branch and a second branch set in parallel; both the first branch and the second branch include a multi-label remote sensing image classification model and a multilayer perceptron; the multi-label remote sensing image classification model includes a feature encoder and a classifier connected in sequence; the multilayer perceptron in the first branch is connected to the output end of the feature encoder in the first branch; the multilayer perceptron in the second branch is connected to the output end of the feature encoder in the second branch.

[0082] The second branch calculation module is used to input the strongly transformed image corresponding to the remote sensing image into the second branch of the sample model under the previous training number to obtain the output vector and latent features of the classifier in the second branch under the current training number; and to obtain the predicted label vector of the strongly transformed image corresponding to the remote sensing image under the current training number based on the output vector of the classifier in the second branch under the current training number.

[0083] The correction module is used to correct the label vector of the remote sensing image based on the output vector of the classifier in the first branch and the output vector of the classifier in the second branch under the current training iterations to obtain the corrected label vector of the remote sensing image under the current training iterations.

[0084] The loss function calculation module is used to calculate the value of the loss function for the current training iteration based on the corrected label vectors of each remote sensing image in the training sample set for the current training iteration, the predicted label vectors and latent features of the weakly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration, and the predicted label vectors and latent features of the strongly transformed images corresponding to each remote sensing image in the training sample set for the current training iteration.

[0085] The training module is used to train the sample model from the previous training iteration based on the loss function value at the current training iteration to obtain the sample model at the current training iteration. It also determines whether the set number of training iterations has been reached. If the set number of training iterations has not been reached, it proceeds to the next training iteration until the set number of training iterations is reached, and a trained sample model is obtained. The multi-label remote sensing image classification model in the first or second branch of the trained sample model is used to perform multi-label classification on the target remote sensing image.

[0086] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores multi-label classification data for remote sensing images. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-label classification method for remote sensing images.

[0087] Those skilled in the art will understand that Figure 4 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0088] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0089] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0091] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0092] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for multi-label classification of remote sensing images, characterized in that, The remote sensing image multi-label classification method comprises the following steps: Obtain a training sample set, wherein the training sample set comprises a plurality of training samples, each training sample comprises a remote sensing image and a label vector of the remote sensing image, and the label vector of the remote sensing image is obtained according to clean labels and noise labels representing a category of the remote sensing image; For the remote sensing image in any one of the training samples, the remote sensing image is processed by using a weak transformation and a strong transformation respectively to obtain a weak transformation image and a strong transformation image corresponding to the remote sensing image; Under the current training number, the weak transformation image corresponding to the remote sensing image is input into the first branch of the sample model under the last training number to obtain an output vector of a classifier in the first branch under the current training number and a hidden feature of the weak transformation image corresponding to the remote sensing image under the current training number; a predicted label vector of the weak transformation image corresponding to the remote sensing image under the current training number is obtained according to the output vector of the classifier in the first branch under the current training number; the sample model comprises a first branch and a second branch arranged in parallel; the first branch and the second branch each comprise a multi-label remote sensing image classification model and a multi-layer perceptron; the multi-label remote sensing image classification model comprises a feature encoder and a classifier connected in sequence; the multi-layer perceptron in the first branch is connected to the output end of the feature encoder in the first branch; the multi-layer perceptron in the second branch is connected to the output end of the feature encoder in the second branch; The strong transformation image corresponding to the remote sensing image is input into the second branch of the sample model under the last training number to obtain an output vector of a classifier in the second branch under the current training number and a hidden feature; a predicted label vector of the strong transformation image corresponding to the remote sensing image under the current training number is obtained according to the output vector of the classifier in the second branch under the current training number; The label vector of the remote sensing image is corrected according to the output vector of the classifier in the first branch under the current training number and the output vector of the classifier in the second branch under the current training number to obtain a corrected label vector of the remote sensing image under the current training number; The value of a loss function under the current training number is calculated according to the corrected label vector of each remote sensing image in the training sample set under the current training number, the predicted label vector and the hidden feature of the weak transformation image corresponding to each remote sensing image in the training sample set under the current training number, and the predicted label vector and the hidden feature of the strong transformation image corresponding to each remote sensing image in the training sample set under the current training number; The sample model under the last training number is trained according to the value of the loss function under the current training number to obtain the sample model under the current training number, and it is determined whether a set training number is reached; if the set training number is not reached, the next training number is entered, until the set training number is reached, a trained sample model is obtained, and the multi-label remote sensing image classification model in the first branch or the second branch of the trained sample model is used for multi-label classification of a target remote sensing image.

2. The method of claim 1, wherein, The label vector of the remote sensing image is corrected according to the output vector of the classifier in the first branch at the current training number and the output vector of the classifier in the second branch at the current training number, and the corrected label vector of the remote sensing image at the current training number is obtained, and the method specifically comprises the following steps: A positive label set and a negative label set of the remote sensing image are constructed; the positive label set comprises a real category of the remote sensing image; and the negative label set comprises a category which the remote sensing image does not belong to; The posterior probability of each positive label in the positive label set and the posterior probability of each negative label in the negative label set are obtained based on the output vector of the classifier in the first branch at the current training number; For any one positive label in the positive label set, the probability that the positive label is a pseudo-positive label is obtained from the posterior probability of the positive label and the posterior probability of each negative label in the negative label set based on the classifier in the first branch; For any one negative label in the negative label set, the probability that the negative label is a pseudo-negative label is obtained from the posterior probability of the negative label and the posterior probability of each positive label in the positive label set based on the classifier in the first branch; The posterior probability of each positive label in the positive label set and the posterior probability of each negative label in the negative label set are obtained based on the output vector of the classifier in the second branch at the current training number; For any one positive label in the positive label set, the probability that the positive label is a pseudo-positive label is obtained from the posterior probability of the positive label and the posterior probability of each negative label in the negative label set based on the classifier in the second branch; For any one negative label in the negative label set, the probability that the negative label is a pseudo-negative label is obtained from the posterior probability of the negative label and the posterior probability of each positive label in the positive label set based on the classifier in the second branch; The maximum positive label is determined from the maximum probability of each positive label being a pseudo-positive label in the first branch and the maximum probability of each positive label being a pseudo-positive label in the second branch; The maximum negative label is determined from the maximum probability of each negative label being a pseudo-negative label in the first branch and the maximum probability of each negative label being a pseudo-negative label in the second branch; The value corresponding to the maximum positive label and the value corresponding to the maximum negative label in the label vector of the remote sensing image are corrected to obtain the corrected label vector of the remote sensing image at the current training number. 3.The method of claim 1, wherein, According to the corrected label vector of each remote sensing image in the training sample set under the current training number, the predicted label vector and the hidden feature of the weak transformed image corresponding to each remote sensing image in the training sample set under the current training number, and the predicted label vector of the strong transformed image corresponding to each remote sensing image in the training sample set under the current training number, the value of the loss function under the current training number is calculated, specifically comprising: According to the hidden feature of the weak transformed image corresponding to each remote sensing image in the training sample set under the current training number and the hidden feature of the strong transformed image corresponding to each remote sensing image in the training sample set under the current training number, the value of the internal consistency function under the current training number is calculated; According to the predicted label vector and the hidden feature of the weak transformed image corresponding to each remote sensing image in the training sample set under the current training number and the predicted label vector and the hidden feature of the strong transformed image corresponding to each remote sensing image in the training sample set under the current training number, the value of the structural consistency function under the current training number is calculated; According to the corrected label vector of each remote sensing image in the training sample set under the current training number, the predicted label vector of the weak transformed image corresponding to each remote sensing image in the training sample set under the current training number, and the predicted label vector of the strong transformed image corresponding to each remote sensing image in the training sample set under the current training number, the value of the supervised loss function under the current training number is calculated; According to the value of the internal consistency function under the current training number, the value of the structural consistency function under the current training number, and the value of the supervised loss function under the current training number, the value of the loss function under the current training number is calculated.

4. The method of claim 3, wherein, According to the hidden feature of the weak transformed image corresponding to each remote sensing image in the training sample set under the current training number and the hidden feature of the strong transformed image corresponding to each remote sensing image in the training sample set under the current training number, the value of the internal consistency function under the current training number is calculated, specifically comprising: According to the formula The internal consistency function value is calculated, wherein C int represents the internal consistency function value, N represents the total number of training samples in the training sample set, represents the implicit feature of the weak transformed image corresponding to the remote sensing image in the i-th training sample, represents the implicit feature of the strong transformed image corresponding to the remote sensing image in the i-th training sample, represents the implicit feature of the strong transformed image corresponding to the remote sensing image in the j-th training sample, τ represents the temperature parameter, exp() represents the exponential function with the natural constant e as the base, represents the transpose of 5. The method of claim 3, wherein, According to the predicted label vector and the hidden feature of the weak transformed image corresponding to each remote sensing image in the training sample set under the current training number and the predicted label vector and the hidden feature of the strong transformed image corresponding to each remote sensing image in the training sample set under the current training number, the value of the structural consistency function under the current training number is calculated, specifically comprising: According to the formula The structure consistency function value is calculated, wherein C str represents the structure consistency function value, represents the Euclidean distance between , μ represents the mean of the sample x, and σ represents the standard deviation of the sample x, represents the Euclidean distance between , represents the Euclidean distance between , represents the Euclidean distance between , represents the hidden feature of the weak transformation image corresponding to the remote sensing image in the i-th training sample, represents the hidden feature of the weak transformation image corresponding to the remote sensing image in the j-th training sample, represents the hidden feature of the strong transformation image corresponding to the remote sensing image in the i-th training sample, represents the hidden feature of the strong transformation image corresponding to the remote sensing image in the j-th training sample, represents the predicted label vector of the weak transformation image corresponding to the remote sensing image in the i-th training sample, represents the predicted label vector of the weak transformation image corresponding to the remote sensing image in the j-th training sample, represents the predicted label vector of the strong transformation image corresponding to the remote sensing image in the i-th training sample, represents the predicted label vector of the strong transformation image corresponding to the remote sensing image in the j-th training sample.

6. The method of claim 3, wherein, According to the corrected label vector of each remote sensing image in the training sample set under the current training number, the predicted label vector of the weak transformed image corresponding to each remote sensing image in the training sample set under the current training number, and the predicted label vector of the strong transformed image corresponding to each remote sensing image in the training sample set under the current training number, the value of the supervised loss function under the current training number is calculated; wherein L sup denotes the supervised loss function value, N denotes the total number of remote sensing images in the training sample set, y' i denotes the corrected label vector of the remote sensing image in the i-th training sample, y' i T denotes the transpose of y' i denotes the predicted label vector of the weakly transformed image corresponding to the remote sensing image in the i-th training sample, (1-y' i ) T denotes the transpose of (1-y' i ) denotes the predicted label vector of the strongly transformed image corresponding to the remote sensing image in the i-th training sample.​ 7. A remote sensing image multi-label classification apparatus, characterized by, The remote sensing image multi-label classification device comprises: An acquisition module is configured to acquire a training sample set, wherein the training sample set comprises a plurality of training samples, each training sample comprises a remote sensing image and a label vector of the remote sensing image; and the label vector of the remote sensing image is obtained according to clean labels and noise labels representing a category of the remote sensing image. The data augmentation module is configured to perform weak transformation and strong transformation on the remote sensing image in any one of the training sample set to obtain a weak transformation image and a strong transformation image corresponding to the remote sensing image. The first branch calculation module is configured to input the weak transformation image corresponding to the remote sensing image into the first branch of the sample model in the previous training to obtain an output vector and hidden features of a classifier in the first branch in the current training; and obtain a predicted label vector of the weak transformation image corresponding to the remote sensing image in the current training according to the output vector of the classifier in the first branch in the current training. The sample model comprises a first branch and a second branch arranged in parallel, and each of the first branch and the second branch comprises a multi-label remote sensing image classification model and a multi-layer perceptron. The multi-label remote sensing image classification model comprises a feature encoder and a classifier connected in sequence. The multi-layer perceptron in the first branch is connected to an output end of the feature encoder in the first branch. The multi-layer perceptron in the second branch is connected to an output end of the feature encoder in the second branch. The second branch calculation module is configured to input the strong transformation image corresponding to the remote sensing image into the second branch of the sample model in the previous training to obtain an output vector and hidden features of a classifier in the second branch in the current training; and obtain a predicted label vector of the strong transformation image corresponding to the remote sensing image in the current training according to the output vector of the classifier in the second branch in the current training. The correction module is configured to correct the label vector of the remote sensing image according to the output vector of the classifier in the first branch in the current training and the output vector of the classifier in the second branch in the current training to obtain a corrected label vector of the remote sensing image in the current training. The loss function calculation module is configured to calculate a value of the loss function in the current training according to the corrected label vector of each remote sensing image in the training sample set in the current training, the predicted label vector and hidden features of the weak transformation image corresponding to each remote sensing image in the training sample set in the current training, and the predicted label vector and hidden features of the strong transformation image corresponding to each remote sensing image in the training sample set in the current training. The training module is configured to train the sample model in the previous training according to the value of the loss function in the current training to obtain the sample model in the current training, determine whether a set training number is reached, and if not, enter the next training number until the set training number is reached to obtain the trained sample model. The multi-label remote sensing image classification model in the first branch or the second branch of the trained sample model is used to perform multi-label classification on a target remote sensing image.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-label remote sensing image classification method in any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the remote sensing image multi-label classification method in any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the remote sensing image multi-label classification method in any one of claims 1-6.

Images