A remote sensing scene open set classification method based on a label-guided auto-encoding network of multi-task learning

Abstract

The application discloses a remote sensing scene open set classification method based on a label guided auto-encoding network of multi-task learning, and comprises the following steps: (1) pre-training a convolutional neural network on a dataset ImageNet; (2) extracting features in a remote sensing dataset; (3) inputting the extracted features into a label guided auto-encoder to perform a feature reconstruction task; (4) constructing a loss function, wherein the loss function comprises two parts, i.e., a classification task loss and a reconstruction task loss; the network parameters are optimized by using a back propagation algorithm, so that the loss function reaches a minimum; (5) inputting image samples of a test dataset into the network to generate reconstruction features corresponding to the test set images, and comparing and calculating reconstruction errors of the features; and a threshold is set to divide known and unknown classes according to the size of the reconstruction errors. Compared with the prior art, the remote sensing scene open set classification method based on the label guided auto-encoding network of multi-task learning can effectively inhibit the interference of a complex background, and can be applied to environmental monitoring, city planning and the like.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing, and specifically relates to a label-guided autoencoder network that can be used for open set classification of remote sensing scenes. Background Technology

[0002] With the development of satellite imaging technology, remote sensing images are readily available, containing rich spatial details and neighborhood relationships of ground features. Remote sensing scene classification aims to accurately assign semantic labels to each remote sensing image, playing a crucial role in urban planning and environmental monitoring.

[0003] Traditional remote sensing scene classification methods typically use color histograms and scale-invariant transforms to extract handcrafted features, which are then encoded or quantized using intermediate methods such as bag-of-words visual representation and probabilistic topic models. These methods rely on shallow features and have relatively weak representational capabilities. Convolutional Neural Networks (CNNs) construct a hierarchical structure to extract discriminative and semantic features. CNN-based networks have been widely used and have achieved satisfactory results in remote sensing scene classification.

[0004] Most remote sensing scene classification methods based on convolutional neural networks operate under the assumption of a "static closed set," meaning all test samples belong to a known class and are labeled data. However, due to the vastness of ground scenes and incomplete data labeling, remote sensing images often contain many unlabeled, unknown scenes, making the "static closed set" assumption difficult to guarantee in remote sensing. Existing remote sensing scene classification methods are prone to forcibly misclassifying samples of unknown classes as known classes. Therefore, correctly classifying test samples into unknown categories becomes crucial. Detecting known and unknown scenes in a given remote sensing image and correctly classifying known scenes is defined as open set classification of remote sensing scenes.

[0005] Open set classification of remote sensing scenes has gained increasing attention in recent years. The key to this method lies in improving the detection capability of unknown samples while maintaining the accuracy of known class classification. One proposed open set domain adaptive algorithm reduces global distribution differences within a domain and local distribution differences of the same class in different domains for remote sensing scene open set classification. However, this method, based on image reconstruction, is susceptible to interference from complex backgrounds, weakening its performance in distinguishing between known and unknown classes.

[0006] In summary, existing open set classification methods for remote sensing scenes based on deep convolutional neural networks are unable to effectively remove interference from complex backgrounds, resulting in low accuracy in the classification of open sets of remote sensing scenes. Summary of the Invention

[0007] To address the shortcomings of existing methods, this invention proposes a remote sensing scene open set classification method using a label-guided autoencoder network with multi-task learning. This method ensures the classification accuracy of known classes while effectively detecting unknown classes, thereby improving the accuracy of remote sensing scene open set classification.

[0008] The present invention adopts the following technical solution:

[0009] A remote sensing scene open set classification method based on label-guided autoencoder networks with multi-task learning includes the following steps:

[0010] (1) Extract features from the training set images and re-represent the image features. Pre-train the convolutional neural network on the ImageNet dataset. Use the pre-trained convolutional neural network to extract global and local features of the image. Remove the last classifier of ResNet50 and use the remaining part as the feature extractor. The output of the remaining part is used as the extracted features of the image.

[0011] (2) Construct a label-guided autoencoder network to perform convolutional encoding on image features. Input the output of step (1) into the encoder for encoding without changing the number of local features in each image;

[0012] (3) Perform label-guided decoding on the image features. Input the output of step (2) into the label-guided layer for solving, and simultaneously input the output of step (2) into the decoder for decoding. Do not change the number of local features in each image;

[0013] (4) Constructing the loss function. This method includes two tasks: classification and reconstruction. The overall loss function includes two parts: classification loss and reconstruction loss. The classification task uses the cross-entropy loss function, and the feature reconstruction task uses the mean squared error function.

[0014] (5) Model training. The Adam optimizer algorithm is used to iteratively train the parameters by minimizing the loss function defined in step (4) until the network converges.

[0015] (6) Model testing. The image features of the test dataset are input into the trained label-guided autoencoder network to obtain the scene category.

[0016] Furthermore, step (1) involves extracting multi-layer convolutional features from the training set images, including the following steps:

[0017] A ResNet50 convolutional neural network was constructed using convolution, pooling, and fully connected operations. The ResNet50 was pre-trained on the ImageNet dataset. Then, remote sensing scene images were input into the pre-trained ResNet50 to extract multi-layer convolutional features. The features were extracted using the convolution operation of the last stage of ResNet50, denoted as x′∈R.M×N×C Where M and N are the width and height of convolutional feature x, and C is the number of channels of convolutional feature x. ResNet50 is pre-trained on ImageNet. Passing an image through the ResNet50 network yields the output feature map of the last convolutional layer, Conv5.x, of the ResNet50 model.

[0018] x′=Conv(I)

[0019] Where Conv represents the feature extraction process, I is the input image, and x′ represents the extracted features.

[0020] Furthermore, the convolutional encoding of the image representation in step (2) includes:

[0021] The image representation in step (1) is convolutionally encoded, and x is input into encoder E to extract the latent vector. The calculation formula is as follows:

[0022] z I =[z I1 ,z I2 ,…,z Im ]=E[x′]=E[x′1,x′2,…,x′ m ]

[0023] Among them, Z I Z represents the set of tensors obtained from the encoding. Im Let x' represent the feature obtained by encoding the m-th input, where m is the number of input images; i Let z represent the feature extracted from the i-th training sample. E represents the encoder, which mainly consists of convolutional layers, pooling layers, and Batch Normalization (BN) layers. The convolutional layers are responsible for acquiring local features of the image, the pooling layers downsample the image and pass the scale-invariant features to the next layer, and the BN layers mainly normalize the distribution of the training image to accelerate learning. The encoder is used to encode the input feature sequence into a low-bit vector representation, and the output result is denoted as z.

[0024] Furthermore, the characteristic feature is that the label-guided decoding of image features in step (3) includes:

[0025] (3a) In order to better learn the reconstruction error distributions of the known and unknown classes during training, the latent vector {z} in step (2) is extracted using encoder E. I1 ,z I2 ,…,z ImThe code performs label decoding training by applying feature linear modulation based on label information to adjust the latent vector. The scaled-down feature vector is then upsampled, followed by convolution to refine the object's geometry and compensate for the detail loss caused by the encoder's pooling layer scaling the object. When conditioned on a label vector matching the input class identifier, the decoder can reconstruct the original input features almost consistently. For input features z containing label information and vector l... j The specific calculation formula is as follows:

[0026] γ j =H γ (l j ), β j =H β (l j )

[0027]

[0028] in,

[0029]

[0030] H γ and H β It has parameter Θ γ and Θ β The neural network calculates the tensor. γ j ,β j Having the same shape, l j Used for conditionalization, here referred to as the label conditional vector, ⊙ represents the Hadamard product, Z I The tensor Z, calculated when matching class identifiers and label vectors, represents the tensor obtained from this process. Im This represents the tensor calculated when the class identifier and label vector do not match.

[0031] (3b) Perform feature reconstruction for a given input; when matching the input class identifier and label vector, denoted as l m Otherwise, denoted as l nm Image information encoding after attention mechanism for a given input x and and The specific calculation formula for tag decoding training is as follows:

[0032]

[0033]

[0034]

[0035]

[0036] The formula explanation is the same as step (3a). Since the label includes both matching and non-matching cases, the superscript of the parameter m indicates that the label is successfully matched, and the superscript of the parameter nm indicates that the label is not matched. and These represent tag information for successful and unsuccessful matches, respectively. and For neural network H γ Match success and failure outputs. and For neural network H β The outputs of successful and unsuccessful matching are then input into feature z. i Linear combinations yield the following results and Finally, the data is fed into the decoder to obtain the reconstructed features of matching and non-matching features. and D stands for Decoder.

[0037] Furthermore, the construction of the loss function in step (4) includes:

[0038] (4a) The encoder E parameter is Θ E The classifier C has the parameter Θ. C Using cross-entropy loss, the loss function formula is as follows:

[0039]

[0040] Where y(i) represents the true label (one-hot vector) corresponding to the i-th sample, and the predicted probability ability score vector. It is the probability that the i-th sample comes from the j-th category.

[0041] (4b) The tag decoding layer includes parameter Θ D The decoder D and its parameters are Θ γ and Θ β The label guidance layer loss function is shown below:

[0042]

[0043]

[0044] Among them, z i This represents the input features of the i-th sample in the label decoding layer. This represents the reconstructed feature indicating a successful match between the label vectors of the i-th sample. The loss function represents the reconstructed feature where the label vector of the i-th sample fails to match. The corresponding loss function for reconstruction using matching label vectors. The corresponding loss for reconstruction using mismatched label vectors, where N represents the total number of samples and K represents the number of known classes;

[0045] (4c) This method includes two tasks: classification and reconstruction. Therefore, the overall loss should also include two parts: classification loss and reconstruction loss. The formula for the overall loss can be written as:

[0046]

[0047] λ1 and λ2 represent the weighting coefficients.

[0048] Furthermore, model training in step (5) includes:

[0049] Using the Adam optimizer algorithm, the parameters Θ are trained iteratively by minimizing the loss function defined in step (4). E ,Θ C ,Θ D ,Θ γ ,Θ β Until the network converges.

[0050] Furthermore, the predicted remote sensing scene category in step (6) includes:

[0051] The remote sensing scene to be tested is input into the trained label autoencoder network to obtain the reconstruction error. A threshold is selected based on the distribution of reconstruction errors of known and unknown classes. The reconstruction error obtained from each test sample is compared with the threshold. If the reconstruction error is greater than the threshold, the sample is considered to be of unknown class; otherwise, the sample is considered to be of known class. The scene category output by the classifier is taken as the category of that class.

[0052] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it performs the steps of the remote sensing scene open set classification method based on a label-guided autoencoder network for multi-task learning described above.

[0053] The present invention also proposes a computer-readable storage medium storing a computer program that causes a computer to perform the steps of the remote sensing scene open set classification method based on a label-guided autoencoder network for multi-task learning disclosed above.

[0054] This invention utilizes a convolutional neural network (CNN) to extract image features of known classes, and then reconstructs the image based on the feature distribution of the known classes at the feature level, making the reconstructed features more discriminative. Through model training, the reconstruction error of known classes can be continuously reduced, and the discriminability between known and unknown categories can be increased, thereby ensuring the accuracy of open set classification of remote sensing scenes. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a remote sensing scene open set classification method based on a label-guided autoencoder network using multi-task learning, as described in this invention. Detailed Implementation

[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the following description is provided in conjunction with the accompanying drawings. Figure 1 The present invention will now be described in further detail. (Refer to...) Figure 1 This invention discloses a remote sensing scene open set classification method based on a label-guided autoencoder network using multi-task learning. The implementation steps are as follows:

[0057] Step (1) involves extracting multi-layer convolutional features from the training set images, including the following steps:

[0058] (1a) First, a ResNet50 convolutional neural network is constructed using convolution, pooling, and fully connected operations. Specifically, the ResNet50 is pre-trained using the ImageNet dataset, and then remote sensing scene images are input into the pre-trained ResNet50 to extract multi-layer convolutional features. Features are extracted using the convolution operation of the last stage of ResNet50, denoted as x′∈R. M×N×C Where M and N are the width and height of the convolutional feature x, and C is the number of channels of the convolutional feature x. That is, ResNet50 is pre-trained on ImageNet. Passing an image through the ResNet50 network yields the output feature map of the last convolutional layer, Conv5.x, of the ResNet50 model, i.e.:

[0059] x′=Conv(I)

[0060] Where Conv represents the feature extraction process, I is the input image, and x′ represents the extracted features.

[0061] Step (2) involves convolutional coding of the image representation.

[0062] (2a) Perform convolutional encoding on the image representation in (1a). Input x into encoder E to extract the latent vector, calculated as follows:

[0063] z I =[z I1 ,z I2 ,…,z Im ]=E[x′]=E[x′1,x′2,…,x′ m ]

[0064] Among them, Z I Z represents the set of tensors obtained from the encoding. ImLet x' represent the feature obtained by encoding the m-th input, where m is the number of input images; i Let z represent the features extracted from the i-th training sample. E represents the encoder, which mainly consists of convolutional layers, pooling layers, and Batch Normalization (BN) layers. The convolutional layers are responsible for acquiring local image features, the pooling layers downsample the image and pass scale-invariant features to the next layer, and BN mainly normalizes the distribution of the training images, accelerating learning. The encoder is used to encode the input feature sequence into a low-bit vector representation. The output is denoted as z.

[0065] Step (3) of label-guided decoding of image features includes:

[0066] (3a) In order to better learn the reconstruction error distributions of known and unknown classes during training, the latent vector {z} is extracted using encoder E in (2a) above. I1 ,z I2 ,…,z Im Label decoding training is performed by applying feature linear modulation based on label information to adjust the latent vector. Upsampling the scaled-down feature vector followed by convolution refines the object's geometry and compensates for the loss of detail caused by the encoder's pooling layer scaling the object. When conditioned on a label vector matching the input class identifier, the decoder can reconstruct the original input features nearly consistently. For input features z containing label information and vector l... j The specific calculation formula is as follows:

[0067] γ j =H γ (l j ), β j =H β (l j )

[0068]

[0069] in,

[0070]

[0071] H γ and H β It has parameter Θ γ and Θ β The neural network. The calculated tensor γ j ,β j Having the same shape, l j Used for conditionalization, here referred to as the label conditional vector, ⊙ represents the Hadamard product.

[0072] (3b) Perform feature reconstruction for a given input. The matching of the input class identifier and label vector is denoted as l. m Otherwise, denoted as l nm Image information encoding after attention mechanism for a given input x and and The specific calculation formula for tag decoding training is as follows:

[0073]

[0074]

[0075]

[0076]

[0077] The formula explanation is the same as (3a). Since the label includes both matching and non-matching cases, the superscript of the parameter m indicates that the label is successfully matched, and the superscript of the parameter nm indicates that the label is not matched. and These represent tag information for successful and unsuccessful matches, respectively. and For neural network H γ Match success and failure outputs. and For neural network H β The outputs of successful and unsuccessful matching are then input into feature z. i Linear combinations yield the following results and Finally, the data is fed into the decoder to obtain the reconstructed features of matching and non-matching features. and D stands for Decoder.

[0078] Step (4): Construct the loss function.

[0079] (4a) The encoder E parameter is Θ E The classifier C has the parameter Θ. C Using cross-entropy loss, the loss function formula is as follows:

[0080]

[0081] Where y(i) represents the true label (one-hot vector) corresponding to the i-th sample, and the predicted probability ability score vector. It is the probability that the i-th sample comes from the j-th category.

[0082] (4b) The tag decoding layer includes decoder D (parameter Θ). D ) and tag guide layer (parameter is Θ)γ and Θ β The loss function is shown below:

[0083]

[0084]

[0085] Among them, z i This represents the input features of the i-th sample in the label decoding layer. This represents the reconstructed feature indicating a successful match between the label vectors of the i-th sample. This represents the reconstructed feature indicating that the label vector of the i-th sample failed to match. Loss function. The corresponding loss function for reconstruction using matching label vectors. The corresponding loss is used for reconstruction using mismatched label vectors.

[0086] (4c) Since this method includes two tasks, classification and reconstruction, the overall loss should also include classification loss and reconstruction loss. The formula for the overall loss can be written as:

[0087]

[0088] λ1 and λ2 represent the weighting coefficients.

[0089] Step (5), model training, to train the network.

[0090] Using the Adam optimizer algorithm, the parameters Θ are trained iteratively by minimizing the loss function defined in step 4. E ,Θ C ,Θ D ,Θ γ ,Θ β Until the network converges.

[0091] Step (6) Predict the remote sensing scene category.

[0092] The remote sensing scene being tested is input into the trained label autoencoder network to obtain the reconstruction error. A threshold is selected based on the distribution of reconstruction errors for known and unknown classes. The reconstruction error obtained from each test sample is compared with this threshold. If the reconstruction error is greater than the threshold, the sample is considered to be of the unknown class; otherwise, the sample is considered to be of the known class, and the scene category output by the classifier is taken as the class of that unknown class.

[0093] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it performs the steps of the remote sensing scene open set classification method based on a label-guided autoencoder network for multi-task learning described above.

[0094] The present invention also proposes a computer-readable storage medium storing a computer program that causes a computer to perform the steps of the remote sensing scene open set classification method based on a label-guided autoencoder network for multi-task learning disclosed above.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit this application. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing scene open set classification method based on label-guided autoencoder networks with multi-task learning, characterized in that, Includes the following steps: (1) Extract the features of the training set images and re-express the image features; pre-train the convolutional neural network on the ImageNet dataset, and use the pre-trained convolutional neural network to extract the global and local features of the image. The last classifier of ResNet50 is removed, and the remaining part is used as the feature extractor. The output of the remaining part is used as the features extracted from the image. Step (1) involves extracting multi-layer convolutional features from the training set images, including the following steps: Constructing a ResNet50 convolutional neural network using convolution, pooling, and fully connected operations; pre-training ResNet50 using the ImageNet dataset; inputting remote sensing scene images into the pre-trained ResNet50 to extract multi-layer convolutional features; and using the convolution operation of the last stage of ResNet50 to extract features, denoted as... Where M and N are the width and height of convolutional feature x, and C is the number of channels of convolutional feature x. ResNet50 is pre-trained on ImageNet. Passing an image through the ResNet50 network yields the output feature map of the last convolutional layer, Conv5.x, of the ResNet50 model. Where Conv represents the feature extraction process, and I is the input image. The extracted features are represented by [the symbol]. (2) Construct a label-guided autoencoder network to perform convolutional encoding on image features; Step (2) involves convolutional encoding of the image representation, including: The image representation in step (1) is convolutionally encoded, and x is input into encoder E to extract the latent vector. The calculation formula is as follows: Among them, Z I Z represents the set of tensors obtained from the encoding. Im This represents the feature obtained by encoding the m-th input, where m is the number of input images; Let z represent the feature extracted from the i-th training sample. Let E represent the encoder, which consists of convolutional layers, pooling layers, and Batch Normalization (BN) layers. The convolutional layers are responsible for acquiring local features of the image, the pooling layers downsample the image and pass the scale-invariant features to the next layer, and the BN layers normalize the distribution of the training image to accelerate learning. The encoder is used to encode the input feature sequence into a low-bit vector representation, and the output is denoted as z. (3) Label-guided decoding of image features; (4) Constructing the loss function: It includes two tasks: classification and reconstruction. The overall loss function includes two parts: classification loss and reconstruction loss. The cross-entropy loss function is used for classification tasks, and the mean squared error function is used for feature reconstruction tasks. (5) Model training; (6) Model testing: Input the image features of the test dataset into the trained label-guided autoencoder network to obtain the scene category.

2. The remote sensing scene open set classification method based on label-guided autoencoder network with multi-task learning as described in claim 1, characterized in that, Step (3) of label-guided decoding of image features includes: (3a) In order to better learn the reconstruction error distribution of known and unknown classes during training, the latent vectors in step (2) are extracted using encoder E. Label decoding training is performed by applying feature linear modulation based on label information to adjust the latent vector; the scaled-down feature vector is upsampled, and then convolutional processing is applied to the upsampled features to refine the object's geometry and compensate for the loss of detail caused by the encoder's pooling layer scaling the object; when conditioned on a label vector matching the input class identifier, the decoder can reconstruct the original input features almost consistently for input features z containing label information and vectors. The specific calculation formula is as follows: ， in, and It has parameters and The neural network calculates the tensor. , , Having the same shape Used for conditionalization, referred to here as the label condition vector. Represents the Hadamard product; (3b) Perform feature reconstruction for a given input; when matching the input class identifier and label vector, it is denoted as... Otherwise, record as Image information encoding after attention mechanism for a given input x and and The specific calculation formula for label decoding training is as follows: ， ， ， , The formula explanation is the same as step (3a). Since the label includes both matching and non-matching cases, the superscript of the parameter m indicates that the label is successfully matched, and the superscript of the parameter nm indicates that the label is not matched. and These represent tag information for successful and unsuccessful matches, respectively. and For neural networks Match success and failure outputs. and For neural networks The outputs of successful and unsuccessful matching are then input into the features respectively. Linear combinations yield the following results and Finally, the data is fed into the decoder to obtain the reconstructed features of matching and non-matching features. and D stands for Decoder.

3. The remote sensing scene open set classification method based on label-guided autoencoder network with multi-task learning as described in claim 1, characterized in that, Step (4) involves constructing the loss function, which includes: (4a) The encoder E parameter is The parameters of classifier C are Using cross-entropy loss, the loss function formula is as follows: Where y(i) represents the true label (one-hot vector) corresponding to the i-th sample, and the predicted probability ability score vector. , It is the probability that the i-th sample comes from the j-th class; (4b) The tag decoding layer includes parameters as follows The decoder D and its parameters are and The label guidance layer loss function is shown below: in, This represents the input features of the i-th sample in the label decoding layer. This represents the reconstructed feature indicating a successful match between the label vectors of the i-th sample. The loss function represents the reconstructed feature where the label vector of the i-th sample fails to match. The corresponding loss function for reconstruction using matching label vectors. The corresponding loss for reconstruction using mismatched label vectors, where N represents the total number of samples and k represents the number of known classes; (4c) This method includes two tasks: classification and reconstruction. Therefore, the overall loss should also include two parts: classification loss and reconstruction loss. The formula for the overall loss can be written as: λ1 and λ2 represent the weighting coefficients.

4. The remote sensing scene open set classification method based on label-guided autoencoder network with multi-task learning as described in claim 1, characterized in that, The model training in step (5) includes: Using the Adam optimizer algorithm, the training parameters are iteratively trained by minimizing the loss function defined in step (4). Until the network converges.

5. The remote sensing scene open set classification method based on label-guided autoencoder network with multi-task learning as described in claim 1, characterized in that, The predicted remote sensing scene categories in step (6) include: The remote sensing scene to be tested is input into the trained label autoencoder network to obtain the reconstruction error. A threshold is selected based on the distribution of reconstruction errors of known and unknown classes. The reconstruction error obtained from each test sample is compared with the threshold. If the reconstruction error is greater than the threshold, the sample is considered to be of unknown class; otherwise, the sample is considered to be of known class. The scene category output by the classifier is taken as the category of that class.

6. An electronic device, characterized in that: The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor runs the computer program, it performs the steps of a remote sensing scene open set classification method based on a label-guided autoencoder network for multi-task learning as described in any one of claims 1-5.

7. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to perform the steps of the remote sensing scene open set classification method based on a label-guided autoencoder network for multi-task learning disclosed in any one of claims 1-5.

Images