Remote sensing image scene classification method based on attribute guidance

Abstract

The invention provides a remote sensing image scene classification method based on attribute guidance, and relates to the technical field of remote sensing image processing, and the method comprises the following steps: obtaining a to-be-classified remote sensing image and a corresponding scene class text; constructing and training a scene classification network; processing the to-be-classified remote sensing image and the scene category text by using the trained scene classification network to obtain a scene classification result corresponding to the to-be-classified remote sensing image; according to the method, the attribute features of the remote sensing image are utilized, so that the language information can contain each part of the image with differentiated content, and the consistency between the language information and the image content is improved.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and more particularly to a remote sensing image scene classification method based on attribute guidance. Background Art

[0002] At present, remote sensing image scene classification technology aims to accurately classify the scenes presented in the image content into specific scene categories such as forests, rivers, grasslands, and farmlands. These scene category names are high-level, general descriptions of the image content. This technology is used to understand surface characteristics, land use, vegetation cover, and urban development, and can provide information support for the decision-making process and promote the optimal allocation and utilization of resources.

[0003] However, in the process of language supervision, due to the monotony limitation of language information, the language representation cannot fully summarize the image features, which affects the classification results. Specifically, the existing language information representation is limited to the name of the scene category, which is not enough to fully integrate with the complex content of the remote sensing image. To address this problem, the present invention proposes a remote sensing image scene classification method based on attribute guidance, which generates attribute nouns by extracting the attribute features of the image, and then uses the attribute nouns to enrich the language information representation. This method integrates the distinguishable attribute features of the image into the language representation, thereby improving the language supervision effect and thus improving the accuracy of scene classification.

[0004] Therefore, how to provide an attribute-guided remote sensing image scene classification method that can solve the above problems is an issue that those skilled in the art urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a remote sensing image scene classification method based on attribute guidance, which can utilize the attribute features of remote sensing images so that the language information can contain content with distinguishing meaning in each part of the image, thereby improving the consistency between the language information and the image content.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A remote sensing image scene classification method based on attribute guidance includes the following steps:

[0008] Obtain the remote sensing image to be classified and the corresponding scene category text;

[0009] Build and train a scene classification network;

[0010] The trained scene classification network is used to process the remote sensing image to be classified and the scene category text to obtain a scene classification result corresponding to the remote sensing image to be classified.

[0011] Preferably, the specific process of processing the remote sensing image to be classified and the scene category text using the trained scene classification network includes:

[0012] The scene classification network includes: an attribute noun generation module, a text encoder, an image encoder, and a language information enrichment module;

[0013] Using the text encoder and the image encoder to perform feature extraction and encoding processing on the remote sensing image to be classified and the scene category text to obtain corresponding image feature vectors and text feature vectors;

[0014] The image feature vector and the text feature vector are processed by using the attribute noun generation module and the language information enrichment module to obtain the scene category of the remote sensing image to be classified.

[0015] Preferably, the specific process of processing the image feature vector and the text feature vector by using the attribute noun generation module and the language information enrichment module includes:

[0016] Get the preset mask text;

[0017] The attribute noun generation module includes an attribute generator, and the preset mask text and the image feature vector are embedded in the attribute generator for prediction to obtain the corresponding predicted attribute noun;

[0018] The preset mask text is replaced by the predicted attribute noun, and the predicted attribute noun and the text feature vector are fused by the language information enrichment module to obtain the scene category of the remote sensing image to be classified.

[0019] Preferably, the specific process of fusing the predicted attribute noun and the text feature vector using the language information enrichment module includes:

[0020] Extracting features of the predicted attribute nouns through the language information enrichment module to obtain corresponding attribute feature vectors;

[0021] The attribute feature vector and the text feature vector are fused through a linear layer to obtain the scene category of the remote sensing image to be classified.

[0022] Preferably, the specific process of fusing the attribute feature vector and the text feature vector through a linear layer to obtain the scene category of the remote sensing image to be classified includes:

[0023] Interactively fusing the attribute feature vector and the text feature vector through a linear layer to obtain a corresponding fused vector;

[0024] Calculating the similarity between the fusion vector and the image feature vector, and processing the similarity by setting a classifier to obtain a plurality of probability values ​​of the remote sensing image to be classified belonging to each type of scene;

[0025] The highest value among the multiple probability values ​​is selected to complete the prediction of the remote sensing image to be classified.

[0026] Preferably, the specific processing of training the scene classification network includes:

[0027] Obtaining an image-text training set and determining a loss function of the scene classification network during the training process;

[0028] The fusion vector is optimized by contrastive learning method, so that the scene classification network calculates the similarity between each sample and the vector through CLIP forward propagation to define a loss function, and a gradient descent algorithm is used to iteratively update the parameters of the scene classification network to minimize the loss function;

[0029] The training of the scene classification network is completed after multiple training iterations.

[0030] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a remote sensing image scene classification method based on attribute guidance, which has the following beneficial effects:

[0031] 1. The present invention adopts a remote sensing image scene classification method based on attribute guidance, which significantly increases the information that can be summarized by text language by combining scene category text with corresponding attribute text, enriches language information, and enables it to more comprehensively connect visual feature representation.

[0032] 2. The present invention uses remote sensing images and their corresponding attribute nouns to pre-train the ViLT model to adapt it to the remote sensing field, and generates accurate attribute nouns through input remote sensing images, enriching the language information while ensuring that the content information it contains is concise and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0034] Figure 1 An overall flow chart of a remote sensing image scene classification method based on attribute guidance provided by the present invention;

[0035] Figure 2A structural principle block diagram of a scene classification network architecture provided by an embodiment of the present invention;

[0036] Figure 3 A structural principle block diagram of an attribute noun generation module provided in an embodiment of the present invention;

[0037] Figure 4 This is a structural principle block diagram of a language information enrichment module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 As shown, the embodiment of the present invention discloses a remote sensing image scene classification method based on attribute guidance, comprising the following steps:

[0040] Get the remote sensing image to be classified X input And the corresponding scene category text T input ;

[0041] Build and train a scene classification network;

[0042] Use the trained scene classification network to classify the remote sensing image X input And the scene category text T input Processing is performed to obtain the remote sensing image X to be classified input The corresponding scene classification results.

[0043] See also Figure 2 As shown, in a specific embodiment, the trained scene classification network is used to classify the remote sensing image X input And the scene category text T input The specific process of processing includes:

[0044] The scene classification network includes: attribute noun generation module E g , Text Encoder E t 、Image encoder E i And the language information enrichment module L r ;

[0045] Using the text encoder E t And the image encoder E i Treat the classified remote sensing image X input And the scene category text T inputPerform feature extraction and encoding processing to obtain the corresponding image feature vector I and text feature vector T. At the same time, the image feature vector I and the text feature vector T can be mapped to the same space;

[0046] Using attribute noun generation module E g And the language information enrichment module L r The image feature vector I and the text feature vector T are processed to obtain the remote sensing image X to be classified input The scene category.

[0047] See also Figure 3 As shown, in a specific embodiment, the attribute noun generation module E is used g And the language information enrichment module L r The specific process of processing the image feature vector I and the text feature vector T includes:

[0048] Get the preset mask text T mo , where the preset mask text T mo for text with masked words;

[0049] Attribute noun generation module E g Includes attribute generator E a , set the preset mask text T mo And the image feature vector I are embedded into the attribute generator E a Make predictions in and get the corresponding predicted attribute nouns;

[0050] Use the predicted attribute noun to replace the preset mask text T mo , and use language information to enrich module L r The predicted attribute noun and text feature vector are fused to obtain the remote sensing image X to be classified input The scene category.

[0051] Specifically, the attribute noun generation module E g Including text embedding layer, image embedding layer, attribute generator E a And the processing module, where the image embedding layer can be implemented by a linear layer, the attribute generator E a It includes a decoder, a linear layer, a linear and activation function layer, and the processing module includes a normalization layer, a multi-head attention mechanism layer, a normalization layer, and a multi-layer perceptron.

[0052] Attribute noun generation module E gThe masked language modeling (MLM) task model of the Vision-and-Language Transformer model (ViLT) pre-trained on remote sensing images is used as the main body. By inputting images and masked texts, the words or symbols corresponding to the content information are output.

[0053] The text feature processing process can regard the text feature vector T as a word sequence, which is converted into a text vector through the text embedding matrix. The image feature processing process can regard the image feature block as a sequence of image blocks, which is converted into an image vector through a linear layer. The two are spliced ​​together and the model ViLT is used to generate attribute nouns with the feature information of the text and image context.

[0054] Attribute nouns with image-distinguishing attribute features can significantly improve the scene category text T with their diverse and highly recognizable visual features. input The language content summarization efficiency is improved, thereby making the language information in the text richer, more refined and accurate, thereby improving the language supervision effect and improving the classification performance.

[0055] See also Figure 4 As shown, in a specific embodiment, the language information enrichment module L r The specific process of fusing the predicted attribute noun and the text feature vector T includes:

[0056] By enriching the module L with language information r Extract features of the predicted attribute nouns to obtain the corresponding attribute feature vectors;

[0057] The attribute feature vector and the text feature vector are fused through a linear layer to obtain the remote sensing image to be classified X input The scene category.

[0058] In a specific embodiment, the attribute feature vector and the text feature vector are fused through a linear layer to obtain the remote sensing image to be classified X input The specific process of the scene category includes:

[0059] The attribute feature vector and the text feature vector are interactively fused through a linear layer to obtain the corresponding fused vector;

[0060] Calculate the similarity between the fusion vector and the image feature vector, and set the classifier to process the similarity to obtain the remote sensing image X to be classified. input Multiple probability values ​​P1, P2, P3, ..., P attributed to each type of scene c ;

[0061] Select the highest value among multiple probability values ​​to complete the remote sensing image X to be classified input predict.

[0062] Specifically, the language information enrichment module L r It includes a text encoder and a linear layer, and the text encoder includes a text embedding layer, a processing module one and a processing module two, wherein the processing module one includes a multi-head attention mechanism layer, a residual and normalization layer, a feedforward neural network and a residual and normalization layer, and the processing module two includes a masked multi-head attention mechanism layer, a residual and normalization layer, a multi-head attention mechanism layer, a residual and normalization layer, a feedforward neural network and a residual and normalization layer (the structure here is consistent with the aforementioned text encoder).

[0063] Language Information Enrichment Module L r It can integrate the attribute features corresponding to each scene category with the text language features. The text encoder can decode the attribute text and scene category text T input Extract and encode features. The specific expression is:

[0064] T a =E t (T atrri_input )

[0065] T=E t (T input )

[0066] T'=Linear(T a ,T)

[0067] Where, T a Represents the attribute text T generated by the text encoder atrri_input The result of the processing, E t represents the text encoder, T represents the text encoder for the scene category text T input The result of processing, T' represents the attribute text T processed by the linear layer atrri_input The processing results and scene category text T input The processing result.

[0068] In a specific embodiment, the specific processing of training the scene classification network includes:

[0069] Obtain the image-text training set and determine the loss function of the scene classification network during the training process;

[0070] The contrastive learning method is used to optimize the fusion vector, so that the scene classification network calculates the similarity between each sample and the vector through CLIP forward propagation to define a loss function, and a gradient descent algorithm is used to iteratively update the parameters of the scene classification network to minimize the loss function;

[0071] The training of the scene classification network is completed after multiple training iterations.

[0072] Specifically, in the process of training the scene classification network using the image text training set, the image text pair loss uses the cross entropy as the loss function, and the specific expression is:

[0073] L=(L CE (P I ,y)+L CE (P T ,y)) / 2

[0074] Where, L CE (P I ,y) is the cross entropy loss from image to text, P I is the probability of predicting the image as the first text; L CE (P T ,y) is the cross entropy loss from text to image, P T is the probability of predicting the text as the image, and y is the true value.

[0075] Attribute text and scene category text T input The contrastive learning loss is used for the loss, and the specific expression is:

[0076]

[0077] Where, T i is the i-th scene category text T input Indicates that T ai For its corresponding attribute text representation, is the text representation of other attributes, C represents the number of categories, and τ is a hyperparameter.

[0078] During the training process, the model uses CLIP to shorten the distance between the feature vectors of text and image. On this basis, the model further uses the contrastive learning method to let the text language feature vector learn the attribute features related to it. For the text vector that integrates the attribute feature information, the similarity between each sample and the vector is calculated through CLIP forward propagation to define the loss function, and the gradient descent algorithm is used to iteratively update the parameters of the model to minimize the loss function. After multiple training iterations, the model effectively enriches the text language information and better connects it with the visual features. The detailed code of the overall training process is shown in Table 1.

[0079] Table 1 Attribute-guided remote sensing image scene classification network training algorithm code

[0080]

[0081] 6.updatetheclipmodel / / Gradient descent algorithm to iteratively update the model parameters

[0082] 4.predict=(100.0*image_feat@text_feat.T).softmax(dim=-1) / / Calculate the text and image features

[0083] The remaining amount

[0084] / / Predict scene categories using chord similarity

[0085] 5.returnpredict / / Return the prediction result

[0086] 6.model = clip.load (args.vis_model, jit = False) / / Initialize the model

[0087] 7.output = model (text_input, image_input) / / predict category output

[0088] 8.return(outputpredict) / / Scene classification results

[0089] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0090] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote sensing image scene classification method based on attribute guidance, characterized in that: The following steps are involved: Obtain the remote sensing image to be classified and the corresponding scene category text; Build and train a scene classification network; The trained scene classification network is used to process the remote sensing image to be classified and the scene category text to obtain a scene classification result corresponding to the remote sensing image to be classified.

2. The remote sensing image scene classification method based on attribute guidance according to claim 1 is characterized in that: The specific process of processing the remote sensing image to be classified and the scene category text using the trained scene classification network includes: The scene classification network includes: an attribute noun generation module, a text encoder, an image encoder, and a language information enrichment module; Using the text encoder and the image encoder to perform feature extraction and encoding processing on the remote sensing image to be classified and the scene category text to obtain corresponding image feature vectors and text feature vectors; The image feature vector and the text feature vector are processed by using the attribute noun generation module and the language information enrichment module to obtain the scene category of the remote sensing image to be classified.

3. The remote sensing image scene classification method based on attribute guidance according to claim 2 is characterized in that: The specific process of processing the image feature vector and the text feature vector by using the attribute noun generation module and the language information enrichment module includes: Get the preset mask text; The attribute noun generation module includes an attribute generator, and the preset mask text and the image feature vector are embedded in the attribute generator for prediction to obtain the corresponding predicted attribute noun; The preset mask text is replaced by the predicted attribute noun, and the predicted attribute noun and the text feature vector are fused by the language information enrichment module to obtain the scene category of the remote sensing image to be classified.

4. The remote sensing image scene classification method based on attribute guidance according to claim 3 is characterized in that: The specific process of fusing the predicted attribute noun and the text feature vector using the language information enrichment module includes: Extracting features of the predicted attribute nouns through the language information enrichment module to obtain corresponding attribute feature vectors; The attribute feature vector and the text feature vector are fused through a linear layer to obtain the scene category of the remote sensing image to be classified.

5. The remote sensing image scene classification method based on attribute guidance according to claim 4 is characterized in that: The specific process of fusing the attribute feature vector and the text feature vector through a linear layer to obtain the scene category of the remote sensing image to be classified includes: Interactively fusing the attribute feature vector and the text feature vector through a linear layer to obtain a corresponding fused vector; Calculating the similarity between the fusion vector and the image feature vector, and processing the similarity by setting a classifier to obtain a plurality of probability values ​​of the remote sensing image to be classified belonging to each type of scene; The highest value among the multiple probability values ​​is selected to complete the prediction of the remote sensing image to be classified.

6. The remote sensing image scene classification method based on attribute guidance according to claim 3 is characterized in that: The specific process of training the scene classification network includes: Obtaining an image-text training set and determining a loss function of the scene classification network during the training process; The fusion vector is optimized by contrastive learning method, so that the scene classification network calculates the similarity between each sample and the vector through CLIP forward propagation to define a loss function, and a gradient descent algorithm is used to iteratively update the parameters of the scene classification network to minimize the loss function; The training of the scene classification network is completed after multiple training iterations.

Images