Small sample remote sensing scene image classification method and device, equipment and medium

Abstract

The application relates to a small sample remote sensing scene image classification method, device, equipment and medium, the method comprising the following steps: calling a pre-trained remote sensing scene image classification model, an image feature extractor in the remote sensing scene image classification model extracts image features in a remote sensing scene image; an image feature mapping network in a BERT model in the remote sensing scene image classification model performs global average pooling on the image features, adopts two fully connected layers to map the image features to a 512-dimensional feature embedding space to determine image mapping features; a semantic feature mapping network in the BERT model extracts semantic features of various category names in the remote sensing scene image, adopts two fully connected layers to map the semantic features to a 512-dimensional feature embedding space to determine semantic mapping features; the image mapping features and the semantic mapping features are subjected to two norm normalization to determine categories corresponding to various scenes in the remote sensing scene image. The application can effectively improve the performance of a small sample learning classification and recognition task.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing, and in particular to a method for classifying small-sample remote sensing scene images, a corresponding device, electronic equipment, and a computer-readable storage medium. Background Technology

[0002] With the improvement of remote sensing technology in terms of observation range and image resolution, the rich data on terrain, vegetation, buildings, and other land features contained in remote sensing images can provide an important data source for applications such as environmental monitoring, land resource management, and urban planning. Remote sensing scene classification is an important component of remote sensing image application research. To achieve more accurate scene classification, effectively extracting features describing the scene has become a key issue. In recent years, with the rapid development of deep learning technology, deep learning methods have achieved state-of-the-art performance in multiple fields such as image classification, semantic segmentation, and object detection.

[0003] Currently, convolutional neural networks have become an effective solution for solving remote sensing scene classification problems, but they still have certain limitations. For example, when there are few training samples (or reference samples) for each class, the model is prone to overfitting. Developing learning models suitable for small sample data is urgently needed. Small sample learning algorithms generally include two learning stages: the first stage is model pre-training on a base class with a large amount of data; the second stage is fine-tuning the pre-trained model on the target small sample task (also known as the new class) (e.g., only 1 or 5 samples per class). This strategy simulates the human visual cognitive recognition mechanism, effectively completing the current new task by transferring prior knowledge. Although self-supervised constraints can effectively improve the performance of small sample learning models, they still have significant limitations. Traditional techniques primarily use visual data, but ignore the semantic attributes in vision.

[0004] In summary, existing technologies using convolutional neural networks for remote sensing scene image classification suffer from problems such as overfitting, and few-shot learning models primarily using visual data while neglecting semantic attributes in the visual data. This application explores solutions to these issues. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a small-sample remote sensing scene image classification method, corresponding device, electronic device and computer-readable storage medium.

[0006] To achieve the various objectives of this application, the following technical solution is adopted:

[0007] A small-sample remote sensing scene image classification method proposed to meet one of the purposes of this application includes:

[0008] Respond to remote sensing scene image classification instructions to acquire the remote sensing scene image to be classified;

[0009] A pre-trained remote sensing scene image classification model is invoked, and the image feature extractor in the remote sensing scene image extracts the image features in the remote sensing scene image;

[0010] The image feature mapping network in the BERT model of the remote sensing scene image classification model performs global average pooling on the image features and uses two fully connected layers to map the image features to a 512-dimensional feature embedding space to determine the image mapping features.

[0011] The semantic feature mapping network in the BERT model extracts the semantic features of each category name in the remote sensing scene image, and uses two fully connected layers to map the semantic features to a 512-dimensional feature embedding space to determine the semantic mapping features.

[0012] The image mapping features and semantic mapping features are normalized using the second norm to determine the corresponding categories of each scene in the remote sensing scene image, thereby completing the classification of the remote sensing scene image.

[0013] Optional steps for training a remote sensing scene image classification model include:

[0014] Remote sensing scene images are obtained from a preset database and divided into a training set, a test set, and a validation set according to a preset ratio.

[0015] A remote sensing scene image x is selected from the training set, and the remote sensing scene image x is rotated by a rotation angle r to determine the rotated remote sensing scene image x. r ;

[0016] In the supervised learning module, an image feature extractor F is used. θ (·) Extract image features from the remote sensing scene image x, and perform a classification task based on the image features using a cosine classifier, wherein the cross-entropy loss function L of the supervised learning module... sup For L sup =-logC ω (F θ (x r )), where θ represents the parameters of the image feature extractor and ω represents the parameters of the cosine classifier;

[0017] In the self-supervised learning module, a rotation classifier R is used. φ (·) Predict the rotated remote sensing scene image x r The true rotation angle, wherein the cross-entropy loss function L of the self-supervised learning module. self For L self =-logR φ (F θ (xr )), R φ (·) is a four-way linear classifier with matrix parameter φ, whose predicted label is the rotation angle r;

[0018] The supervised learning module is combined with the self-supervised learning module to work together on the image feature extractor F. θ (·), where the objective loss function L all Represented as L all =L sup +αL self α is a hyperparameter;

[0019] The model parameters are updated according to the target loss function until the change in the target loss function is less than a preset value or the number of training iterations is greater than a preset value. Then, the model parameters are saved and the training of the remote sensing scene image classification model is completed.

[0020] Optional steps for training a remote sensing scene image classification model include:

[0021] Each task starts from D n Randomly select N classes, and then randomly select K samples from the selected N classes as the support set. Define the N×K samples as D. s ,in, A new class for use in the second phase;

[0022] Randomly select from the selected N classes that correspond to the D class. s The query set consists of M non-overlapping samples, and the N×M samples are defined as D. q ;

[0023] Freeze the image feature extractor F during the first stage of training θ (·), adopting the support set D in the new task s For the cosine classifier C ω The matrix parameter ω in (·) k Make fine adjustments;

[0024] The matrix parameter ω k Set the mean of all features of class k in the new task support set, and calculate the query set D. q Image features and various representative features ω k The similarity between the images is used to classify remote sensing scene images.

[0025] Optional steps for training a remote sensing scene image classification model include:

[0026] The remote sensing scene image classification model is trained using the stochastic gradient descent algorithm, with an initial learning rate of 0.1 and a momentum of 0.9.

[0027] Optionally, after the step of acquiring the remote sensing scene image to be classified, the following steps are included:

[0028] In response to the remote sensing scene image preprocessing command, the remote sensing scene image is normalized.

[0029] The preset image normalization algorithm is invoked to uniformly adjust the remote sensing scene image to 128×128 pixels.

[0030] Optionally, before the step of global average pooling of the image features in the image feature mapping network of the BERT model in the remote sensing scene image classification model, the following steps are included:

[0031] A loss function is constructed based on the cosine similarity of the image mapping features and semantic mapping features, with the cross-entropy loss L of the semantic regularization term. sema The calculation process is as follows:

[0032] P sema =softmax[sim(G λ (x r ),H μ (m) m∈M )],

[0033] L sema =-log(P sema ),

[0034] Among them, P sema Representing image mapping features and D b The similarity distribution of semantic mapping features for each class name in the dataset, where m is the class name and M represents D. b The set of category names. λ represents the parameters of the image feature mapping network, μ represents the parameters of the semantic feature mapping network, sim(·) represents the cosine similarity between the two, and softmax(·) represents the softmax activation function. This is the base class used in the first stage, for remote sensing scene images x∈X. b And the label y∈Y b .

[0035] Optionally, the remote sensing scene image includes one or more of the following: island remote sensing image, lake remote sensing image, mountain remote sensing image, and snow castle remote sensing image.

[0036] A small-sample remote sensing scene image classification device provided for another purpose of this application includes:

[0037] The remote sensing image acquisition module is configured to acquire remote sensing scene images to be classified in response to remote sensing scene image classification instructions;

[0038] The image feature extraction module is configured to call a pre-trained remote sensing scene image classification model, wherein the image feature extractor in the remote sensing scene image classification model extracts image features from the remote sensing scene image;

[0039] The image mapping determination module is configured to perform global average pooling on the image features in the BERT model of the remote sensing scene image classification model, and use two fully connected layers to map the image features to a 512-dimensional feature embedding space to determine the image mapping features.

[0040] The semantic mapping determination module is configured to extract the semantic features of each category name in the remote sensing scene image from the semantic feature mapping network in the BERT model, and use two fully connected layers to map the semantic features to a 512-dimensional feature embedding space to determine the semantic mapping features.

[0041] The remote sensing image classification module is configured to perform L2 norm normalization on the image mapping features and semantic mapping features to determine the corresponding category of each scene in the remote sensing scene image, thereby completing the classification of the remote sensing scene image.

[0042] An electronic device provided for another purpose of this application includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the small sample remote sensing scene image classification method of this application.

[0043] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the small sample remote sensing scene image classification method, which, when called by a computer, executes the steps included in the corresponding method.

[0044] Compared to existing technologies, this application addresses the problems of overfitting in convolutional neural networks used for remote sensing scene image classification, and the fact that few-shot learning models primarily use visual data but neglect semantic attributes. This application projects image features and semantic features derived from class names into a common feature embedding space. In this embedding space, the discriminative performance of image features is improved by aligning various image features with their corresponding semantic features. By effectively balancing self-supervised constraints and semantic regularization constraints, the model can learn image representation features that possess both good generalization and discriminative power, thereby effectively improving the performance of few-shot learning classification tasks.

[0045] Furthermore, the remote sensing scene image classification model of this application can greatly improve the classification accuracy of remote sensing scene images. By projecting image features and semantic features obtained from various class names into a common feature embedding space, it can better cope with complex and realistic scenes, thereby better identifying and predicting the scene category of remote sensing scene images. Attached Figure Description

[0046] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 This is a flowchart illustrating the small-sample remote sensing scene image classification method in this application embodiment;

[0048] Figure 2 This is a schematic diagram of a remote sensing scene image dataset in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of a remote sensing scene image classification model in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of semantic regular expressions in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of bridge categories in the remote sensing scene dataset in this application embodiment;

[0052] Figure 6 This is a schematic diagram of river categories in the remote sensing scene dataset in the embodiments of this application;

[0053] Figure 7 This is a schematic diagram of church categories in the remote sensing scene dataset in this application embodiment;

[0054] Figure 8 This is a schematic diagram of the principle of the small sample remote sensing scene image classification device in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0056] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0057] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0059] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0060] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0061] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0062] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0063] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0064] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0065] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0066] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0067] Please see Figure 1 The small-sample remote sensing scene image classification method of this application, in one embodiment, includes:

[0068] Step S10: Respond to the remote sensing scene image classification instruction and acquire the remote sensing scene image to be classified;

[0069] The computer terminal device can respond to remote sensing scene image classification instructions and acquire remote sensing scene images to be classified; the remote sensing scene images include one or more of the following: island remote sensing images, lake remote sensing images, mountain remote sensing images, and snow castle remote sensing images.

[0070] In some embodiments, please refer to Figure 2 The remote sensing scene image dataset was determined, including the UCMerced Land Use, NWPU-RESISC45, and AID remote sensing scene image datasets, etc.

[0071] The aforementioned UC Merced Land Use (UCM) dataset was manually extracted from urban areas of the USGS National Map by the United States Geological Survey. This dataset has a pixel resolution of 0.3 meters, a pixel size of 256×256, and contains 21 scene categories, with 100 images in each category. It is a relatively small-scale remote sensing scene dataset.

[0072] The NWPU-RESISC45 (NWPU) dataset, released by Northwestern Polytechnical University, exhibits high intra-class diversity and inter-class similarity. Except for categories with lower spatial resolution such as islands, lakes, mountains, and snow castles, most scene categories have spatial resolutions ranging from approximately 30 meters to 0.2 meters per pixel. The dataset has a pixel size of 256×256, contains 45 scene categories, and has 700 samples per category.

[0073] The aforementioned AID dataset was jointly proposed by Huazhong University of Science and Technology and Wuhan University. Constructed using images collected from Google Earth, it exhibits high intra-class diversity and inter-class similarity. This dataset has a pixel size of 600×600 pixels and contains 30 scene categories, with each category containing between 220 and 420 images.

[0074] In some embodiments, few-shot learning involves two learning phases with no class overlap (base class and new class), meaning the base class and new class have no overlapping classes. This application defines... This is the base class used in the first stage, where the remote sensing scene image x∈X b And the label y∈Y b .also, Defined as a new class used in the second phase.

[0075] In the second stage of few-shot learning, a large number of N-way K-shot tasks are performed from D. n Data collection. Specifically, each task starts from D. n Randomly select N classes, and then randomly select K samples from the selected N classes as the support set. These N×K samples are defined as D.s Then, randomly select from the selected N classes and choose one corresponding to D. s The query set consists of M non-overlapping samples, and these N×M samples are defined as D. q .

[0076] In some embodiments, after the step of acquiring the remote sensing scene image to be classified, the following steps are specifically included:

[0077] Step S101: In response to the remote sensing scene image preprocessing instruction, normalize the remote sensing scene image;

[0078] Step S103: Call the preset image normalization algorithm to uniformly adjust the remote sensing scene image to 128×128 pixels.

[0079] The image normalization algorithm includes one or more of the linear normalization algorithm and the nonlinear normalization algorithm.

[0080] Step S20: Call the pre-trained remote sensing scene image classification model, and the image feature extractor in the remote sensing scene image extracts the image features in the remote sensing scene image;

[0081] Please see Figure 3 After acquiring the remote sensing scene image to be classified, a pre-trained remote sensing scene image classification model is invoked. This model includes a supervised learning module, a self-supervised learning module, and a BERT model, among others.

[0082] In some embodiments, firstly, the supervised learning module in the remote sensing scene image classification model, following the traditional supervised learning approach, uses a cosine classifier C... ω (·) Complete the classification task; secondly, the self-supervised learning module in the remote sensing scene image classification model uses the rotation angle as supervision information, and through the rotation classifier R φ (·) Predicts the image rotation angle to solve the rotation prediction problem; finally, the semantic regularization term uses the BERT model to extract semantic features, denoted as B(·).

[0083] In some embodiments, the steps of training a remote sensing scene image classification model specifically include:

[0084] Step S201: Obtain remote sensing scene images from a preset database, and divide the remote sensing scene images into a training set, a test set, and a validation set according to a preset ratio;

[0085] Step S202: Select a remote sensing scene image x from the training set, rotate the remote sensing scene image x by a rotation angle r, and determine the rotated remote sensing scene image x. r ;

[0086] Step S203: In the supervised learning module, the image feature extractor F is used. θ (·) Extract image features from the remote sensing scene image x, and perform a classification task based on the image features using a cosine classifier, wherein the cross-entropy loss function L of the supervised learning module... sup For L sup =-logC ω (F θ (x r )), where θ represents the parameters of the image feature extractor and ω represents the parameters of the cosine classifier;

[0087] Step S204: In the self-supervised learning module, a rotation classifier R is used. φ (·) Predict the rotated remote sensing scene image x r The true rotation angle, wherein the cross-entropy loss function L of the self-supervised learning module. self For L self =-logR φ (F θ (x r )), R φ (·) is a four-way linear classifier with matrix parameter φ, whose predicted label is the rotation angle r;

[0088] Step S205: Combine the supervised learning module and the self-supervised learning module to work together on the image feature extractor F. θ (·), where the objective loss function L all Represented as L all =L sup +αL self α is a hyperparameter;

[0089] Step S206: Update the model parameters according to the target loss function until the change value of the target loss function is less than a preset value or the number of training times is greater than a preset value, then save the model parameters and complete the training of the remote sensing scene image classification model.

[0090] Specifically, in the first stage of learning, starting with D b Select a remote sensing scene image x, and randomly rotate the remote sensing scene image x by an angle r between R = {0°, 90°, 180°, 270°} to obtain the rotated image x. r We can assume x r Feature extractor F θ (·) The output image features are F θ (x r );

[0091] First, through the cosine classifier C ω(·) Complete the classification task, for example, if the input image is x r Then the cross-entropy loss L of the supervised learning module sup It is expressed as follows:

[0092] L sup =-logC ω (F θ (x r ))

[0093] In the formula, θ represents the parameters of the image feature extractor, and ω represents the parameters of the cosine classifier.

[0094] Secondly, the self-supervised module uses a rotating classifier R. φ (·) Predict x r The actual rotation angle, for example, inputting the rotated remote sensing scene image as x. r Then the self-supervised cross-entropy loss L self It is expressed as follows:

[0095] L self =-logR φ (F θ (x r ))

[0096] In the above formula, R φ (·) can be a four-way linear classifier with matrix parameter φ, whose predicted label is the rotation angle r.

[0097] Finally, the model combines supervised learning with self-supervised learning, which work together on the image feature extractor F. θ (·). Then the target loss function L all It is expressed as follows:

[0098] L all =L sup +αL self

[0099] In the above formula, the remote sensing scene image classification model is trained end-to-end using the stochastic gradient descent (SGD) algorithm, with an initial learning rate of 0.1 and a momentum of 0.9.

[0100] As can be seen from the above embodiments, this remote sensing scene image classification model greatly improves the generalization of the features learned by the model by introducing self-supervised learning.

[0101] For further details, please refer to Figure 4 By introducing semantic regularization terms through various category names, and by constraining the alignment of various image features with category semantic features, the discriminative power of image features can be improved.

[0102] To this end, this application employs a BERT model B(·) trained on a database of billions of texts for semantic feature extraction, which can extract high-quality semantic features from category text labels. Specifically, when the category name is input into the BERT model, BERT will output a 768-dimensional semantic feature vector.

[0103] In some embodiments, the step of training a remote sensing scene image classification model further includes:

[0104] Step S100, each task starts from D n Randomly select N classes, and then randomly select K samples from the selected N classes as the support set. Define the N×K samples as D. s ,in, A new class for use in the second phase;

[0105] Step S200: Randomly select from the selected N classes that match the D class. s The query set consists of M non-overlapping samples, and the N×M samples are defined as D. q ;

[0106] Step S300: Freeze the image feature extractor F trained in the first stage. θ (·), adopting the support set D in the new task s For the cosine classifier C ω The matrix parameter ω in (·) k Make fine adjustments;

[0107] Step S400: Set the matrix parameter ω k Set the mean of all features of class k in the new task support set, and calculate the query set D. q Image features and various representative features ω k The similarity between the images is used to classify remote sensing scene images.

[0108] Specifically, in the second stage of few-shot learning, in D n The N-way K-shot task with random sampling is used for few-shot learning, fine-tuning, and evaluation; during this stage, the image feature extractor F trained in the first stage is frozen. θ (·), and use the support set D in the new task. s For the cosine classifier C ω (·) Matrix parameter ω k Fine-tuning; in evaluating the settlement, for a given query set D q If the image is x, then the output similarity distribution P is obtained. few for:

[0109]

[0110] This application adopts the ProtoNet classification idea, that is, ω k Set the mean of all features of class k in the new task support set; during the evaluation phase, query set D q Image features and various representative features ω k Perform similarity calculations to determine the query set D. q Image features and various representative features ω k The similarity between the images is used to classify remote sensing scene images.

[0111] Step S30: The image feature mapping network in the BERT model of the remote sensing scene image classification model performs global average pooling on the image features and uses two fully connected layers to map the image features to a 512-dimensional feature embedding space to determine the image mapping features.

[0112] Specifically, the semantic regularization term adopts the BERT model mentioned above. The BERT model includes two mapping networks: an image feature mapping network G. λ (·) and semantic feature mapping network H μ (·); This image feature mapping network G λ (·) First, the image feature extractor F θ The extracted image features are subjected to global average pooling, and then the image features are mapped to a 512-dimensional feature embedding space through two fully connected layers to determine the image mapping features.

[0113] Step S40: The semantic feature mapping network in the BERT model extracts the semantic features of each category name in the remote sensing scene image, and uses two fully connected layers to map the semantic features to a 512-dimensional feature embedding space to determine the semantic mapping features;

[0114] Semantic Feature Mapping Network H μ (·) can be extracted from BERT model B(·) b The semantic features of each category name are obtained, and then the semantic features are mapped to a 512-dimensional feature embedding space through two fully connected layers. Then, the image mapping features and semantic mapping features are normalized by L2 norm.

[0115] In some embodiments, to align image features with semantic features and guide the model to learn the semantic attribute features of the image, this method constructs a loss function based on the cosine similarity between the two. The cross-entropy loss L of the semantic regularization term... sema The calculation process is as follows:

[0116] P sema =softmax[sim(G λ (x r),H μ (m) m∈M )];

[0117] L sema =-log(P sema );

[0118] Among them, P sema Representing image mapping features and D b The similarity distribution of semantic mapping features for each class name in the dataset, where m is the class name and M represents D. b The set of category names. λ represents the parameters of the image feature mapping network, μ represents the parameters of the semantic feature mapping network, sim(·) represents the cosine similarity between the two, and softmax(·) represents the softmax activation function. This is the base class used in the first stage, for remote sensing scene images x∈X. b And the label y∈Y b .

[0119] Step S50: Normalize the image mapping features and semantic mapping features using the second norm to determine the category corresponding to each scene in the remote sensing scene image, so as to complete the classification of the remote sensing scene image.

[0120] After determining the image mapping features and semantic mapping features based on the BERT model, the image mapping features and semantic mapping features are normalized to L2 to determine the category corresponding to each scene in the remote sensing scene image, so as to complete the classification of the remote sensing scene image.

[0121] After acquiring image and semantic features, the model maps both to a feature embedding space and constructs a loss function based on the similarity distribution between the image-mapped features and the semantic-mapped features for each category. Furthermore, this paper designs a novel loss function that combines supervised learning, self-supervised tasks, and semantic regularization terms, all acting together on the image feature extractor F. θ (·)

[0122] In some embodiments, to make the model more interpretable and to more intuitively demonstrate its effects, the Grad-CAM method can be used to generate a model heatmap, visually presenting class discrimination features. Please refer to [link to relevant documentation]. Figure 5 , Figure 6 as well as Figure 7 These images represent categories such as bridges, rivers, and churches in a remote sensing scene dataset, with three sets of example images for each category. In each set, the left image is the original image, and the right image is a heatmap. In the heatmap, the darker the red area, the greater its contribution to the final classification; the darker the blue area, the smaller its contribution. Through comparison... Figure 5 and Figure 6The central image reveals similarities between different categories of remote sensing scenes, with relatively small differences between categories. This is a detrimental factor affecting remote sensing scene classification. However, Figure 5 and Figure 6 The mid-heat activation map shows that the model identification can focus on the key features of both the bridge and the river. Additionally, Figure 7 For images where the church occupies most of the area, the discriminative features of the embedding model are concentrated on the church roof. Furthermore, for image areas not occupied by the church, the discriminative features of the embedding model can be concentrated on the entire church. Even when faced with the characteristics of remote sensing images such as large inter-class discrepancies and small intra-class discrepancies, the remote sensing scene image classification model of this application can still accurately capture class discriminative features, demonstrating the effectiveness of the few-sample remote sensing scene image classification method in solving the few-sample scene classification problem.

[0123] As can be seen from the above embodiments, compared with the prior art, this application addresses the problems in the prior art where convolutional neural networks are used for remote sensing scene image classification, such as the model being prone to overfitting, and the fact that few-shot learning models basically use visual data but ignore the semantic attributes in vision. This application projects the semantic features obtained by image features and class names into a common feature embedding space. In this embedding space, the discriminative performance of image features is improved by aligning the image features of each class with the corresponding semantic features. By effectively balancing self-supervised constraints and semantic regularization constraints, the model can learn image expression features that have both good generalization and good discriminative power, thereby effectively improving the performance of few-shot learning classification and recognition tasks.

[0124] Furthermore, the remote sensing scene image classification model of this application can greatly improve the classification accuracy of remote sensing scene images. By projecting image features and semantic features obtained from various class names into a common feature embedding space, it can better cope with complex and realistic scenes, thereby better identifying and predicting the scene category of remote sensing scene images.

[0125] Please see Figure 8A small-sample remote sensing scene image classification device provided for one of the purposes of this application includes a remote sensing image acquisition module 1100, an image feature extraction module 1200, an image mapping determination module 1300, a semantic mapping determination module 1400, and a remote sensing image classification module 1500. The system includes the following modules: a remote sensing image acquisition module 1100, configured to acquire a remote sensing scene image to be classified in response to a remote sensing scene image classification command; an image feature extraction module 1200, configured to call a pre-trained remote sensing scene image classification model, wherein the image feature extractor in the remote sensing scene image classification model extracts image features from the remote sensing scene image; an image mapping determination module 1300, configured to perform global average pooling on the image features using the image feature mapping network in the BERT model of the remote sensing scene image classification model, and use two fully connected layers to map the image features to a 512-dimensional feature embedding space to determine the image mapping features; a semantic mapping determination module 1400, configured to extract the semantic features of each category name in the remote sensing scene image using the semantic feature mapping network in the BERT model, and use two fully connected layers to map the semantic features to a 512-dimensional feature embedding space to determine the semantic mapping features; and a remote sensing image classification module 1500, configured to perform L2 normalization on the image mapping features and the semantic mapping features to determine the category corresponding to each scene in the remote sensing scene image, thereby completing the classification of the remote sensing scene image.

[0126] Based on any embodiment of this application, please refer to Figure 9 Another embodiment of this application also provides an electronic device, which can be implemented by a computer device, such as... Figure 9 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When executed by the processor, the computer-readable instructions enable the processor to implement a small-sample remote sensing scene image classification method. The processor provides computational and control capabilities, supporting the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to execute the small-sample remote sensing scene image classification method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] In this embodiment, the processor is used to execute... Figure 8 The system contains the specific functions of each module and its sub-modules, and the memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the small-sample remote sensing scene image classification device of this application, and the server can call the server's program code and data to execute the functions of all sub-modules.

[0128] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the small sample remote sensing scene image classification method described in any embodiment of this application.

[0129] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the small sample remote sensing scene image classification method described in any embodiment of this application.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0131] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

[0132] In summary, the remote sensing scene image classification model of this application can greatly improve the classification accuracy of remote sensing scene images. By projecting image features and semantic features obtained from various class names into a common feature embedding space, it can better cope with complex and realistic scenes, thereby better identifying and predicting the scene category of remote sensing scene images.

Claims

1. A method for classifying remote sensing scene images with a small sample size, characterized in that, include: Respond to remote sensing scene image classification instructions to acquire the remote sensing scene image to be classified; The steps for training a remote sensing scene image classification model include: Remote sensing scene images are obtained from a preset database and divided into a training set, a test set, and a validation set according to a preset ratio. Remote sensing scene images are selected from the training set. The remote sensing scene image According to the rotation angle Perform rotation to determine the rotated remote sensing scene image. ; In the supervised learning module, an image feature extractor is used. Extract the remote sensing scene image The image features are used to perform a classification task based on the image features, wherein the cross-entropy loss function of the supervised learning module is... for , This represents the parameters of the image feature extractor. The parameters represent the cosine classifier. Represents a cosine classifier; In the self-supervised learning module, a rotation classifier is used. Predicting the rotation of remote sensing scene images The true rotation angle, wherein the cross-entropy loss function of the self-supervised learning module. for , For matrix parameters A four-way linear classifier whose predicted label is the rotation angle. ; The supervised learning module is combined with the self-supervised learning module to work together in the image feature extractor. , where the objective loss function Represented as , For hyperparameters; The model parameters are updated according to the target loss function until the change in the target loss function is less than a preset value or the number of training iterations is greater than a preset value. Then, the model parameters are saved and the training of the remote sensing scene image classification model is completed. Each task from Randomly select N classes, and then randomly select K samples from the selected N classes as the support set. Define the N×K samples as... ,in, A new class for use in the second phase; Randomly select from the selected N classes that match the above. The query set consists of M non-overlapping samples. The N×M samples are defined as... ; Freeze the image feature extractor during the first stage of training The new task supports set Cosine classifier Matrix parameters in Make fine adjustments; The matrix parameters Set as a new task to support the first The mean of all features of a class is used to calculate and determine the query set. Image features and various representative features The similarity between the images is used to classify remote sensing scene images. A pre-trained remote sensing scene image classification model is invoked, and the image feature extractor in the remote sensing scene image extracts the image features in the remote sensing scene image; The image feature mapping network in the BERT model of the remote sensing scene image classification model performs global average pooling on the image features and uses two fully connected layers to map the image features to a 512-dimensional feature embedding space to determine the image mapping features. The semantic feature mapping network in the BERT model extracts the semantic features of each category name in the remote sensing scene image, and uses two fully connected layers to map the semantic features to a 512-dimensional feature embedding space to determine the semantic mapping features. The image mapping features and semantic mapping features are normalized using the second norm to determine the corresponding categories of each scene in the remote sensing scene image, thereby completing the classification of the remote sensing scene image.

2. The small-sample remote sensing scene image classification method according to claim 1, characterized in that, The steps for training a remote sensing scene image classification model include: The remote sensing scene image classification model is trained using the stochastic gradient descent algorithm, with an initial learning rate of 0.1 and a momentum of 0.

9.

3. The small-sample remote sensing scene image classification method according to claim 1, characterized in that, After acquiring the remote sensing scene image to be classified, the following steps are included: In response to the remote sensing scene image preprocessing command, the remote sensing scene image is normalized. The preset image normalization algorithm is invoked to uniformly adjust the remote sensing scene image to 128×128 pixels.

4. The small-sample remote sensing scene image classification method according to claim 1, characterized in that, Before the step of global average pooling of the image features in the image feature mapping network of the BERT model in the remote sensing scene image classification model, the following steps are included: A loss function is constructed based on the cosine similarity of the image mapping features and semantic mapping features, with cross-entropy loss as the semantic regularization term. The calculation process is as follows: ， ， in, Representing image mapping features and The similarity distribution of semantic mapping features for each class name in the dataset. It is the category name. express A collection of category names, Represents the parameters of the image feature mapping network. Represents the semantic feature mapping network parameters. Indicates the cosine similarity between the two. express Activation function This is the base class used in the first phase, for remote sensing scene images. And tags .

5. The small-sample remote sensing scene image classification method according to any one of claims 1 to 4, characterized in that, The remote sensing scene images include one or more of the following: island remote sensing images, lake remote sensing images, mountain remote sensing images, and snow castle remote sensing images.

6. A small-sample remote sensing scene image classification device, characterized in that, include: The remote sensing image acquisition module is configured to acquire remote sensing scene images to be classified in response to remote sensing scene image classification instructions; The steps for training a remote sensing scene image classification model include: Remote sensing scene images are obtained from a preset database and divided into a training set, a test set, and a validation set according to a preset ratio. Remote sensing scene images are selected from the training set. The remote sensing scene image According to the rotation angle Perform rotation to determine the rotated remote sensing scene image. ; In the supervised learning module, an image feature extractor is used. Extract the remote sensing scene image The image features are used to perform a classification task based on the image features, wherein the cross-entropy loss function of the supervised learning module is... for , This represents the parameters of the image feature extractor. The parameters represent the cosine classifier. Represents a cosine classifier; In the self-supervised learning module, a rotation classifier is used. Predicting the rotation of remote sensing scene images The true rotation angle, wherein the cross-entropy loss function of the self-supervised learning module. for , For matrix parameters A four-way linear classifier whose predicted label is the rotation angle. ; The supervised learning module is combined with the self-supervised learning module to work together in the image feature extractor. , where the objective loss function Represented as , For hyperparameters; The model parameters are updated according to the target loss function until the change in the target loss function is less than a preset value or the number of training iterations is greater than a preset value. Then, the model parameters are saved and the training of the remote sensing scene image classification model is completed. Each task from Randomly select N classes, and then randomly select K samples from the selected N classes as the support set. Define the N×K samples as... ,in, A new class for use in the second phase; Randomly select from the selected N classes that match the above. The query set consists of M non-overlapping samples. The N×M samples are defined as... ; Freeze the image feature extractor during the first stage of training The new task supports set Cosine classifier Matrix parameters in Make fine adjustments; The matrix parameters Set as a new task to support the first The mean of all features of a class is used to calculate and determine the query set. Image features and various representative features The similarity between the images is used to classify remote sensing scene images. The image feature extraction module is configured to call a pre-trained remote sensing scene image classification model, wherein the image feature extractor in the remote sensing scene image classification model extracts image features from the remote sensing scene image; The image mapping determination module is configured to perform global average pooling on the image features in the BERT model of the remote sensing scene image classification model, and use two fully connected layers to map the image features to a 512-dimensional feature embedding space to determine the image mapping features. The semantic mapping determination module is configured to extract the semantic features of each category name in the remote sensing scene image from the semantic feature mapping network in the BERT model, and use two fully connected layers to map the semantic features to a 512-dimensional feature embedding space to determine the semantic mapping features. The remote sensing image classification module is configured to perform L2 norm normalization on the image mapping features and semantic mapping features to determine the corresponding category of each scene in the remote sensing scene image, thereby completing the classification of the remote sensing scene image.

7. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 5, which, when invoked by a computer, executes the steps included in the corresponding method.

Images