A pure text remote sensing scene classification method based on semantic agent

Abstract

The present application belongs to the technical field of remote sensing image processing, and particularly relates to a pure text remote sensing scene classification method based on semantic agent. The present application inputs remote sensing scene images and agent generation prompts into a visual language model to generate text semantic agents corresponding to single images; constructs a pure text data set based on the text semantic agents and corresponding scene labels; encodes the text semantic agents into text features by using a text embedding model, performs label space adaptation through a projection adapter containing layer normalization, a linear layer, a GELU activation layer and a dropout layer, and then outputs scene categories by a classifier. After the semantic agent is generated, the training and inference of the classifier are both performed in the text space without accessing original image pixels or image visual features. The present application can reduce transmission bandwidth, inference parameter quantity and storage overhead, and is suitable for privacy-sensitive sharing, low-resource deployment and semantic priority analysis scenes.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a pure text remote sensing scene classification method based on semantic proxy. Background Technology

[0002] Remote sensing scene classification is used to assign semantic land use or land cover categories to aerial or satellite remote sensing image patches, and is a fundamental step in applications such as urban planning, resource surveys, environmental monitoring, and disaster response. Existing methods typically use convolutional neural networks, visual Transformers, or basic remote sensing vision models to extract visual features from image pixels, and then perform scene prediction based on these visual features. Even in visual language classification methods that incorporate textual cues or image-text alignment, the final classification usually still relies on the visual features output by the image encoder.

[0003] The aforementioned visual feature-centric classification paradigm can achieve high accuracy when images can be centrally stored and continuously transmitted. However, in bandwidth-constrained scenarios such as satellite-to-ground links, UAV backhaul, or edge terminals, transmitting high-resolution images requires significant communication overhead. When sensitive areas or controlled data sharing are involved, directly transmitting raw images may expose unnecessary details. Deploying a visual backbone network on low-computing-power terminals will result in a high number of parameters, high memory usage, and high inference latency.

[0004] Recent visual language models have been able to convert image content into natural language descriptions. However, current technologies mostly use these descriptions as auxiliary information, category hints, or interactive responses for image classification, and have not yet formed a complete remote sensing scene classification process with image-specific text as the sole classification input. Ordinary image descriptions may also only list salient objects, omitting discriminative evidence that determines the category of a remote sensing scene, such as object co-occurrence, spatial layout, density, scale, and functional context. Therefore, how to compress remote sensing images into a verifiable and classifiable semantic representation, and complete downstream classification without accessing image pixels and visual features, is a problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the bandwidth, privacy, computation, and interpretability issues caused by the continued reliance on image input in existing remote sensing scene classification methods, this invention proposes a pure text remote sensing scene classification method based on semantic proxies.

[0006] The technical solution adopted in this invention is as follows:

[0007] A pure text remote sensing scene classification method based on semantic agents includes the following steps:

[0008] S1. Obtain the remote sensing scene image dataset, scene labels corresponding to each remote sensing scene image, and agent generation prompts;

[0009] S2. Input the remote sensing scene image and the proxy-generated prompt into the visual language model, and perform semantic abstraction of natural language through visual observation through the visual language model to generate a text semantic proxy corresponding to a single remote sensing scene image; the text semantic proxy represents at least one discriminative information among global scene, key objects, object attributes, co-occurrence relationships, spatial organization, density, scale and functional context;

[0010] S3. Combine each text semantic agent with its corresponding scene label to form a text sample pair and construct a pure text scene classification dataset; when using different visual language models or different agents to generate prompts, construct multiple text agent sets for the same remote sensing scene image dataset respectively.

[0011] S4. Use a text embedding model to encode text semantic proxies into text features, use a projection adapter to map the general text embedding space to the remote sensing scene label space, and use a classifier to output the scene category probability; use scene labels to train the projection adapter and the classifier to obtain a pure text remote sensing scene classification model.

[0012] S5. For the remote sensing scene image to be classified, first generate the semantic agent of the text to be classified, then input the semantic agent of the text to be classified into the trained pure text remote sensing scene classification model, and finally obtain the scene classification result without accessing the image pixels or image visual features of the remote sensing scene image to be classified.

[0013] Furthermore, in S2, the agent-generated prompts include tiling prompts and structured prompts; the tiling prompts are used to instruct the visual language model to describe the remote sensing scene in continuous natural language paragraphs, the description content covering at least one of the overall scene, key objects, spatial relationships between objects, and scene functions, and prohibiting the output of only category names; the structured prompts are used to instruct the visual language model to output scene information according to the order of preset semantic fields, the preset semantic fields including at least two of the following: global scene, object-level attributes, functional cues, spatial layout, object co-occurrence, density, and scale;

[0014] The visual language model includes a locally deployed open-source visual language model, a closed-source visual language model accessed through an interface, or a dedicated visual language model fine-tuned for remote sensing data.

[0015] Furthermore, in S3, the sample partitioning of the pure text scene classification dataset maintains the same sample index and category distribution as the original remote sensing scene image dataset; a unified data partitioning index is used for different text agent sets to eliminate the impact of data partitioning differences on model evaluation.

[0016] Furthermore, in S4, the projection adapter sequentially performs layer normalization, linear mapping, GELU activation, and discarding operations to rearrange the general text features to a label space that adapts to remote sensing scene classification; the text embedding model keeps its parameters frozen during training, or updates its parameters through efficient parameter fine-tuning or full fine-tuning.

[0017] The classifier is a linear classifier that uses the cross-entropy loss function to update the projection adapter and classifier parameters through backpropagation, and selects the optimal model parameters by using the classification accuracy of the validation set.

[0018] Furthermore, it also includes cross-generator inference adaptation methods: when the proxy generator in the inference stage is different from the proxy generator in the training stage, the trained classifier is directly used for inference, or multiple generator text semantic proxies are mixed in the training set, the text semantic proxies are normalized, or generator invariant constraints are added to the projection adapter to reduce the impact of proxy distribution offset on classification accuracy.

[0019] Furthermore, the method supports distributed deployment: the image acquisition end, satellite end, or drone edge end runs a visual language model to generate text semantic agents, and only the text semantic agents are transmitted to the classification end via the communication link; the classification end runs a text embedding model, projection adapter, and classifier to complete scene classification without access to the original image.

[0020] A system for classifying plain text remote sensing scenes based on semantic agents, comprising:

[0021] The data acquisition module is used to acquire remote sensing scene images, scene labels corresponding to each remote sensing scene image, and proxy generation prompts;

[0022] The semantic proxy generation module is used to input the remote sensing scene image and the proxy generation prompt into the visual language model to generate a text semantic proxy corresponding to a single remote sensing scene image.

[0023] The plain text dataset construction module is used to combine the text semantic agents and corresponding scene labels to form text sample pairs and construct a plain text scene classification dataset.

[0024] The text encoding module is used to encode the text semantic proxy into text features using a text embedding model;

[0025] The projection adaptation module is used to map the text features to the remote sensing scene label space using a projection adapter to obtain adapted text features.

[0026] The classification module is used to output scene category probabilities based on the adapted text features using a classifier, and to train the projection adapter and the classifier using scene labels;

[0027] In the inference stage, the classification result is output only based on the semantic agent of the text to be classified, without accessing the image pixels or image visual features of the remote sensing scene image to be classified.

[0028] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the plain text remote sensing scene classification method based on semantic agents as described in this invention.

[0029] The beneficial effects of this invention are as follows:

[0030] (1) By using text semantic proxy to carry high-level semantic evidence of remote sensing scene, after the proxy is generated, the classification training and reasoning are transferred from visual space to text space, reducing the dependence of the classification end on original image, visual backbone network and large-scale visual features.

[0031] (2) Text semantic agents can be generated once at the image acquisition end or edge end and reused for classification, retrieval or statistical analysis. In low bandwidth links, only text can be transmitted to support scene prediction and avoid directly sharing the original image details with the classification end.

[0032] (3) Text semantic agents can be manually checked and reviewed, which makes it easier to locate the classification basis and agent generation error; the visual language model, agent generation prompts, text embedding model and classifier are decoupled from each other, which makes it easy to replace, reuse and extend them respectively.

[0033] (4) A lightweight projection adapter is used to adapt the label space of general text features. While keeping the classification network structure simple, it improves the separability of different remote sensing scene categories in the text feature space and reduces the number of training parameters, storage overhead and inference resource requirements. Attached Figure Description

[0034] Figure 1 This diagram illustrates a comparison of the task paradigms and application scenarios of traditional remote sensing scene classification, visual language remote sensing scene classification, and the plain text remote sensing scene classification of this application.

[0035] Figure 2 A schematic diagram of the overall framework of the semantic proxy-based plain text remote sensing scene classification method provided for the embodiments. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] like Figure 1As shown, traditional remote sensing scene classification directly inputs remote sensing scene images into a visual encoder and outputs scene categories based on visual features. While visual language remote sensing scene classification introduces scene descriptions or category cues, it still uses images or visual features as classification input. The pure text remote sensing scene classification proposed in this application first converts the image into a text semantic proxy using a visual language model, and then the text encoder and classifier output scene categories based solely on the text semantic proxy.

[0038] The "text semantic proxy" referred to in this application is a natural language intermediate representation generated for a specific remote sensing scene image. Its function is to carry visual evidence related to downstream scene classification in text form and replace direct access to the original image or image visual features during the classification stage. The text semantic proxy is not limited to a fixed language format and can be continuous paragraphs, field-based descriptions, short summaries, or other text that can be processed by a text embedding model, as long as it can preserve scene discriminative information.

[0039] Text semantic agents differ from category labels, category hints, and ordinary image descriptions. Category labels only define the prediction target, category hints provide category-level priors, and both usually lack image specificity; ordinary image descriptions primarily aim to state content; while text semantic agents aim to support plain text classification and should preserve as much evidence as possible, including global scene, key objects, object attributes, co-occurrence relationships, spatial organization, density, scale, and functional context.

[0040] like Figure 2 As shown, in Phase I, the remote sensing scene image is... And the proxy generates prompts p, using a visual language model to generate text semantic proxies. The relationship is expressed as:

[0041] ,

[0042] In the formula, i represents the sample number; Represents the i-th remote sensing scene image; p represents the proxy-generated prompt; Represents the semantic abstraction function executed by the visual language model; This represents the text semantic proxy corresponding to the i-th remote sensing scene image. At this stage, the visual language model acts as a semantic abstraction module, rather than a classifier that directly outputs the final scene label.

[0043] In one embodiment, the proxy-generated prompt is a tiling prompt. This prompt instructs the visual language model to describe the remote sensing scene as a continuous natural language paragraph, including at least one of the following: the overall scene, key objects, spatial relationships between objects, and scene functions. It also requires avoiding outputting only category names. Tiling prompts facilitate the generation of semantically coherent scene-level descriptions that match a general text embedding space.

[0044] In another embodiment, the proxy generation prompt is a structured prompt. This prompt instructs the visual language model to output multiple semantic fields in a fixed order of "global scene—object-level attributes—functional cues"; spatial layout, object co-occurrence, density, or scale fields can also be added as needed for the task. Structured prompts facilitate the verification of different types of scene evidence. Both tiling prompts and structured prompts are optional proxy generation methods, and there is no requirement that one format necessarily has higher classification performance.

[0045] In one embodiment, one or more visual language models are used to generate text semantic proxies. The visual language models can be locally deployed open-source models, closed-source models accessed via an interface, or dedicated models fine-tuned for remote sensing data. For the same dataset, combinations of different visual language models and different proxy-generated cues form separate sets of text proxies to compare semantic abstraction behaviors or perform cross-generator transfer within the same downstream classifier architecture.

[0046] Suppose that the remote sensing scene image dataset D consists of N image label pairs. ,in This is one of C preset scene categories. For each image... After performing the proxy generation process shown in equation (1), a plain text scene classification dataset is obtained:

[0047]

[0048] In the formula, This represents the i-th text semantic proxy generated under prompt p. The pure text scene classification dataset only saves the text semantic proxy and scene labels; in Stage II, the original remote sensing scene image or image features output by the visual encoder are no longer provided to the classification model.

[0049] When partitioning the data, plain text sample pairs can be divided into training, validation, and test sets according to a preset ratio, while maintaining the same sample index and category distribution as the original remote sensing scene image dataset. For text proxy sets generated by different proxy generators, the same partition index can be used to avoid differences in data partitioning affecting the comparison results.

[0050] In Phase II, text semantic proxy will be implemented. Input text embedding model To obtain text features :

[0051]

[0052] The text embedding model can employ a general sentence vector model, a remote sensing domain text encoder, or other models capable of mapping natural language into fixed-length vectors. In one embodiment, the text embedding model is frozen during classification training so that the classification results primarily reflect the scene information retained by the text semantic agent; in other embodiments, the text embedding model can be updated using efficient parameter fine-tuning or full fine-tuning.

[0053] To adapt the general text embedding space to the remote sensing scene label space, text features are... Input projection adapter A to obtain adapted text features This adapter is used to compress or rearrange common text features, making agents of the same category more aggregated in the target label space.

[0054] Adapting text features Inputting the linear classifier yields probability vectors for C preset scene categories. :

[0055]

[0056] In the formula, and and represent the weights and biases of the classifier, respectively. The class with the highest probability is taken as the scene classification result. The cross-entropy loss function is used during the training phase.

[0057]

[0058] In the formula, This indicates whether the i-th sample belongs to the c-th class. This represents the probability that the model predicts the i-th sample belongs to class c. The projection adapter and classifier parameters are updated through backpropagation, and the optimal model parameters can be selected based on the classification accuracy on the validation set.

[0059] During the training phase, text semantic proxies corresponding to the training, validation, and test sets are first generated offline. Then, only the text semantic proxies and their scene labels are loaded, and text embeddings are calculated or read in batches to train the projection adapter and classifier. Since images are not included in Phase II, classification training can be performed in a computing environment without image access permissions. The generated text semantic proxies can also be cached and reused for different classifiers, retrieval systems, or statistical analysis models.

[0060] During the inference phase, proxy generation and plain text classification can be performed sequentially on the same device, or a distributed deployment can be adopted. In distributed deployment, the visual language model runs at the image acquisition end, satellite end, or drone edge end and generates text semantic proxies; then only the text semantic proxies are sent to the classification end via the communication link. The classification end runs a text embedding model, projection adapter, and classifier, and outputs the scene category without receiving the original image.

[0061] When the proxy generator differs from the proxy generator used during the training phase, the text semantic proxy may have different language styles, granularities, or descriptive preferences. Cross-generator inference can be performed directly using the trained classifier, or multiple generator text semantic proxies can be mixed in the training set, the text can be normalized, or generator-invariant constraints can be added to reduce proxy distribution bias. These processes do not change the fundamental characteristic of using only text semantic proxies during the classification phase.

[0062] To verify the feasibility of the method, three remote sensing scene classification datasets—UCM, AID, and NWPU-RESISC45—were selected in one example. These datasets contain 21 classes with 2100 images, 30 classes with 10000 images, and 45 classes with 31500 images, respectively. For each image, a lightweight local visual language model, a medium-sized local visual language model, and an interface visual language model were used, combined with tiling cues or structured cues to generate text semantic proxies. Subsequently, the bge-base-en-v1.5 text embedding model and a linear classifier were used for pure text classification. Each experiment was repeated five times.

[0063] In this example, using the best-performing proxy generator and tiling cues, UCM achieves overall accuracy of 96.19% and 95.39% at 50% and 80% training ratios, respectively; AID achieves 93.95% and 95.18% at 20% and 50% training ratios, respectively; and NWPU-RESISC45 achieves 89.13% and 90.08% at 10% and 20% training ratios, respectively. These results demonstrate that text semantic proxies can retain a large amount of scene-level discriminative information and support effective classification without using image features.

[0064] In the efficiency comparison with 50% training ratio on the AID dataset, this example's projection adapter and classifier together contain approximately 0.54M trainable parameters, with a single iteration training time of approximately 0.0701 seconds, a testing time of approximately 0.0935 seconds, and a peak GPU memory usage of approximately 1266MB. Compared to the scheme that runs the visual backbone network on the classification end, Stage II has a lower parameter count and resource consumption. It should be noted that the offline cost of generating text semantic agents using the visual language model is not included in the above Stage II efficiency data, and the model name, dataset, parameters, and results mentioned above are for illustrative purposes only and do not constitute a limitation on the scope of protection of this application.

[0065] This application also provides a plain text remote sensing scene classification system based on semantic proxies. The data acquisition module acquires remote sensing scene images, scene labels, and proxy generation prompts; the semantic proxy generation module calls a visual language model to generate text semantic proxies; the plain text dataset construction module forms text sample pairs; the text encoding module generates text features; the projection adaptation module performs layer normalization, linear mapping, GELU activation, and discarding; and the classification module trains a linear classifier and outputs the scene category. These modules can be deployed on the same computing node, or the semantic proxy generation module can be deployed on the image processing side, while the other modules are deployed on the classification side.

[0066] This application also provides an electronic device, including a processor, a memory, a communication interface, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above method embodiments. The electronic device can be a server, workstation, edge computing device, spaceborne computing unit, UAV-borne computing unit, or other device with data processing capabilities.

[0067] This application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method. The storage medium may include a read-only memory, random access memory, solid-state memory, magnetic disk, optical disk, or other media capable of storing program code.

[0068] The technical features in the above embodiments can be combined arbitrarily without contradicting each other. The terms "comprising" and "including" used herein are open-ended expressions; unless otherwise expressly defined, the singular form also includes the plural form. For those skilled in the art, any equivalent substitutions, simple modifications, or improvements made without departing from the concept of this application should fall within the protection scope of this application.

Claims

1. A pure text remote sensing scene classification method based on semantic proxy, characterized in that, Includes the following steps: S1. Obtain the remote sensing scene image dataset, scene labels corresponding to each remote sensing scene image, and agent generation prompts; S2. Input the remote sensing scene image and the proxy generated prompt into the visual language model, and perform semantic abstraction of natural language through the visual language model to generate a text semantic proxy corresponding to a single remote sensing scene image. The text semantic proxy represents at least one discriminative information among the global scene, key objects, object attributes, co-occurrence relationships, spatial organization, density, scale, and functional context; S3. Combine each text semantic agent with its corresponding scene label to form a text sample pair and construct a pure text scene classification dataset. When using different visual language models or different agents to generate prompts, multiple sets of text agents are constructed for the same remote sensing scene image dataset. S4. Use a text embedding model to encode text semantic proxies into text features, use a projection adapter to map the general text embedding space to the remote sensing scene label space, and use a classifier to output the scene category probability; use scene labels to train the projection adapter and the classifier to obtain a pure text remote sensing scene classification model. S5. For the remote sensing scene image to be classified, first generate the semantic agent of the text to be classified, then input the semantic agent of the text to be classified into the trained pure text remote sensing scene classification model, and finally obtain the scene classification result without accessing the image pixels or image visual features of the remote sensing scene image to be classified.

2. The plain text remote sensing scene classification method based on semantic proxy according to claim 1, characterized in that, In S2, the agent-generated prompts include tiling prompts and structured prompts; the tiling prompts are used to instruct the visual language model to describe the remote sensing scene in continuous natural language paragraphs, and the description content covers at least one of the following: the overall scene, key objects, spatial relationships between objects, and scene functions, and it is prohibited to output only category names; The structured prompts are used to instruct the visual language model to output scene information in the order of preset semantic fields. The preset semantic fields include at least two of the following: global scene, object-level attributes, functional cues, spatial layout, object co-occurrence, density, and scale. The visual language model includes a locally deployed open-source visual language model, a closed-source visual language model accessed through an interface, or a dedicated visual language model fine-tuned for remote sensing data.

3. The plain text remote sensing scene classification method based on semantic proxy according to claim 1, characterized in that, In S3, the sample partitioning of the pure text scene classification dataset maintains the same sample index and category distribution as the original remote sensing scene image dataset; a unified data partitioning index is used for different text agent sets to eliminate the impact of data partitioning differences on model evaluation.

4. The plain text remote sensing scene classification method based on semantic proxy according to claim 1, characterized in that, In S4, the projection adapter sequentially performs layer normalization, linear mapping, GELU activation, and discarding operations to rearrange the general text features to a label space that adapts to remote sensing scene classification; the text embedding model keeps its parameters frozen during training, or updates its parameters through efficient parameter fine-tuning or full fine-tuning. The classifier is a linear classifier that uses the cross-entropy loss function to update the projection adapter and classifier parameters through backpropagation, and selects the optimal model parameters by using the classification accuracy of the validation set.

5. The plain text remote sensing scene classification method based on semantic proxy according to claim 1, characterized in that, It also includes cross-generator inference adaptation methods: when the proxy generator in the inference stage is different from the proxy generator in the training stage, the trained classifier is directly used for inference, or multiple generator text semantic proxies are mixed in the training set, the text semantic proxies are normalized, or generator invariant constraints are added to the projection adapter to reduce the impact of proxy distribution offset on classification accuracy.

6. The plain text remote sensing scene classification method based on semantic proxy according to claim 1, characterized in that, The method supports distributed deployment: the visual language model is run at the image acquisition end, satellite end, or drone edge end to generate text semantic agents, and only the text semantic agents are transmitted to the classification end via the communication link; The classification module runs a text embedding model, a projection adapter, and a classifier to perform scene classification without access to the original images.

7. The system for pure text remote sensing scene classification based on semantic proxy according to any one of claims 1-6, characterized in that, include: The data acquisition module is used to acquire remote sensing scene images, scene labels corresponding to each remote sensing scene image, and proxy generation prompts; The semantic proxy generation module is used to input the remote sensing scene image and the proxy generation prompt into the visual language model to generate a text semantic proxy corresponding to a single remote sensing scene image. The plain text dataset construction module is used to combine the text semantic agents and corresponding scene labels to form text sample pairs and construct a plain text scene classification dataset. The text encoding module is used to encode the text semantic proxy into text features using a text embedding model; The projection adaptation module is used to map the text features to the remote sensing scene label space using a projection adapter to obtain adapted text features. The classification module is used to output scene category probabilities based on the adapted text features using a classifier, and to train the projection adapter and the classifier using scene labels; In the inference stage, the classification result is output only based on the semantic agent of the text to be classified, without accessing the image pixels or image visual features of the remote sensing scene image to be classified.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the plain text remote sensing scene classification method based on semantic agents as described in any one of claims 1 to 6.

Images