Remote sensing image subtitle generation method and system

Abstract

The invention discloses a remote sensing image subtitle generation method and system. The method comprises the steps of obtaining a combined remote sensing image data set; for each remote sensing image in the combined remote sensing image data set, generating first annotation information of the remote sensing image based on a remote sensing image category, a target detection bounding box and a semantic segmentation mask, determining image complexity based on the first annotation information, and determining a subtitle generation mode according to the image complexity, the subtitle generation mode comprises generation of subtitles according to rules, generation of subtitles by a multi-modal large model and generation of subtitles by combining generation of subtitles according to rules and generation of subtitles by the multi-modal large model; generating subtitles of the remote sensing image based on a subtitle generation mode; all the image subtitle pairs form a remote sensing image-text pairing data set; and training a CLIP model based on the remote sensing image-text pairing data set. According to the method, the workload of manual labeling is effectively reduced, and the construction cost of the high-quality remote sensing image-text data set is reduced.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and specifically to a method and system for generating captions for remote sensing images. Background Technology

[0002] With the rapid development of remote sensing technology, the demand for automated analysis of remote sensing images is constantly increasing. To address this demand, visual language models (such as CLIP) are increasingly being applied to image-text pairing tasks, demonstrating powerful capabilities in various remote sensing image recognition tasks. However, the current large-scale remote sensing image-text pairing datasets are of insufficient quality, severely limiting the further development of these models in the field of remote sensing. Most existing high-quality remote sensing image-text datasets are small in scale and rely on manual annotation, resulting in high costs and low efficiency. Manually annotating remote sensing image captions requires not only high levels of expertise but is also time-consuming, labor-intensive, and inefficient, making it difficult to meet the needs of large-scale data. For example, datasets such as UCMcaption and RSICD contain manually annotated captions, but their limited size makes it difficult to meet the needs of training large-scale visual language models. Furthermore, some studies have attempted to sift through large-scale remote sensing image data from public natural image-text datasets; however, these datasets are mostly sourced from the internet, contain a lot of noise, and do not perfectly match the characteristics of remote sensing images. Other studies use rule-based methods to generate captions for large-scale remote sensing images. While this method can generate a large amount of image-text pairing data using existing labeled data, the resulting captions often lack contextual information, have repetitive sentence structures, and fail to provide natural and semantically rich content. To address these issues, there is an urgent need for an efficient and low-cost remote sensing image captioning generation method to automatically generate high-quality image captions for large-scale remote sensing images, thereby improving the performance of visual language models in remote sensing applications. Summary of the Invention

[0003] In view of this, the present invention provides a method and system for generating captions for remote sensing images, which can solve the above-mentioned technical problems.

[0004] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0005] A method for generating captions for remote sensing images, comprising: Step S1: Obtain multiple open-source multi-source remote sensing image datasets, preprocess each multi-source remote sensing image dataset separately, and merge the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset. Step S2: For each remote sensing image in the merged remote sensing image dataset: generate the first annotation information of the remote sensing image based on the remote sensing image category, target detection bounding box, and semantic segmentation mask; determine the image complexity based on the first annotation information; determine the caption generation method based on the image complexity; the caption generation method includes rule-based caption generation, multimodal large model caption generation, and rule-based caption generation combined with multimodal large model caption generation; generate the caption for the remote sensing image based on the caption generation method. Step S3: Each remote sensing image and its corresponding caption form an image-caption pair, and all image-caption pairs form a remote sensing image-text pairing dataset; train the CLIP model based on the remote sensing image-text pairing dataset to obtain the trained CLIP model.

[0006] Preferably, step S1 involves: acquiring multiple open-source multi-source remote sensing image datasets, preprocessing each multi-source remote sensing image dataset separately, and merging the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset, wherein: Multi-source remote sensing image datasets include multispectral composite image datasets, UAV aerial target detection datasets, satellite semantic segmentation datasets, remote sensing target detection datasets, and remote sensing scene classification datasets; Preprocessing includes: deleting remote sensing images without annotation information from each multi-source remote sensing image dataset; deduplicating remote sensing images by hashing and identifying the URLs corresponding to each remote sensing image; performing semantic alignment on the annotations of each remote sensing image and deleting remote sensing images corresponding to annotations without specific entity information; for remote sensing images whose size exceeds a first preset threshold, using a sliding window to divide them into several non-overlapping sub-images, and treating each sub-image as a remote sensing image; and constructing a category dictionary including multiple categories, mapping the category to which the remote sensing image belongs to the category defined in the category dictionary.

[0007] Preferably, in step S2, generating the first annotation information of the remote sensing image based on the remote sensing image category, target detection bounding box, and semantic segmentation mask includes: Generate category labels based on remote sensing image categories; determine the spatial location information of bounding boxes based on target detection bounding boxes, including entity objects, entity object categories, and quantities within the bounding boxes; determine the contour of semantic segmentation regions based on semantic segmentation masks; The first annotation information is generated by integrating category labels, bounding box spatial location information, semantic segmentation region outline, entity objects inside the bounding box, entity object category and quantity; wherein, the bounding box spatial location information includes the coordinates of the bounding box center point and the bounding box size; the first annotation information includes category labels, entity objects inside the bounding box, entity object category and quantity, bounding box spatial location information and semantic segmentation boundary.

[0008] Preferably, in step S2, determining the image complexity based on the first annotation information and determining the subtitle generation method based on the image complexity includes: The image complexity is determined based on the first annotation information. The formula for calculating the image complexity is as follows:

[0009] Where A is the image complexity. Nobj Represents the number of bounding boxes detected for the target. Ncls Represents the number of entity object categories; Roverlap This represents the average overlap ratio of the bounding boxes detected by the target. Hentropy Distribution entropy, representing the category of entity objects, is used to measure the semantic diversity of an image; Dspatial Represents spatial density, used to measure the uniformity of the distribution of physical objects in remote sensing images; α , β , γ , δ , ζ The weighting parameters are set through cross-validation or experimental experience. The subtitle generation method is determined based on image complexity, including: When the image complexity is less than the first complexity threshold, the subtitle generation method is determined to be rule-based subtitle generation; When the image complexity is less than the second complexity threshold and greater than or equal to the first complexity threshold, the subtitle generation method is determined to be multimodal large model subtitle generation. When the image complexity exceeds the second complexity threshold, the subtitle generation method is determined to be rule-based subtitle generation combined with multimodal large model subtitle generation.

[0010] Preferably, the method for generating subtitles according to rules is as follows: Construct a rule template, which includes a first subtitle, second subtitle, third subtitle, and fourth subtitle arranged in sequence; Acquire the remote sensing image of the subtitle to be generated, determine the bounding box of the remote sensing image of the subtitle to be generated, and determine the central region and edge region of the remote sensing image of the subtitle to be generated; wherein, the central region is the intersection of the region of 1 / 4 to 3 / 4 of the width and the region of 1 / 4 to 3 / 4 of the height of the remote sensing image of the subtitle to be generated; the edge region is the part of the remote sensing image of the subtitle to be generated excluding the central region. Obtain the first annotation information of the remote sensing image to be subtitled, extract entity objects from the first annotation information, and generate the first subtitle "There is * in the image", where * is an entity object in the first annotation information; Determine the target detection bounding boxes and the target recognition results corresponding to the target detection bounding boxes in the central region of the remote sensing image to be captioned, and generate the second caption "# is in the central region of the image", where # is the target recognition result corresponding to the central region; Determine the target detection bounding boxes and the target recognition results corresponding to the target detection bounding boxes in the edge region of the remote sensing image to be subtitled, and generate the third subtitle "The image edge region has &", where & is the target recognition result corresponding to the center region; The distance of each target detection bounding box in the edge region of the remote sensing image to be subtitled is determined from the image edge. When the distance is less than a second preset threshold, the fourth subtitle "Near the edge with $" is generated, where $ is the target recognition result corresponding to the target detection bounding box whose distance from the image edge is less than the second preset threshold. When the distance of all target detection bounding boxes in the edge region of the remote sensing image to be subtitled is greater than or equal to the second preset threshold, the fourth subtitle is empty. Based on the rule templates for filling in the first, second, third, and fourth subtitles, subtitles for the remote sensing image to be generated are formed; Specifically, for target detection bounding boxes that coexist in the central region and the edge region, a first area covering the central region and a second area covering the edge region are determined. If the first area is greater than or equal to the second area, the target detection bounding box is determined to be a target detection bounding box in the central region; otherwise, the target detection bounding box is determined to be a target detection bounding box in the edge region.

[0011] Preferably, the method for generating subtitles using a multimodal large model is as follows: Construct a first prompt template, a second prompt template, and a third prompt template, wherein: The first prompt template is "Describe this remote sensing image using ϴ"; The second prompt template is "Describe this remote sensing image using ¥ and @"; The third prompt template is: "Where is the '@' symbol that represents the symbol '￥' in this remote sensing image?" When the first annotation information corresponding to the remote sensing image to which the subtitle is to be generated contains only category labels, the first prompt template is used as the prompt template; where ϴ is the category label and ¥ is the entity object category; When the first annotation information corresponding to the remote sensing image for which the subtitle is to be generated determines that the remote sensing image for which the subtitle is to be generated contains one or two target detection bounding boxes, the second prompt template and the third prompt template are used as prompt templates at the same time; where @ represents the target detection bounding box; When the first annotation information corresponding to the remote sensing image for which the subtitle is to be generated determines that the remote sensing image for which the subtitle is to be generated contains more than two target detection bounding boxes, the first prompt template is used as the prompt template. Input the prompt template and the remote sensing image for which the subtitles are to be generated into the multimodal large model. The multimodal large model then generates the subtitles for the remote sensing image for which the subtitles are to be generated.

[0012] Preferably, the method of generating subtitles by combining rule-based subtitle generation with multimodal large model generation is as follows: First, generate subtitles from the remote sensing image using rules, and use these subtitles as the first candidate subtitles; Then, use a multimodal large model to generate subtitles as the second candidate subtitles; The first and second candidate captions are scored for semantic consistency, and the candidate caption with the higher score is used as the caption for the remote sensing image to be generated.

[0013] This invention provides a remote sensing image caption generation system, comprising: Preprocessing module: Configured to acquire multiple open-source multi-source remote sensing image datasets, preprocess each multi-source remote sensing image dataset separately, and merge the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset; The caption generation module is configured to generate first annotation information for each remote sensing image in the merged remote sensing image dataset based on the remote sensing image category, target detection bounding box, and semantic segmentation mask. Based on the first annotation information, the image complexity is determined, and the caption generation method is determined according to the image complexity. The caption generation methods include rule-based caption generation, multimodal large model caption generation, and rule-based caption generation combined with multimodal large model caption generation. The module then generates captions for the remote sensing image based on the caption generation method. Training module: Configured to form an image-text pair for each remote sensing image and its corresponding caption, and all image-text pairs form a remote sensing image-text pairing dataset; train the CLIP model based on the remote sensing image-text pairing dataset to obtain the trained CLIP model.

[0014] The present invention provides a computer-readable storage medium storing a plurality of instructions; the plurality of instructions are used by a processor to load and execute the method as described above.

[0015] The present invention provides an electronic device, characterized in that the electronic device comprises: A processor is used to execute multiple instructions; Memory, used to store multiple instructions; The plurality of instructions are to be stored in the memory and loaded and executed by the processor as described above.

[0016] Beneficial effects: (1) By combining multi-source remote sensing image datasets, the remote sensing image-text dataset of the present invention has significant advantages in terms of scene and category diversity, which enhances the generalization ability of visual language models in remote sensing tasks.

[0017] (2) By using an automated caption generation method, this invention effectively reduces the workload of manual annotation and lowers the cost of constructing a high-quality remote sensing image-text dataset.

[0018] (3) This invention can generate a high-quality remote sensing image-text pairing dataset containing rich semantic information and spatial relationship descriptions. It achieves fine-grained semantic description of remote sensing images, combines rule-based and MLLMs-based caption generation methods, and generates high-quality captions. The generated captions not only contain rich semantic information, but also provide spatial location descriptions of objects, which significantly improves the matching quality of images and texts and enhances the performance of visual language models in remote sensing tasks such as classification, retrieval, and semantic localization.

[0019] (4) Experimental results show that the RSM-CLIP model trained on the remote sensing image and text dataset constructed based on this method has shown significant performance improvement in zero-shot classification, cross-modal retrieval and semantic localization tasks, especially in fine-grained semantic description and spatial relationship expression. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the remote sensing image caption generation method of the present invention; Figure 2 This is a simplified schematic diagram illustrating the generation of subtitles for remote sensing images according to the present invention; Figure 3 A schematic diagram illustrating the construction of the dataset for this invention; Figure 4 This is a schematic diagram illustrating the rule-based subtitle generation method of the present invention. Detailed Implementation

[0021] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] like Figures 1-2 As shown, this invention proposes a method for generating captions for remote sensing images, the method comprising: Step S1: Obtain multiple open-source multi-source remote sensing image datasets, preprocess each multi-source remote sensing image dataset separately, and merge the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset. Step S2: For each remote sensing image in the merged remote sensing image dataset: generate the first annotation information of the remote sensing image based on the remote sensing image category, target detection bounding box, and semantic segmentation mask; determine the image complexity based on the first annotation information; determine the caption generation method based on the image complexity; the caption generation method includes rule-based caption generation, multimodal large model caption generation, and rule-based caption generation combined with multimodal large model caption generation; generate the caption for the remote sensing image based on the caption generation method. Step S3: Each remote sensing image and its corresponding caption form an image-caption pair, and all image-caption pairs form a remote sensing image-text pairing dataset; train the CLIP model based on the remote sensing image-text pairing dataset to obtain the trained CLIP model.

[0023] This invention first constructs a high-quality remote sensing image dataset covering various categories and scenes by combining multi-source data such as UAV images, satellite images, and multispectral composite RGB images. Annotation information for the remote sensing images is generated based on the remote sensing image category, object detection bounding boxes, and semantic segmentation masks, providing fine-grained image annotation. Subsequently, high-quality captions are generated using two caption generation methods: rule-based caption generation accurately describes the category, quantity, and spatial information of objects using custom rules; and multimodal large model caption generation uses an annotation-to-instruction algorithm (A2I) to convert the annotation information of the remote sensing images into input understandable by multimodal large models (MLLMs), generating richer captions. Finally, a fully fine-tuned approach is used to train a visual language model (such as CLIP), and experiments verify the significant performance improvement of the method in remote sensing image classification, retrieval, and semantic localization tasks.

[0024] Further, in step S1: multiple open-source multi-source remote sensing image datasets are acquired, each multi-source remote sensing image dataset is preprocessed, and the preprocessed multi-source remote sensing image datasets are merged as a merged remote sensing image dataset, wherein: Multi-source remote sensing image datasets include multispectral composite image datasets, UAV aerial target detection datasets, satellite semantic segmentation datasets, remote sensing target detection datasets, and remote sensing scene classification datasets; Preprocessing includes: deleting remote sensing images without annotation information from each multi-source remote sensing image dataset; deduplicating remote sensing images by hashing and identifying the URLs corresponding to each remote sensing image; performing semantic alignment on the annotations of each remote sensing image and deleting remote sensing images corresponding to annotations without specific entity information; for remote sensing images with a size exceeding a first preset threshold (e.g., greater than 4,000,000 pixels), dividing them into several non-overlapping sub-images using a sliding window, and treating each sub-image as a remote sensing image; constructing a category dictionary including multiple categories, and mapping the category to which the remote sensing image belongs to the category defined in the category dictionary.

[0025] like Figure 3 As shown, this invention collected 23 datasets and thoroughly cleaned the images and annotations. These datasets include: (1) a multispectral composite image dataset (MSrgb*1): fMoW, (2) four UAV aerial target detection datasets (Drone*4): AU-AIR, CARPK, Stanford Drone, VisDrone, (3) four satellite semantic segmentation datasets (Seg*4): iSAID, LoveDA, Potsdam, Vaihingen, (4) six remote sensing target detection datasets (DET*6): DIOR, DOTA, HRRSD, HRSC, LEVIR, RSOD, and (5) eight remote sensing scene classification datasets (CLS*8): NWPU-RESISC45, AID, RSI-CB128, RSI-CB256, WHURS19, OPTIMAL-31, MLRSNet, EuroSAT.

[0026] After conversion using the M2B algorithm, Seg*4 and DET*6 are combined to form DET-10. This invention collects Drone*4, Seg*4, and DET*6 used in RemoteCLIP to compare data construction methods. CLS*8 and MSrgb*1 are collected to further increase data diversity. For the fMoW dataset, only samples from the validation set were selected because GeoRSCLIP has already generated captions for the training set, thus avoiding data redundancy.

[0027] For preprocessing, unlabeled images were first removed from each dataset. For images that were too large (greater than 4,000,000 pixels), a sliding window method was used to divide them into several non-overlapping small image patches. To prevent data leakage, strict deduplication methods using p-hash and URLs were employed. Ultimately, approximately one-fifth of the samples were removed. Noise reduction was also performed on the annotations. For example, annotations labeled "ignored region" and "other" in the VisDrone dataset were removed. The wording of the original annotations was adjusted where necessary. For example, the annotation for "Human" was changed to "person".

[0028] In step S2, the first annotation information of the remote sensing image is generated based on the remote sensing image category, target detection bounding box, and semantic segmentation mask, including: Generate category labels based on remote sensing image categories; determine the spatial location information of bounding boxes based on target detection bounding boxes, including entity objects, entity object categories, and quantities within the bounding boxes; determine the contour of semantic segmentation regions based on semantic segmentation masks; The first annotation information is generated by integrating category labels, bounding box spatial location information, semantic segmentation region outline, entity objects inside the bounding box, entity object category and quantity; wherein, the bounding box spatial location information includes the coordinates of the bounding box center point and the bounding box size; the first annotation information includes category labels, entity objects inside the bounding box, entity object category and quantity, bounding box spatial location information and semantic segmentation boundary.

[0029] In step S2, the image complexity is determined based on the first annotation information, and the subtitle generation method is determined according to the image complexity, including: The image complexity is determined based on the first annotation information. The formula for calculating the image complexity is as follows:

[0030] Where A is the image complexity. Nobj Represents the number of bounding boxes detected for the target. Ncls Represents the number of entity object categories; Roverlap This represents the average overlap ratio of the bounding boxes detected by the target. Hentropy Distribution entropy, representing the category of entity objects, is used to measure the semantic diversity of an image; Dspatial Represents spatial density, used to measure the uniformity of the distribution of physical objects in remote sensing images; α , β , γ , δ , ζ The weighting parameters are set through cross-validation or experimental experience. The subtitle generation method is determined based on image complexity, including: When the image complexity is less than the first complexity threshold, the subtitle generation method is determined to be rule-based subtitle generation; When the image complexity is less than the second complexity threshold and greater than or equal to the first complexity threshold, the subtitle generation method is determined to be multimodal large model subtitle generation. When the image complexity exceeds the second complexity threshold, the subtitle generation method is determined to be rule-based subtitle generation combined with multimodal large model subtitle generation.

[0031] The formula for calculating image complexity in this invention can also be adjusted as needed, for example, as follows: .

[0032] In the present invention, when A < T1, the scenario is simple, with few targets and a single relationship. The method of generating captions according to rules (A2C algorithm) is adopted to ensure the coverage of all entities and reduce ambiguity. When T1 ≤ A < T2, the number of targets in the scenario is relatively large but the category relationships are clear. The method of generating captions using a multimodal large model (A2I algorithm) is adopted to generate more natural and delicate descriptions by utilizing its context modeling ability. When A ≥ T2, the scenario is extremely complex, with many categories and dense relationships. A hybrid generation method combining rule-based caption generation and multimodal large model-based caption generation is adopted. T1 is the first complexity threshold, used to distinguish between simple and moderately complex scenarios; T2 is the second complexity threshold, used to distinguish between moderately complex and highly complex scenarios. The specific values of T1 and T2 can be set based on the experimental results of different datasets to ensure the rationality and adaptability of the generation method. Through the above mechanism, the present invention introduces dynamic decision-making based on quantitative indicators during the generation process, enhancing the flexibility and accuracy of caption generation and solving the problem that a single generation method cannot balance efficiency and semantic naturalness.

[0033] Furthermore, as Figure 4 shown, the method of generating captions according to rules is as follows: Construct a rule template, which includes a first caption, a second caption, a third caption, and a fourth caption combined in sequence; Obtain the remote sensing image for which the caption is to be generated, determine the bounding box of the remote sensing image for which the caption is to be generated, and determine the central area and the edge area of the remote sensing image for which the caption is to be generated; wherein, the central area is the intersection of the area from 1 / 4 to 3 / 4 of the width and the area from 1 / 4 to 3 / 4 of the height of the remote sensing image for which the caption is to be generated; the edge area is the part of the remote sensing image for which the caption is to be generated excluding the central area; Obtain the first annotation information of the remote sensing image for which the caption is to be generated, extract the entity objects from the first annotation information, and generate the first caption "There is * in the image", where * is the entity object in the first annotation information; Determine the object detection bounding boxes and the corresponding object recognition results in the central area of the remote sensing image for which the caption is to be generated, and generate the second caption "There is # in the central area of the image", where # is the object recognition result corresponding to the central area; Determine the object detection bounding boxes and the corresponding object recognition results in the edge area of the remote sensing image for which the caption is to be generated, and generate the third caption "There is & in the edge area of the image", where & is the object recognition result corresponding to the central area; The distance of each target detection bounding box in the edge region of the remote sensing image to be subtitled is determined from the image edge. When the distance is less than a second preset threshold, the fourth subtitle "Near the edge with $" is generated, where $ is the target recognition result corresponding to the target detection bounding box whose distance from the image edge is less than the second preset threshold. When the distance of all target detection bounding boxes in the edge region of the remote sensing image to be subtitled is greater than or equal to the second preset threshold, the fourth subtitle is empty. Based on the rule templates for filling in the first, second, third, and fourth subtitles, subtitles for the remote sensing image to be generated are formed; Specifically, for target detection bounding boxes that coexist in the central region and the edge region, a first area covering the central region and a second area covering the edge region are determined. If the first area is greater than or equal to the second area, the target detection bounding box is determined to be a target detection bounding box in the central region; otherwise, the target detection bounding box is determined to be a target detection bounding box in the edge region.

[0034] Furthermore, the rule template also includes a croppable fifth subtitle, which determines the entity object category corresponding to all target detection bounding boxes of the remote sensing image to be subtitled. When the number of times the same entity object category appears is greater than a third preset threshold, the fifth subtitle "There are multiple ¥ in the image" is generated, where ¥ is the entity object category. Based on the rule templates for filling in the first, second, third, fourth, and fifth subtitles, subtitles are generated for the remote sensing image to be subtitled.

[0035] The method for generating subtitles according to rules in this invention is referred to as the A2C algorithm. It can ensure that the subtitles contain all labeled objects, reduce the risk of category ambiguity, and provide a more complete description.

[0036] Furthermore, the method for generating subtitles using a multimodal large model is as follows: Construct a first prompt template, a second prompt template, and a third prompt template, wherein: The first prompt template is "Describe this remote sensing image using ϴ"; The second prompt template is "Describe this remote sensing image using ¥ and @"; The third prompt template is: "Where is the '@' symbol that represents the symbol '￥' in this remote sensing image?" When the first annotation information corresponding to the remote sensing image to which the subtitle is to be generated contains only category labels, the first prompt template is used as the prompt template; where ϴ is the category label and ¥ is the entity object category; When the first annotation information corresponding to the remote sensing image for which the subtitle is to be generated determines that the remote sensing image for which the subtitle is to be generated contains one or two target detection bounding boxes, the second prompt template and the third prompt template are used as prompt templates at the same time; where @ represents the target detection bounding box; When the first annotation information corresponding to the remote sensing image for which the subtitle is to be generated determines that the remote sensing image for which the subtitle is to be generated contains more than two target detection bounding boxes, the first prompt template is used as the prompt template. Input the prompt template and the remote sensing image for which the subtitles are to be generated into the multimodal large model. The multimodal large model then generates the subtitles for the remote sensing image for which the subtitles are to be generated.

[0037] Furthermore, the method of generating subtitles by combining rule-based subtitle generation with multimodal large model generation is as follows: First, generate subtitles from the remote sensing image using rules, and use these subtitles as the first candidate subtitles; Then, use a multimodal large model to generate subtitles as the second candidate subtitles; The first and second candidate captions are scored for semantic consistency, and the candidate caption with the higher score is used as the caption for the remote sensing image to be generated.

[0038] In this invention, multimodal large language models (MLLMs) can perceive annotation information and generate text descriptions for images, such as chatGPT-4V, BLIP-2, and Lava. However, chatGPT-4V is closed-source and has a high cost of use. Other MLLMs, such as BLIP-2 and Lava, lack visual alignment capabilities. Models like Kosmos-2 and Qwen-VL are open-source and can utilize text instructions to perceive and associate labeled bounding boxes with the generated captions. They provide more accurate, informative, and comprehensive caption descriptions for images. In this process, any model with visual grounding capabilities can be used for multimodal large model caption generation. Here, Kosmos-2 is used as an example to introduce multimodal large model caption generation.

[0039] When using large multimodal models, the instruction format has a significant impact on the performance of the large multimodal models. To obtain better results, this invention explored the effects of different instruction templates before formal experiments. 1000 images were randomly selected from the entire dataset to test the effects of 10 different instruction templates for Kosmos-2. After careful design and extensive experiments, this invention adopted the following three instruction templates: ①Describe this image with [class] in detail ②Describe this image with [class + bbox] in detail ③Where is / are the [class + bbox]? Answer: For images with only category-level annotations, use template ① to generate an accurate and comprehensive description. For images containing one or two bounding boxes, use both templates ② and ③ to obtain accurate location information and a comprehensive image description. For images with more than two bounding boxes, use template ① to achieve a comprehensive image description.

[0040] Since Kosmos-2 cannot directly accept bounding boxes or semantic segmentation masks as input, the method used in this invention for generating captions using a multimodal large model is denoted as the A2I (Annotation to Instruction) algorithm. This algorithm can automatically convert categories, object detection bounding boxes, and semantic segmentation masks into instructions that Kosmos-2 can perceive. These instructions, along with the image, are input into the Kosmos-2 model to generate the corresponding captions.

[0041] This invention provides specific embodiments. The annotation file for each dataset is read using `json.load`, filtering records with annotation lengths of 0. If the image width × height > 4000000px, a sliding window (1024 × 1024 window, stride 768) is used for segmentation to ensure that each image retains a complete target bounding box. A p-hash is calculated using `imagehash.phash` with a threshold of 5 bits; images with the same hash are retained based on their URLs. For each semantic segmentation image in Seg*4, connected components are extracted using `skimage.measure.regionprops`; the minimum bounding rectangle (xmin, ymin, xmax, ymax) is calculated for each connected component. The obtained boxes are merged with the existing target detection boxes in DET*6, and redundant boxes with IoU > 0.8 are removed to form DET-10. Using a mapping table between WordNet and OpenImages, 156 original categories from 23 datasets were mapped to 97 unified labels (e.g., "Human" → "person", "Vehicle" → "car / truck"), and saved as category_map.json. Experimental results show that the captions generated by A2C in this invention improve accuracy by 12.3% and 10.8% on BLEU-4 and ROUGE-L, respectively; the captions generated by A2I improve accuracy by 15.6% on CIDEr, and significantly improve the accuracy of spatial relationship description (from 71% to 89%). RSM-CLIP trained using this system improves Top-1 accuracy by 6.4% in Zero-Shot classification and Recall@1 across modalities by 8.1%.

[0042] This invention addresses the problems of insufficient quality in existing large-scale remote sensing image text pairing datasets, high cost and low efficiency of manual annotation, and lack of contextual information in rule-based captions. The captions generated by this invention contain rich semantic and spatial relationship information. Datasets constructed based on this method can significantly improve the performance of visual language models in remote sensing image classification, retrieval, and semantic localization tasks, while enhancing model generalization ability and greatly reducing annotation costs, providing an effective solution for automatically generating high-quality captions for large-scale remote sensing images.

[0043] The present invention also provides a remote sensing image caption generation system, the system comprising: Preprocessing module: Configured to acquire multiple open-source multi-source remote sensing image datasets, preprocess each multi-source remote sensing image dataset separately, and merge the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset; The caption generation module is configured to generate first annotation information for each remote sensing image in the merged remote sensing image dataset based on the remote sensing image category, target detection bounding box, and semantic segmentation mask. Based on the first annotation information, the image complexity is determined, and the caption generation method is determined according to the image complexity. The caption generation methods include rule-based caption generation, multimodal large model caption generation, and rule-based caption generation combined with multimodal large model caption generation. The module then generates captions for the remote sensing image based on the caption generation method. Training module: Configured to form an image-text pair for each remote sensing image and its corresponding caption, and all image-text pairs form a remote sensing image-text pairing dataset; train the CLIP model based on the remote sensing image-text pairing dataset to obtain the trained CLIP model.

[0044] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A method for generating subtitles for remote sensing images, characterized in that, include: Step S1: Obtain multiple open-source multi-source remote sensing image datasets, preprocess each multi-source remote sensing image dataset separately, and merge the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset. Step S2: For each remote sensing image in the merged remote sensing image dataset: generate the first annotation information of the remote sensing image based on the remote sensing image category, target detection bounding box and semantic segmentation mask, determine the image complexity based on the first annotation information, and determine the subtitle generation method according to the image complexity. The subtitle generation method includes generating subtitles according to rules, generating subtitles using a multimodal large model, and generating subtitles by combining rule-based subtitles with a multimodal large model. The captions for this remote sensing image are generated based on the caption generation method; Step S3: Each remote sensing image and its corresponding caption form an image-caption pair, and all image-caption pairs form a remote sensing image-text pairing dataset; train the CLIP model based on the remote sensing image-text pairing dataset to obtain the trained CLIP model.

2. The method as described in claim 1, characterized in that, Step S1: Obtain multiple open-source multi-source remote sensing image datasets, preprocess each multi-source remote sensing image dataset separately, and merge the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset, wherein: Multi-source remote sensing image datasets include multispectral composite image datasets, UAV aerial target detection datasets, satellite semantic segmentation datasets, remote sensing target detection datasets, and remote sensing scene classification datasets; Preprocessing includes: deleting remote sensing images without annotation information from each multi-source remote sensing image dataset; deduplicating remote sensing images by hashing and identifying the URLs corresponding to each remote sensing image; performing semantic alignment on the annotations of each remote sensing image and deleting remote sensing images corresponding to annotations without specific entity information; for remote sensing images whose size exceeds a first preset threshold, using a sliding window to divide them into several non-overlapping sub-images, and treating each sub-image as a remote sensing image; and constructing a category dictionary including multiple categories, mapping the category to which the remote sensing image belongs to the category defined in the category dictionary.

3. The method according to any one of claims 1-2, characterized in that, In step S2, the first annotation information of the remote sensing image is generated based on the remote sensing image category, target detection bounding box, and semantic segmentation mask, including: Generate category labels based on remote sensing image categories; determine the spatial location information of bounding boxes based on target detection bounding boxes, including entity objects, entity object categories, and quantities within the bounding boxes; determine the contour of semantic segmentation regions based on semantic segmentation masks; The first annotation information is generated by integrating category labels, bounding box spatial location information, semantic segmentation region outline, entity objects inside the bounding box, entity object category and quantity; wherein, the bounding box spatial location information includes the coordinates of the bounding box center point and the bounding box size; the first annotation information includes category labels, entity objects inside the bounding box, entity object category and quantity, bounding box spatial location information and semantic segmentation boundary.

4. The method as described in claim 3, characterized in that, In step S2, the image complexity is determined based on the first annotation information, and the subtitle generation method is determined according to the image complexity, including: The image complexity is determined based on the first annotation information. The formula for calculating the image complexity is as follows: Where A is the image complexity. Nobj Represents the number of bounding boxes detected for the target. Ncls Represents the number of entity object categories; Roverlap This represents the average overlap ratio of the bounding boxes detected by the target. Hentropy Distribution entropy, representing the category of entity objects, is used to measure the semantic diversity of an image; Dspatial Represents spatial density, used to measure the uniformity of the distribution of physical objects in remote sensing images; α , β , γ , δ , ζ The weighting parameters are set through cross-validation or experimental experience. The subtitle generation method is determined based on image complexity, including: When the image complexity is less than the first complexity threshold, the subtitle generation method is determined to be rule-based subtitle generation; When the image complexity is less than the second complexity threshold and greater than or equal to the first complexity threshold, the subtitle generation method is determined to be multimodal large model subtitle generation. When the image complexity exceeds the second complexity threshold, the subtitle generation method is determined to be rule-based subtitle generation combined with multimodal large model subtitle generation.

5. The method as described in claim 4, characterized in that, The method for generating subtitles according to rules is as follows: Construct a rule template, which includes a first subtitle, second subtitle, third subtitle, and fourth subtitle arranged in sequence; Acquire the remote sensing image of the subtitle to be generated, determine the bounding box of the remote sensing image of the subtitle to be generated, and determine the central region and edge region of the remote sensing image of the subtitle to be generated; wherein, the central region is the intersection of the region of 1 / 4 to 3 / 4 of the width and the region of 1 / 4 to 3 / 4 of the height of the remote sensing image of the subtitle to be generated; the edge region is the part of the remote sensing image of the subtitle to be generated excluding the central region. Obtain the first annotation information of the remote sensing image to be subtitled, extract entity objects from the first annotation information, and generate the first subtitle "There is a * in the image", where * is an entity object in the first annotation information; Determine the target detection bounding boxes and the target recognition results corresponding to the target detection bounding boxes in the central region of the remote sensing image to be captioned, and generate the second caption "# is in the central region of the image", where # is the target recognition result corresponding to the central region; Determine the target detection bounding boxes and the target recognition results corresponding to the target detection bounding boxes in the edge region of the remote sensing image to be subtitled, and generate the third subtitle "The image edge region has &", where & is the target recognition result corresponding to the center region; The distance of each target detection bounding box in the edge region of the remote sensing image to be subtitled is determined from the image edge. When the distance is less than a second preset threshold, the fourth subtitle "Near the edge there is $" is generated, where $ is the target recognition result corresponding to the target detection bounding box whose distance from the image edge is less than the second preset threshold. When the distance of all target detection bounding boxes in the edge region of the remote sensing image to be subtitled is greater than or equal to the second preset threshold, the fourth subtitle is empty. Based on the rule templates for filling in the first, second, third, and fourth subtitles, subtitles for the remote sensing image to be generated are formed; Specifically, for target detection bounding boxes that coexist in the central region and the edge region, a first area covering the central region and a second area covering the edge region are determined. If the first area is greater than or equal to the second area, the target detection bounding box is determined to be a target detection bounding box in the central region; otherwise, the target detection bounding box is determined to be a target detection bounding box in the edge region.

6. The method as described in claim 5, characterized in that, The method for generating subtitles using a multimodal large model is as follows: Construct a first prompt template, a second prompt template, and a third prompt template, wherein: The first prompt template is "Describe this remote sensing image using ϴ"; The second prompt template is "Describe this remote sensing image using ¥ and @"; The third prompt template is: "Where is the '@' symbol that represents the symbol '￥' in this remote sensing image?" When the first annotation information corresponding to the remote sensing image to which the subtitle is to be generated contains only category labels, the first prompt template is used as the prompt template; where ϴ is the category label and ¥ is the entity object category; When the first annotation information corresponding to the remote sensing image for which the subtitle is to be generated determines that the remote sensing image for which the subtitle is to be generated contains one or two target detection bounding boxes, the second prompt template and the third prompt template are used as prompt templates at the same time; where @ represents the target detection bounding box; When the first annotation information corresponding to the remote sensing image for which the subtitle is to be generated determines that the remote sensing image for which the subtitle is to be generated contains more than two target detection bounding boxes, the first prompt template is used as the prompt template. Input the prompt template and the remote sensing image for which the subtitles are to be generated into the multimodal large model. The multimodal large model then generates the subtitles for the remote sensing image for which the subtitles are to be generated.

7. The method as described in claim 6, characterized in that, The method of generating subtitles by combining rule-based subtitle generation with multimodal large model generation is as follows: First, generate subtitles from the remote sensing image using rules, and use these subtitles as the first candidate subtitles; Then, use a multimodal large model to generate subtitles as the second candidate subtitles; The first and second candidate captions are scored for semantic consistency, and the candidate caption with the higher score is used as the caption for the remote sensing image to be generated.

8. A remote sensing image caption generation system, characterized in that, include: Preprocessing module: Configured to acquire multiple open-source multi-source remote sensing image datasets, preprocess each multi-source remote sensing image dataset separately, and merge the preprocessed multi-source remote sensing image datasets as a merged remote sensing image dataset; The caption generation module is configured to generate first annotation information for each remote sensing image in the merged remote sensing image dataset based on the remote sensing image category, target detection bounding box, and semantic segmentation mask. Based on the first annotation information, the image complexity is determined, and the caption generation method is determined according to the image complexity. The caption generation methods include rule-based caption generation, multimodal large model caption generation, and rule-based caption generation combined with multimodal large model caption generation. The captions for this remote sensing image are generated based on the caption generation method; Training module: Configured to form an image-text pair for each remote sensing image and its corresponding caption, and all image-text pairs form a remote sensing image-text pairing dataset; train the CLIP model based on the remote sensing image-text pairing dataset to obtain the trained CLIP model.

9. A computer-readable storage medium, characterized in that, The storage medium stores a plurality of instructions; the plurality of instructions are loaded by a processor and executed as described in any one of claims 1-7.

10. An electronic device, characterized in that, The electronic device includes: A processor is used to execute multiple instructions; Memory, used to store multiple instructions; The plurality of instructions are to be stored in the memory and loaded by the processor and executed as described in any one of claims 1-7.

Images