Semantic labeling method for remote sensing generated images and electronic device
By combining a general segmentation model, a multimodal language model, and an expert model, the problem of high efficiency, accuracy, and low cost in semantic annotation of remote sensing generated images is solved, achieving efficient automated annotation and high-precision semantic annotation of remote sensing generated images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 启元实验室
- Filing Date
- 2026-01-19
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies cannot efficiently, accurately, and cost-effectively acquire category semantic label information of remotely generated images, making it difficult to use generated scene images for algorithm testing and model training.
By combining a general segmentation model and a multimodal language model with an expert model, semantic labels for remotely sensed images are automatically labeled through multi-model fusion and manual calibration. The general segmentation model is used for initial segmentation, the multimodal language model is used for semantic labeling, the expert model is used for calibration, and finally the labeling accuracy is improved through fusion and manual review.
It achieves efficient and automated annotation of remotely sensed images, reduces annotation costs, improves the accuracy and consistency of semantic annotation, and meets the needs of high-precision algorithm testing.
Smart Images
Figure CN121527765B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a semantic annotation method and electronic device for remotely generated images. Background Technology
[0002] With the rapid development of science and technology in recent years, the integration of artificial intelligence and information technology has propelled various algorithms into the era of informatization and intelligence. In this process, intelligent algorithms are playing an increasingly crucial role in many key areas, especially in remote sensing image analysis and scene understanding. The performance verification, continuous optimization, and final application of algorithms highly depend on rich, diverse, and realistic test scenarios. In recent years, remote sensing scene generation technology has been able to efficiently generate diverse and highly complex virtual scenes, including towns, forests, parks, and rivers, providing a potentially vast scene library for algorithm testing. However, these automatically generated scene images contain a large number of dense targets and complex spatial relationships, lacking accurate category semantic label information, which restricts the practical application of generated scenes in algorithm testing and model training. Therefore, it is urgent to efficiently, accurately, and cost-effectively obtain category semantic label information from remote sensing generated images and generate category-related segmentation masks. Summary of the Invention
[0003] Based on the above analysis, the embodiments of the present invention aim to provide a semantic annotation method and electronic device for remotely sensed generated images, in order to solve the problem that existing methods cannot efficiently, accurately, and cost-effectively obtain category semantic label information of remotely sensed generated images and generate category-related segmentation masks.
[0004] On one hand, embodiments of the present invention provide a semantic annotation method for remotely sensed generated images. The method includes: determining a remotely sensed generated image to be annotated; obtaining a first mask result containing a first category semantic label for each pixel of the remotely sensed generated image using a general segmentation large model and a first multimodal large language model; obtaining a second mask result containing a second category semantic label for each pixel of the remotely sensed generated image using a first expert model and a second multimodal large language model; obtaining a third mask result containing a third category semantic label for each pixel of the remotely sensed generated image using a second expert model and a third multimodal large language model; fusing the first mask result, the second mask result, and the third mask result to determine a fourth category semantic label for the pixels of the remotely sensed generated image; and obtaining a category-related segmentation mask for the remotely sensed generated image based on the fourth category semantic label.
[0005] A further improvement to the above method, the step of obtaining the first mask result containing the first category semantic label for each pixel of the remotely sensed image using a general segmentation model and a first multimodal large language model, includes: inputting the remotely sensed image into the general segmentation model to obtain a category-independent segmentation mask; based on the category-independent segmentation mask, using the first multimodal large language model to output the first category semantic label and the first confidence score of the mask region; retaining the first category semantic labels corresponding to the first confidence scores ranked high to low according to a predetermined proportion, and setting the first category semantic labels corresponding to the other first confidence scores ranked low as background, to obtain the first mask result.
[0006] A further improvement to the above method involves obtaining a second mask result containing a second category semantic label for each pixel of the remotely sensed image using a first expert model and a second multimodal large language model. This includes: inputting the remotely sensed image into the first expert model for inference to obtain a fourth mask result containing a second category semantic label for each pixel of the remotely sensed image; inputting image regions of the same category into the second multimodal large language model based on the fourth mask result to output a fifth category semantic label and a second confidence score for the image regions; and updating the category of the second category semantic label based on the fifth category semantic label if the second confidence score is greater than a preset threshold and the fifth category semantic label is inconsistent with the second category semantic label corresponding to the image region.
[0007] A further improvement to the above method involves obtaining a third mask result containing a third category semantic label for each pixel of the remotely sensed image using a second expert model and a third multimodal large language model. This includes: inputting the remotely sensed image into the second expert model for inference to obtain a fifth mask result containing a third category semantic label for each pixel of the remotely sensed image; inputting image regions of the same category into the third multimodal large language model based on the fifth mask result to output a sixth category semantic label and a third confidence score for the pixels of the image region; and updating the category of the third category semantic label based on the sixth category semantic label if the third confidence score is greater than a preset threshold and the sixth category semantic label is inconsistent with the third category semantic label corresponding to the image region.
[0008] Based on a further improvement of the above method, the step of fusing the first mask result, the second mask result, and the third mask result to determine the fourth category semantic label of the pixels in the remote sensing generated image includes: in response to the second category semantic label and the third category semantic label being consistent, determining the fourth category semantic label of the pixels in the remote sensing generated image based on the second category semantic label or the third category semantic label; in response to the second category semantic label and the third category semantic label being inconsistent, but the first category semantic label and the second category semantic label being consistent, determining the fourth category semantic label of the pixels in the remote sensing generated image based on the first category semantic label or the second category semantic label; in response to the second category semantic label and the third category semantic label being inconsistent, but the first category semantic label and the third category semantic label being consistent, determining the fourth category semantic label of the pixels in the remote sensing generated image based on the first category semantic label or the third category semantic label; and in response to the first category semantic label, the second category semantic label, and the third category semantic label all being inconsistent, including the corresponding pixels in the area to be reviewed.
[0009] Based on a further improvement of the above method, after determining the fourth category semantic label of the pixels in the remotely sensed image, the method further includes: receiving manually labeled data for the region to be reviewed, the over-segmented region, the under-segmented region, and the incorrectly labeled region, and correcting or supplementing the fourth category semantic label of the pixels in the remotely sensed image based on the manually labeled data.
[0010] Based on a further improvement of the above method, determining the current remote sensing generated image to be labeled includes: sampling images in rounds from the generated remote sensing generated image library; wherein, for each round of image sampling, a preset number of remote sensing generated images to be labeled are extracted according to different scene categories; and the current remote sensing generated image to be labeled is determined from the preset number of remote sensing generated images to be labeled in a preset order.
[0011] Based on a further improvement of the above method, before each round of image sampling, the method further includes: obtaining a first training sample; wherein the first training sample includes a remote sensing public dataset and reconstructed category semantic label annotation data; obtaining a second training sample; wherein the second training sample includes remote sensing generated images obtained from the image sampling in the previous round and the fourth category semantic labels; and training the first expert model and the second expert model based on the first training sample and / or the second training sample, respectively.
[0012] Based on a further improvement of the above method, the method further includes: for each round of image sampling, ensuring that the preset number of remote sensing generated images to be labeled do not overlap with historical sampling results by using a random seed.
[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the semantic annotation method for remote sensing generated images as described above.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semantic annotation method for remotely generated images as described above.
[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a semantic annotation method for remotely sensed generated images as described above.
[0016] The semantic annotation method and electronic device for remotely sensed generated images provided by this invention obtain a first mask result by using a general segmentation model and a first multimodal large language model, obtain a second mask result by using a first expert model and a second multimodal large language model, obtain a third mask result by using a second expert model and a third multimodal large language model, and fuse the first mask result, the second mask result, and the third mask result to obtain the category semantic labels of the pixels of the remotely sensed generated image. This achieves efficient and automated annotation of remotely sensed generated images, reduces annotation costs, and fully combines the high reliability of expert models in task-specific scenarios with the generalization ability of general segmentation models in open domains, as well as the zero-shot learning capability of multimodal large language models, to achieve more robust multi-model fusion and result consistency control, thereby improving the accuracy of semantic annotation of remotely sensed generated images.
[0017] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0018] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0019] Figure 1 This is one of the flowcharts illustrating the semantic annotation method for remotely sensed generated images provided by the present invention.
[0020] Figure 2 This is a schematic diagram of the self-training process in the semantic annotation method for remotely sensed generated images provided by the present invention.
[0021] Figure 3 This is the second flowchart illustrating the semantic annotation method for remotely sensed generated images provided by this invention.
[0022] Figure 4 This is a schematic diagram of the semantic annotation device for remote sensing generated images provided by the present invention.
[0023] Figure 5 A schematic diagram of the physical structure of an electronic device is provided. Detailed Implementation
[0024] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0025] The semantic annotation of remotely sensed images mainly includes the following methods:
[0026] First, it relies entirely on manual annotation by professionals, which is inefficient and costly, and makes it difficult to scale up applications when faced with massive amounts of generated data.
[0027] Secondly, traditional image processing algorithms, such as edge detection, region growing, or early deep learning models based on supervised learning, are employed. Traditional algorithms exhibit poor robustness when processing generated images containing artifacts, noise, and non-realistic texture features, struggling to accurately capture complex boundaries and high-level semantic information, resulting in insufficient segmentation accuracy. While supervised learning models hold great potential, their training and performance are highly dependent on large-scale, high-quality labeled data, which is precisely what generated scenes lack, creating a cyclical dependency. Even attempts to use object detection combined with simple rules for weak labeling struggle to cover all elements in the scene, especially the background and continuous geographic entities, and cannot provide accurate pixel-level boundary information, failing to meet the requirements of high-precision algorithm testing.
[0028] To address the key shortcomings of the aforementioned technologies in the field of semantic annotation of remote sensing generated scenes, this invention proposes a remote sensing generated image annotation method based on a large model, aiming to efficiently, accurately, and cost-effectively acquire category semantic label information for large-scale complex remote sensing generated images.
[0029] Figure 1 This is one of the flowcharts illustrating the semantic annotation method for remotely sensed generated images provided by this invention. For example... Figure 1 As shown, the method includes:
[0030] Step S1: Determine the remote sensing image to be annotated.
[0031] Generative remote sensing images are remote sensing scene images generated using remote sensing scene generation technology, also known as generative remote sensing images. When semantically annotating generative remote sensing images, each image needs to be semantically annotated sequentially. The image to be annotated can be determined from a generative remote sensing image database.
[0032] Semantic annotation of remote sensing generated images refers to assigning geographically meaningful category semantic labels (such as "buildings," "roads," "vegetation," "water bodies," "farmland," "bare land," etc.) to each pixel / region in remote sensing scene images (including 2D imagery) generated by generative AI (such as GANs and diffusion models). The core objective is to endow the generated remote sensing images with interpretable geographic semantic information to meet the needs of subsequent applications such as GIS analysis, scene understanding, and model training. This invention can achieve pixel-level semantic annotation of remote sensing generated images.
[0033] Step S2: Using the general segmentation model and the first multimodal language model, obtain the first mask result containing the first category semantic label for each pixel of the remote sensing generated image.
[0034] First, a general segmentation model (such as the Segment Anything Model, SAM) is used to process the remotely generated image to be labeled, resulting in a category-independent segmentation mask. The category-independent segmentation mask only divides regions based on the recognition results of features, but does not provide semantic annotation information. A multimodal large language model possesses zero-shot learning capabilities. Based on the category-independent segmentation mask, a first multimodal large language model is used to semantically annotate the mask, resulting in a first mask containing the semantic labels of the first category for each pixel of the remotely generated image. Based on the target category to be classified in the remotely generated image, questions can be posed to the first multimodal large language model to obtain the first mask containing the semantic labels of the first category for each pixel of the remotely generated image. Target categories include, for example, roads, buildings, vegetation, vehicles, water, grassland, bare soil, domed buildings, playgrounds, plazas, and piers; other categories are background.
[0035] Step S3: Use the first expert model and the second multimodal large language model to obtain the second mask result containing the second category semantic label for each pixel of the remote sensing generated image, and use the second expert model and the third multimodal large language model to obtain the third mask result containing the third category semantic label for each pixel of the remote sensing generated image.
[0036] The first and second expert models employ heterogeneous semantic segmentation architectures customized for remote sensing scenarios. Each model performs inference on the remote sensing image to be labeled. The first expert model extracts the corresponding image region according to category based on its inference result and inputs it into the second multimodal large language model. After category update and calibration by the second multimodal large language model, a second mask result containing the second category semantic label is obtained. The second expert model extracts the corresponding image region according to category based on its inference result and inputs it into the third multimodal large language model. After category update and calibration by the third multimodal large language model, a third mask result containing the third category semantic label is obtained.
[0037] Step S4: Merge the first mask result, the second mask result, and the third mask result to determine the fourth category semantic label of the pixels in the remote sensing generated image, and obtain the category-related segmentation mask of the remote sensing generated image based on the fourth category semantic label.
[0038] The first, second, and third mask results all include mask information containing category semantic labels for each pixel of the remotely sensed image. By fusing the first, second, and third mask results obtained based on different paths, the fourth category semantic label of the pixels in the remotely sensed image is obtained, which is the semantic annotation data of the pixels in the remotely sensed image. Based on the fourth category semantic label of the pixels in the remotely sensed image, the category-related segmentation mask of the remotely sensed image is obtained.
[0039] It should be noted that the first, second, and third multimodal language models can be the same multimodal language model or different multimodal language models. For simplicity, the first, second, and third multimodal language models can be set to the same multimodal language model, such as GPT4o mini.
[0040] The semantic annotation method for remotely sensed generated images provided by this invention obtains a first mask result by using a general segmentation model and a first multimodal large language model, a second mask result by using a first expert model and a second multimodal large language model, and a third mask result by using a second expert model and a third multimodal large language model. The first, second, and third mask results are then fused to obtain the category semantic labels for the pixels of the remotely sensed generated image. This achieves efficient and automated annotation of remotely sensed generated images, reduces annotation costs, and fully combines the high reliability of expert models in task-specific scenarios with the generalization ability of general segmentation models in open domains, as well as the zero-shot learning capability of multimodal large language models. This results in more robust multi-model fusion and result consistency control, improving the accuracy of semantic annotation of remotely sensed generated images.
[0041] According to a semantic annotation method for remotely sensed generated images provided by the present invention, the method for obtaining a first mask result containing a first category semantic label for each pixel of the remotely sensed generated image using a general segmentation model and a first multimodal large language model includes: inputting the remotely sensed generated image into the general segmentation model to obtain a category-independent segmentation mask; based on the category-independent segmentation mask, outputting a first category semantic label and a first confidence score for the masked region using the first multimodal large language model; retaining the first category semantic labels corresponding to the first confidence scores with the highest ranking from high to low, and setting the first category semantic labels corresponding to the other first confidence scores with the lowest ranking as background, thereby obtaining the first mask result.
[0042] The remote sensing image to be labeled is input into a general segmentation model. Taking SAM as an example, the "everything" mode is used to perform segmentation based on preset grid point prompts, resulting in a category-independent segmentation mask. Subsequently, a first multimodal large language model is used to perform rule filtering on each mask region. That is, based on the query, the first multimodal large language model outputs the first category semantic label and the first confidence score of each segmented region mask. Only the categories of mask regions with higher confidence rankings (such as the top 70% of mask regions) are retained, and low-confidence regions are set as background. Finally, a first mask result containing the first category semantic label is obtained.
[0043] When posing questions to the first multimodal large model, for each mask region, the semantic label of the first category is obtained through prompts such as "This is a mask of a remotely sensed image. Please determine and output which category it belongs to and its confidence level, in the form of <category: xx, confidence level: xx>". For example, a certain mask is identified as "building" with a confidence level of 0.85. Finally, the masks are integrated and merged into a single mask, where each pixel position is composed of any number from 0, 1, 2...N, and each number represents a category ID.
[0044] The semantic annotation method for remotely sensed generated images provided by this invention is based on category-independent segmentation masks. It uses a first multimodal large language model to output the first category semantic label and the first confidence score of the masked region, retains the first category semantic label of the high confidence region, and sets the low confidence region as background, thereby further improving the accuracy of semantic annotation of remotely sensed generated images.
[0045] According to a semantic annotation method for remotely sensed generated images provided by the present invention, the step of obtaining a second mask result containing a second category semantic label for each pixel of the remotely sensed generated image using a first expert model and a second multimodal large language model includes: inputting the remotely sensed generated image into the first expert model for inference to obtain a fourth mask result containing a second category semantic label for each pixel of the remotely sensed generated image; based on the fourth mask result, inputting image regions of the same category into the second multimodal large language model to output a fifth category semantic label and a second confidence score for the image regions; and responding to a second confidence score greater than a preset threshold and the fifth category semantic label being inconsistent with the second category semantic label corresponding to the image region, updating the category of the second category semantic label according to the fifth category semantic label.
[0046] The remotely sensed image is input into the first expert model for inference, resulting in a fourth mask containing the second category semantic label for each pixel of the remotely sensed image. Based on the fourth mask, the corresponding image region is selected according to the category and input into the second multimodal large language model. The output is required to be structured (including corrected category and confidence score). If the confidence score is greater than a preset threshold (e.g., 0.7), the category is updated and calibrated; otherwise, the original category information is retained. Based on the target category to be classified in the remotely sensed image, the second multimodal large language model can be questioned to obtain the fifth category semantic label and the second confidence score for the corresponding image region. For example, if the first expert model predicts a region as "car" and the second multimodal large language model determines it as "boat," with a confidence score of 0.85 > the threshold of 0.7, then the category of the second category semantic label is updated to "boat." The preset threshold can be set according to the actual situation.
[0047] The semantic annotation method for remotely sensed generated images provided by this invention further improves the accuracy of semantic annotation of remotely sensed generated images by updating and calibrating the second category semantic labels with inconsistent categories when the second confidence score of the fifth category semantic label output by the second multimodal large speech model is greater than a preset threshold.
[0048] According to a semantic annotation method for remotely sensed generated images provided by the present invention, the step of obtaining a third mask result containing a third category semantic label for each pixel of the remotely sensed generated image using a second expert model and a third multimodal large language model includes: inputting the remotely sensed generated image into the second expert model for inference to obtain a fifth mask result containing a third category semantic label for each pixel of the remotely sensed generated image; based on the fifth mask result, inputting image regions of the same category into the third multimodal large language model to output a sixth category semantic label and a third confidence score for the pixels of the image regions; and responding to the third confidence score being greater than a preset threshold and the sixth category semantic label being inconsistent with the third category semantic label corresponding to the image region, updating the category of the third category semantic label according to the sixth category semantic label.
[0049] The remotely sensed image is input into a second expert model for inference, resulting in a fifth mask containing the third category semantic label for each pixel of the remotely sensed image. Based on the fifth mask, the corresponding image region is selected according to the category and input into a third multimodal large language model. The output is required to be structured (including corrected category and confidence score). If the confidence score is greater than a preset threshold (e.g., 0.7), the category is updated and calibrated; otherwise, the original category information is retained. Based on the target category to be classified in the remotely sensed image, questions can be posed to the third multimodal large language model to obtain the sixth category semantic label and the third confidence score for the corresponding image region. For example, if the second expert model predicts a region as "car" and the third multimodal large language model determines it as "boat," with a confidence score of 0.65 < the threshold of 0.7, the category of the second category semantic label is not updated and remains "car."
[0050] The semantic annotation method for remotely sensed images provided by this invention further improves the accuracy of semantic annotation of remotely sensed images by updating and calibrating the third category semantic labels that are inconsistent in category when the third confidence score of the sixth category semantic label output by the third multimodal large speech model is greater than a preset threshold.
[0051] According to a semantic annotation method for remotely sensed generated images provided by the present invention, the step of fusing the first mask result, the second mask result, and the third mask result to determine the fourth category semantic label of the pixels of the remotely sensed generated image includes: in response to the second category semantic label and the third category semantic label being consistent, determining the fourth category semantic label of the pixels of the remotely sensed generated image based on the second category semantic label or the third category semantic label; in response to the second category semantic label and the third category semantic label being inconsistent, but the first category semantic label and the second category semantic label being consistent, determining the fourth category semantic label of the pixels of the remotely sensed generated image based on the first category semantic label or the second category semantic label; in response to the second category semantic label and the third category semantic label being inconsistent, but the first category semantic label and the third category semantic label being consistent, determining the fourth category semantic label of the pixels of the remotely sensed generated image based on the first category semantic label or the third category semantic label; and in response to the first category semantic label, the second category semantic label, and the third category semantic label all being inconsistent, including the corresponding pixels in the area to be reviewed.
[0052] A fourth category semantic label for the pixels of the remotely sensed image can be obtained by fusing the first mask result, the second mask result, and the third mask result based on a preset label result voting strategy. A pixel-by-pixel voting strategy is then implemented for each pixel location, specifically as follows:
[0053] If the second category semantic label of the second mask result is consistent with the third category semantic label of the third mask result, then according to the principle of expert priority, the fourth category semantic label of the pixels of the remote sensing generated image is determined based on the second category semantic label or the third category semantic label. That is, the fourth category semantic label can be either the second category semantic label or the third category semantic label.
[0054] If the second category semantic label of the second mask result is inconsistent with the third category semantic label of the third mask result, then according to the principle of majority rule, if the first category semantic label of the first mask result is consistent with the second category semantic label of the second mask result, then the fourth category semantic label is either the first category semantic label or the second category semantic label; if the first category semantic label of the first mask result is consistent with the third category semantic label of the third mask result, then the fourth category semantic label is either the first category semantic label or the third category semantic label.
[0055] If the semantic labels of the first category, the second category, and the third category are all inconsistent, the corresponding pixels will be included in the area to be reviewed, and the category semantic labels of the area to be reviewed can be temporarily determined.
[0056] For example, if the second and third category semantic labels of a pixel are both "building", while the first category semantic label is "road", the system will directly label the pixel as "building" based on the "expert priority" principle. If the second category semantic label is "building", the third category semantic label is "field", and the first and second category semantic labels are both "building", then the final label will still be determined as "building" based on the "consensus priority" principle. If the prediction results of the three categories are different, the pixel will be marked as "area to be reviewed" for subsequent manual review.
[0057] The semantic annotation method for remotely sensed generated images provided by this invention further improves the accuracy of semantic annotation of remotely sensed generated images by determining a fourth category of semantic labels based on the principles of expert priority and majority rule during the fusion of the first mask result, the second mask result, and the third mask result.
[0058] According to a semantic annotation method for remotely sensed generated images provided by the present invention, after obtaining the fourth category semantic labels of the pixels of the remotely sensed generated image, the method further includes: receiving manually labeled data for the region to be reviewed, the over-segmented region, the under-segmented region, and the incorrectly labeled region, and correcting or supplementing the fourth category semantic labels of the pixels of the remotely sensed generated image based on the manually labeled data.
[0059] The fourth category semantic labels of pixels in the remotely sensed image, determined by fusing the first, second, and third mask results, undergo manual review and calibration. This includes manual review and calibration of areas to be reviewed, over-segmented areas, under-segmented areas, and incorrectly labeled areas. Manual review and calibration is primarily performed through category correction, boundary optimization, and new labeling. For example, for category correction, areas incorrectly labeled as "parking lot" are changed to "plaza"; for boundary optimization, the outline of over-segmented docks is manually adjusted; and for new labeling, vehicle category areas not detected by SAM are added.
[0060] The semantic annotation method for remotely sensed generated images provided by this invention provides an effective mechanism for allowing domain experts to intervene in key areas for review and correction by manually reviewing and calibrating the fourth category semantic labels of pixels in the remotely sensed generated image determined by fusing the first mask result, the second mask result, and the third mask result. This further improves the accuracy of semantic annotation of remotely sensed generated images and prevents erroneous results from directly flowing into downstream applications.
[0061] According to a semantic annotation method for remotely sensed generated images provided by the present invention, the step of determining the remotely sensed generated image to be annotated includes: sampling images in rounds from a library of generated remotely sensed generated images; wherein, for each round of image sampling, a preset number of remotely sensed generated images to be annotated are extracted according to different scene categories; and the current remotely sensed generated image to be annotated is determined from the preset number of remotely sensed generated images to be annotated in a preset order.
[0062] Images to be labeled are randomly sampled from the existing remote sensing image library. A stratified sampling method is used to select a certain number of unlabeled remote sensing images according to different scene categories (such as residential, beach, industrial port, urban block).
[0063] Taking four scene types—residential, beach, industrial port, and urban block—as an example, unlabeled images are extracted in a 1:1:1:1 ratio, with each image being 512×512 pixels. If 200 images are extracted each time, then there can be 50 samples each for residential, beach, industrial port, and urban block scenes. Then, based on these 200 images, the remote sensing image to be labeled is determined in a preset order.
[0064] The semantic annotation method for remote sensing generated images provided by this invention improves the acquisition efficiency of the currently unannotated remote sensing generated images by determining the current unannotated remote sensing generated images based on image sampling and in sequence; and improves the richness of the unannotated remote sensing generated images obtained from each round of image sampling by extracting a preset number of unannotated remote sensing generated images according to different scene categories for each round of image sampling.
[0065] According to a semantic annotation method for remotely sensed generated images provided by the present invention, before each round of image sampling, the method further includes: obtaining a first training sample; wherein the first training sample includes a publicly available remote sensing dataset and reconstructed category semantic label annotation data; obtaining a second training sample; wherein the second training sample includes remotely sensed generated images obtained from the image sampling in the previous round and the fourth category semantic labels; and training the first expert model and the second expert model based on the first training sample and / or the second training sample, respectively.
[0066] Before each round of image sampling, a self-training process is performed on the first expert model and the second expert model. The training samples for the first expert model and the second expert model may include the previously sampled remote sensing generated images and the predicted fourth category semantic labels, and may also include the category semantic label annotation data based on the publicly available remote sensing dataset and the reconstructed category semantic labels.
[0067] Before image sampling, a first training sample is obtained, including publicly available remote sensing datasets and reconstructed category semantic label annotations. A second training sample is obtained, including remote sensing generated images obtained from previous rounds of image sampling and fourth-category semantic labels. The first and second expert models are trained based on the first and / or second training samples, respectively. Open-source remote sensing datasets such as DLRSD, WHDLD, and LoveDA can be used. The original labels include multiple categories such as buildings, roads, vehicles, and trees. Through label reconstruction, only the target category is retained, such as roads, buildings, vegetation, vehicles, water, grassland, bare soil, domed buildings, playgrounds, plaza roads, and piers, while other categories are used as background to generate a new mask.
[0068] Since image sampling and the determination of the fourth category labels for the remote sensing-generated images to be labeled were not performed during the first training of the expert model, the first training of the expert model was based solely on the publicly available remote sensing dataset and the reconstructed category semantic label annotation data. Subsequent training can incorporate the remote sensing-generated images and fourth category semantic labels obtained from previous sampling rounds as training samples.
[0069] Figure 2 This is a schematic diagram of the self-training process in the semantic annotation method for remotely sensed generated images provided by this invention. For example... Figure 2 As shown, before reasoning about the image extracted in the i-th time, the expert model is trained by using the images extracted in the (i-1)th time and before and the category semantic labels, together with the remote sensing public dataset and the reconstructed category semantic label annotation data.
[0070] Taking the sampling of 200 images each time as an example, the training data for the i-th iteration consists of the following: 2000 examples from the reconstructed public dataset; historical generated images from the (i-1)-th iteration (with precise generated annotations); and 200 newly added generated images from the (i-1)-th sampling (with precise generated annotations). The newly added data will update the generated data, so the total mixed data volume is [200*(i-1) + 2000] examples. For example, the training data for the first iteration = 2000 examples from the reconstructed public dataset; the training data for the second iteration = 2000 examples from the reconstructed public dataset + 200 generated images from the first iteration (with precise generated annotations); the training data for the third iteration = the training data from the second iteration + 200 generated images from the second iteration (with precise generated annotations). Accurate annotation is the result of inference performed by the expert-trained model during the (i-1)th iteration, followed by category update and calibration by the multimodal large language model, and result fusion (including voting fusion of the general segmentation model and the annotation results of all current expert models, and manual calibration). The corrected, high-quality annotated data is then fed back to the expert model for fine-tuning, forming a closed-loop iterative optimization process. This process continues, with the expert model absorbing new annotated data through incremental self-training. Experiments show that the expert model's performance improves significantly after self-training; the mIoU segmentation metric is 67% after the first training iteration, increasing to 72% after adding 200 annotated data points in the second iteration. This self-training process continues until all generated data is annotated.
[0071] The first and second expert models can use different semantic segmentation models, such as DocNet and LogCan. During training, the cross-entropy loss function can be used to train for 150 epochs (training rounds), and the initial learning rate can be 1e-4.
[0072] The semantic annotation method for remotely sensed generated images provided by this invention, before image sampling, utilizes publicly available remote sensing datasets and reconstructed category semantic label annotation data, along with remotely sensed generated images obtained from previous rounds of image sampling and fourth-category semantic labels, to self-train a first expert model and a second expert model. This provides the expert models with richer and more target-appropriate mixed training samples, improving the inference accuracy of both the first and second expert models, thereby further enhancing the accuracy of semantic annotation for remotely sensed generated images. The expert models can continuously utilize new annotation information to improve their performance, providing a better starting point for subsequent processing.
[0073] According to a semantic annotation method for remotely sensed generated images provided by the present invention, the method further includes: for each round of image sampling, ensuring that the preset number of remotely sensed generated images to be annotated do not overlap with historical sampling results by using a random seed.
[0074] To achieve non-overlapping sampling and avoid unnecessary calculations, a random seed is used for each round of image sampling to ensure that a preset number of remote sensing images to be labeled do not overlap with historical sampling results.
[0075] The semantic annotation method for remotely generated images provided by this invention ensures that a preset number of remotely generated images to be annotated do not overlap with historical sampling results by using random seeds, thereby avoiding unnecessary calculations and saving computing resources.
[0076] Figure 3 This is the second flowchart illustrating the semantic annotation method for remotely sensed generated images provided by this invention. For example... Figure 3 As shown, the method includes:
[0077] A first number of remote sensing generated images are randomly sampled from the remote sensing generated image library, and the current remote sensing generated image to be labeled is determined sequentially from the first number of remote sensing generated images.
[0078] Input the currently unannotated remote sensing image into the general segmentation model to obtain a category-independent segmentation mask;
[0079] Based on the category-independent segmentation mask, rule filtering is performed using a multimodal large language model (only retaining high-confidence category semantic labels), resulting in a mask result of 0;
[0080] The remote sensing image to be labeled is input into expert model 1 and expert model 2 for inference. The predicted categories of expert model 1 and expert model 2 are updated and calibrated using a multimodal large language model to obtain mask result 1 and mask result 2.
[0081] The category semantic labels of the remote sensing generated image to be labeled are obtained by fusing mask result 0, mask result 1 and mask result 2 using a label result voting strategy. The category semantic labels are then calibrated by manual review to obtain the category-related segmentation mask.
[0082] Before each round of image sampling, expert model 1 and expert model 2 are self-trained based on the publicly available remote sensing dataset reconstructed from labels and the labeled data of remote sensing generated images from previous rounds.
[0083] To address the pain points of existing remote sensing image generation annotation methods, this invention innovatively proposes a semantic annotation method for remote sensing generated images. Specifically, it leverages the zero-shot learning capability of a general segmentation model to rapidly generate initial pixel-level semantic labels, effectively overcoming the challenge of traditional image processing algorithms failing to handle artifacts, noise, and dense small targets in generated images. By combining the high reliability of expert models in task-specific scenarios with the generalization ability of general segmentation models in open domains, and utilizing the zero-shot learning capability of a multimodal large language model, it achieves more robust multi-model fusion and result consistency control, improving the accuracy of semantic annotation for remote sensing generated images. Through an efficient human-computer interaction mechanism, professionals can efficiently review, correct, or supplement the annotation of key complex regions in the preliminary results, avoiding the efficiency bottleneck of purely manual annotation while accurately overcoming the uncontrollable error defects of fully automated solutions. Finally, the corrected high-quality annotation data is fed back in real time for fine-tuning the expert model, and the annotation accuracy is continuously optimized during the iteration process. This solution successfully achieves the complementary advantages of "large model efficiency + expert accuracy," realizing low-cost, high-precision, and scalable acquisition of semantic labels for remote sensing generated images.
[0084] The semantic annotation apparatus for remotely sensed generated images provided by the present invention will be described below. The semantic annotation apparatus for remotely sensed generated images described below can be referred to in correspondence with the semantic annotation method for remotely sensed generated images described above.
[0085] Figure 4 This is a schematic diagram of the semantic annotation device for remotely sensed images provided by the present invention. For example... Figure 4 As shown, the device includes a determining module 10, a first annotation module 20, a second annotation module 30, and a fusion module 40, wherein: the determining module 10 is used to: determine the remote sensing generated image to be annotated; the first annotation module 20 is used to: obtain a first mask result containing a first category semantic label for each pixel of the remote sensing generated image using a general segmentation large model and a first multimodal large language model; the second annotation module 30 is used to: obtain a second mask result containing a second category semantic label for each pixel of the remote sensing generated image using a first expert model and a second multimodal large language model, and obtain a third mask result containing a third category semantic label for each pixel of the remote sensing generated image using a second expert model and a third multimodal large language model; the fusion module 40 is used to: fuse the first mask result, the second mask result, and the third mask result to determine a fourth category semantic label for the pixels of the remote sensing generated image, obtain a category-related segmentation mask for the remote sensing generated image based on the fourth category semantic label, and obtain a category-related segmentation mask containing the fourth category semantic label.
[0086] The semantic annotation device for remotely sensed generated images provided by this invention obtains a first mask result by using a general segmentation model and a first multimodal large language model, a second mask result by using a first expert model and a second multimodal large language model, and a third mask result by using a second expert model and a third multimodal large language model. The first, second, and third mask results are then fused to obtain the category semantic labels for the pixels of the remotely sensed generated image. This achieves efficient and automated annotation of remotely sensed generated images, reduces annotation costs, and fully combines the high reliability of expert models in task-specific scenarios with the generalization ability of general segmentation models in open domains, as well as the zero-shot learning capability of multimodal large language models. This results in more robust multi-model fusion and result consistency control, improving the accuracy of semantic annotation of remotely sensed generated images.
[0087] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can call logical instructions in the memory 530 to execute a semantic annotation method for remotely sensed generated images. This method includes: determining the currently annotated remotely sensed generated image; obtaining a first mask result containing a first category semantic label for each pixel of the remotely sensed generated image using a general segmentation model and a first multimodal large language model; obtaining a second mask result containing a second category semantic label for each pixel of the remotely sensed generated image using a first expert model and a second multimodal large language model; obtaining a third mask result containing a third category semantic label for each pixel of the remotely sensed generated image using a second expert model and a third multimodal large language model; fusing the first mask result, the second mask result, and the third mask result to determine a fourth category semantic label for the pixels of the remotely sensed generated image; and obtaining a category-related segmentation mask for the remotely sensed generated image based on the fourth category semantic label.
[0088] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the semantic annotation method for remotely sensed generated images provided by the above methods. The method includes: determining the remotely sensed generated image to be annotated; obtaining a first mask result containing a first category semantic label for each pixel of the remotely sensed generated image using a general segmentation large model and a first multimodal large language model; obtaining a second mask result containing a second category semantic label for each pixel of the remotely sensed generated image using a first expert model and a second multimodal large language model; obtaining a third mask result containing a third category semantic label for each pixel of the remotely sensed generated image using a second expert model and a third multimodal large language model; fusing the first mask result, the second mask result, and the third mask result to determine a fourth category semantic label for the pixels of the remotely sensed generated image; and obtaining a category-related segmentation mask for the remotely sensed generated image based on the fourth category semantic label.
[0090] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a semantic annotation method for remotely sensed generated images provided by the methods described above. The method includes: determining a remotely sensed generated image to be annotated; obtaining a first mask result containing a first category semantic label for each pixel of the remotely sensed generated image using a general segmentation large model and a first multimodal large language model; obtaining a second mask result containing a second category semantic label for each pixel of the remotely sensed generated image using a first expert model and a second multimodal large language model; obtaining a third mask result containing a third category semantic label for each pixel of the remotely sensed generated image using a second expert model and a third multimodal large language model; fusing the first mask result, the second mask result, and the third mask result to determine a fourth category semantic label for the pixels of the remotely sensed generated image; and obtaining a category-related segmentation mask for the remotely sensed generated image based on the fourth category semantic label.
[0091] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
[0095] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A semantic annotation method for remotely sensed images, characterized in that, include: Determine the remote sensing image to be annotated; A first mask result containing a first category semantic label is obtained for each pixel of the remotely sensed image using a general segmentation model and a first multimodal large language model; wherein, the general segmentation model processes the remotely sensed image to obtain a category-independent segmentation mask; the first multimodal large language model performs semantic annotation on the category-independent segmentation mask to obtain a first mask result containing a first category semantic label for each pixel of the remotely sensed image. A second masking result containing a second category semantic label is obtained for each pixel of the remotely sensed image using a first expert model and a second multimodal large language model. A third masking result containing a third category semantic label is obtained for each pixel of the remotely sensed image using a second expert model and a third multimodal large language model. The second multimodal large language model is used to perform category update calibration on the inference result of the first expert model, and the third multimodal large language model is used to perform category update calibration on the inference result of the second expert model. The fourth category semantic label of the pixels of the remote sensing generated image is determined by fusing the first mask result, the second mask result and the third mask result through a semantic consistency reasoning mechanism, and the category-related segmentation mask of the remote sensing generated image is obtained based on the fourth category semantic label.
2. The semantic annotation method for remotely sensed generated images according to claim 1, characterized in that, The first mask result, which includes the first category semantic label, for each pixel of the remotely sensed image obtained by using a general segmentation model and a first multimodal large language model includes: The remotely sensed image is input into the general segmentation model to obtain a category-independent segmentation mask. Based on the category-independent segmentation mask, the first category semantic label and the first confidence score of the mask region are output using the first multimodal large language model; According to the first confidence score in descending order, retain the first category semantic labels corresponding to the first confidence score with the highest ranking of the preset proportion, and set the first category semantic labels corresponding to the other first confidence scores with the lowest ranking as the background to obtain the first mask result.
3. The semantic annotation method for remotely sensed generated images according to claim 1, characterized in that, The second mask result, which includes a second category semantic label, is obtained for each pixel of the remotely sensed image using a first expert model and a second multimodal large language model. This includes: The remotely sensed image is input into the first expert model for inference to obtain a fourth mask result containing the second category semantic label for each pixel of the remotely sensed image. Based on the fourth mask result, image regions of the same category are input into the second multimodal large language model, and the fifth category semantic label and the second confidence score of the image region are output. If the second confidence score is greater than a preset threshold and the fifth category semantic label is inconsistent with the second category semantic label corresponding to the image region, then the second category semantic label is updated according to the fifth category semantic label.
4. The semantic annotation method for remotely sensed generated images according to claim 3, characterized in that, The third mask result, which includes a third category semantic label, is obtained for each pixel of the remotely sensed image using a second expert model and a third multimodal large language model. This includes: The remotely sensed image is input into the second expert model for inference to obtain a fifth mask result containing a third category semantic label for each pixel of the remotely sensed image. Based on the fifth mask result, image regions of the same category are input into the third multimodal large language model, and the sixth category semantic label and third confidence score of the pixels of the image region are output. If the third confidence score is greater than the preset threshold and the sixth category semantic label is inconsistent with the third category semantic label corresponding to the image region, then the third category semantic label is updated according to the sixth category semantic label.
5. The semantic annotation method for remotely sensed generated images according to claim 1, characterized in that, The step of determining the fourth category semantic label of the pixels in the remotely sensed image by fusing the first mask result, the second mask result, and the third mask result through a semantic consistency reasoning mechanism includes: In response to the second category semantic label and the third category semantic label being consistent, a fourth category semantic label for the pixel of the remote sensing generated image is determined based on the second category semantic label or the third category semantic label; In response to the inconsistency between the second category semantic label and the third category semantic label, and the consistency between the first category semantic label and the second category semantic label, the fourth category semantic label of the pixel of the remote sensing generated image is determined based on the first category semantic label or the second category semantic label; In response to the inconsistency between the second category semantic label and the third category semantic label, and the consistency between the first category semantic label and the third category semantic label, the fourth category semantic label of the pixel of the remote sensing generated image is determined based on the first category semantic label or the third category semantic label. If the first category semantic tag, the second category semantic tag, and the third category semantic tag are all inconsistent, the corresponding pixel will be included in the area to be reviewed.
6. The semantic annotation method for remotely sensed generated images according to claim 5, characterized in that, After determining the fourth category semantic label of the pixels in the remotely sensed image, the method further includes: Receive manually labeled data for the regions to be reviewed, over-segmented regions, under-segmented regions, and incorrectly labeled regions, and correct or supplement the fourth category semantic labels of the pixels in the remote sensing generated image based on the manually labeled data.
7. The semantic annotation method for remotely sensed generated images according to claim 1, characterized in that, The process of determining the currently generated remote sensing image to be labeled includes: Image sampling is performed in rounds from the existing remote sensing image library; for each round of image sampling, a preset number of remote sensing images to be labeled are extracted according to different scene categories. According to a preset order, the currently generated remote sensing image to be labeled is determined from the preset number of generated remote sensing images to be labeled.
8. The semantic annotation method for remotely sensed generated images according to claim 7, characterized in that, Prior to each round of image sampling, the method further includes: Obtain the first training sample; wherein, the first training sample includes a publicly available remote sensing dataset and reconstructed category semantic label annotation data; Obtain a second training sample; wherein the second training sample includes remotely generated images obtained from the image sampling in the previous round and the fourth category semantic labels; The first expert model and the second expert model are trained based on the first training sample and / or the second training sample, respectively.
9. The semantic annotation method for remotely sensed generated images according to claim 7, characterized in that, The method further includes: For each round of image sampling, a random seed is used to ensure that the preset number of remote sensing images to be labeled do not overlap with historical sampling results.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the semantic annotation method for remotely sensed generated images as described in any one of claims 1 to 9.