Method and device for constructing optical remote sensing image annotation engine based on iterative reasoning

Abstract

The present invention discloses a method and device for constructing an optical remote sensing image annotation engine based on iterative reasoning; the method can be used as a supplement to the manual annotation method of optical remote sensing image training samples, reducing the annotation cost and difficulty of optical remote sensing image training data. The method is oriented to the training operation scenario of multimodal large models, and aims to generate multi-granular multimodal synthetic annotation data sets based on massive real optical remote sensing images, thereby supporting the training of multimodal large models in various optical remote sensing image interpretation tasks such as image classification, image description, image question and answer, image region classification, image region description, image region question and answer, target detection, vision grounding, target mask extraction, instance segmentation, reference expression segmentation, etc. The present invention enables a single model to have the ability to handle various spatial hierarchies and semantic granularities.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic annotation of deep learning data, and in particular to a method and device for constructing an optical remote sensing image annotation engine based on iterative reasoning. Background Art

[0002] With the rapid development of deep learning, especially in the fields of computer vision, natural language processing and speech recognition, the demand for training data is increasing. Deep learning models usually require a large amount of high-quality annotated data for training. However, the manual annotation process usually requires a lot of human resources, involving the recruitment, training and management of annotators. This leads to high costs, especially when large-scale annotation is required, the economic burden is significant. In addition, manual data annotation is a time-consuming process, especially in tasks that require fine annotation (such as image segmentation, sentiment analysis, etc.). The speed of manual annotation often cannot meet the needs of rapid iteration and updating of deep learning models, slowing down the progress of research and product development.

[0003] Compared with the manual annotation of natural images, the manual annotation of remote sensing images faces more unique difficulties and challenges. The annotation of remote sensing images usually requires rich professional knowledge, including geographic information systems (GIS), environmental science, etc., which requires annotators to have higher professional qualities. Remote sensing images usually contain a variety of land types (such as water bodies, vegetation, cities, etc.), which show different characteristics under different environmental conditions, making the annotation process more complicated. Remote sensing images have different spatial resolutions. Low-resolution images may cause blurred targets, while high-resolution images increase the requirements for detailed annotation. The same object has significant differences in shape and size in remote sensing images of different resolutions. Annotators need to deal with multi-scale problems, which increases the difficulty of identification and annotation.

[0004] In addition, in the field of remote sensing images, existing public datasets have significant problems in terms of annotation scale, annotation modality, and annotation granularity. Many existing remote sensing datasets are often annotated only for a single visual task, which means that the annotation information in a single dataset often only covers a single annotation modality and annotation granularity. On the one hand, this leads to the inability to fully utilize the rich ground object information in remote sensing images. On the other hand, this incomplete annotation mode leads to the model being able to learn only information of a single modality and a single granularity in the image. This incompleteness of learning limits the scope of application and generalization ability of the model.

[0005] In general, although manual annotation of deep learning model training data for remote sensing images is still indispensable in some cases, its high cost, time consumption, single modality and granularity and other limitations seriously hinder the large-scale application of computer vision technology in the field of remote sensing. This has prompted researchers to explore automated annotation technology and other alternative methods to improve annotation efficiency and quality.

[0006] Automatic labeling technology mainly involves multiple fields such as computer vision, natural language processing and machine learning. Its basic idea is to use existing labeled data, generative models or semi-supervised learning methods to automatically generate high-quality labeled data, thereby reducing manual intervention. In this process, the main technologies used include transfer learning, generative adversarial networks (GANs), self-supervised learning and active learning. Summary of the invention

[0007] In view of the deficiencies in the prior art, the object of the present invention is to provide a method and device for constructing an optical remote sensing image annotation engine based on iterative reasoning.

[0008] The objective of the present invention is achieved through the following technical solution: A method for constructing an optical remote sensing image annotation engine based on iterative reasoning, comprising the following steps:

[0009] Collect an open source optical remote sensing data set with annotations; set a set L of annotation items for each image in the open source optical remote sensing data set, and divide the set L into a set A, a set B, and a set C;

[0010] For each annotation item in the set A and the set B, an expert system is constructed respectively, and the expert system is used to perform reasoning on each image in the open source optical remote sensing dataset, and the reasoning results are filtered and fused with the original annotation content of the corresponding image on the annotation item, and the fused annotation content is used as the annotation content of the annotation item on the image;

[0011] Repeat the following steps: train a large multimodal model M using each image of the open source optical remote sensing dataset and the annotation content of each annotation item of the image in set A and set B; for each annotation item in set A and set B, perform reasoning on each image in the open source optical remote sensing dataset based on the large multimodal model M, filter the reasoning results, and fuse them with the current annotation content of the corresponding image on the annotation item, and use the fused annotation content as the annotation content of the annotation item on the image;

[0012] After the iteration is completed, for each labeled item in the set C, a small number of labeled samples corresponding to the labeled items are obtained by manual labeling, and the multimodal large model M is trained using the labeled samples;

[0013] For each annotation item in the set C, the trained multimodal large model M is used to perform reasoning on each image of the open source optical remote sensing dataset, and the reasoning results are filtered and fused with the current annotation content of the corresponding image on the annotation item, and the fused annotation content is used as the annotation content of the annotation item on the image;

[0014] Repeat the following steps: retrain the trained multimodal large model M using each image of the open source optical remote sensing dataset and the annotation content of each annotation item of the image in the set C; for each annotation item in the set C, perform reasoning on each image in the open source optical remote sensing dataset based on the retrained multimodal large model M, filter the reasoning results, and fuse them with the current annotation content of the corresponding image on the annotation item, and use the fused annotation content as the annotation content of the annotation item on the image;

[0015] After the iteration is completed, for each image of the open source optical remote sensing dataset, the annotation content obtained after two repeated iterations is used as the final annotation content of the image, thereby obtaining a multi-granularity and multi-modal optical remote sensing image annotation data set.

[0016] Furthermore, the open source optical remote sensing dataset includes an image classification dataset, a target detection dataset, a visual grounding dataset, an instance segmentation dataset, an image description dataset and a VQA dataset.

[0017] Furthermore, the set L contains 7 elements, namely: image category, a brief description of the image, a description of the relationship between objects in the image, a horizontal object box of each object in the image, a pixel mask of each object in the image, a category name of each object in the image, and a brief description of each object in the image.

[0018] Further, the fusion method is specifically as follows: if the currently targeted annotation item is the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is empty, the filtered reasoning result is used as the fused annotation content; if the currently targeted annotation item is the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is not empty, the filtered reasoning result and the current annotation content are subjected to non-maximum suppression processing, and the processing result is used as the fused annotation content; if the currently targeted annotation item is not the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is not manually annotated, the filtered reasoning result is used as the fused annotation content; if the currently targeted annotation item is not the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is manually annotated, the current annotation content of the image is used as the fused annotation content; if the currently targeted annotation item is not the annotation item of the horizontal target box of each object in the image, and the filtered reasoning result is empty, the current annotation content of the image is used as the fused annotation content.

[0019] Furthermore, the division of set L into set A, set B and set C is specifically as follows: if the number of annotated samples of the same type of annotated item in the open source optical remote sensing dataset exceeds 100,000, the annotated item is divided into set A; if the number of annotated samples of the same type of annotated item in the open source optical remote sensing dataset is less than 100,000, but the open source multimodal large model can perform high-quality reasoning for the annotated item, the annotated item is divided into set B; if the number of annotated samples of the same type of annotated item in the open source optical remote sensing dataset is less than 100,000, and the open source multimodal large model cannot perform high-quality reasoning for the annotated item, the annotated item is divided into set C.

[0020] Furthermore, for each annotation item in the set A and the set B, an expert system is constructed respectively, the expert system constructed for each annotation item in the set A is composed of multiple neural network models, and each neural network model is trained based on the original annotation content contained in the open source optical remote sensing dataset; the expert system constructed for each annotation item in the set B is composed of an open source multimodal large model.

[0021] Furthermore, in the process of using the expert system to perform reasoning on each image in the open source optical remote sensing dataset, the reasoning process of the expert system constructed for each annotation item in the set A includes:

[0022] Each neural network model in the expert system performs reasoning independently;

[0023] The inference results of multiple neural network models are fused based on voting rules, and the fused inference results are output.

[0024] Furthermore, the filtering method adopts confidence-based filtering or heuristic rule-based filtering.

[0025] Furthermore, the step of repeating the iterations is performed no more than three times.

[0026] The present invention also provides a device for constructing an optical remote sensing image annotation engine based on iterative reasoning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for constructing an optical remote sensing image annotation engine based on iterative reasoning.

[0027] The beneficial effects of the present invention are:

[0028] 1. It can automatically generate a large-scale optical remote sensing image annotation dataset based on the existing open source optical remote sensing dataset with a single annotation modality and annotation granularity, reducing the difficulty and cost of data annotation.

[0029] 2. It can support the training of multimodal and multi-task visual models in various optical remote sensing image interpretation tasks such as image classification, image description, image question and answer, image region classification, image region description, image region question and answer, object detection, Vision Grounding, object mask extraction, instance segmentation, and reference expression segmentation, so that a single model has the ability to handle various spatial hierarchies and semantic granularities. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0031] Figure 1 A schematic diagram of a flow chart of a method for constructing an optical remote sensing image annotation engine based on iterative reasoning provided by one embodiment of the present invention;

[0032] Figure 2 A schematic diagram of the structure of an optical remote sensing image annotation engine construction device based on iterative reasoning provided by one embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a process for dividing a set of marked items L into three subsets A, B, and C provided in one embodiment of the present invention;

[0034] Figure 4 A schematic diagram of a process flow of an expert system providing an embodiment of the present invention for reasoning about annotated items in a set A;

[0035] Figure 5 A schematic diagram of a process for filtering inference results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0038] like Figure 1 As shown, a method for constructing an optical remote sensing image annotation engine based on iterative reasoning in an embodiment of the present invention includes the following steps:

[0039] S1. Collect a large number of annotated open source optical remote sensing datasets.

[0040] S2. Set a set L of annotation items for each image in the open source optical remote sensing dataset.

[0041] S3. Divide the set L of annotation items into three subsets: set A, set B and set C.

[0042] S4. For each annotation item in set A, an expert system is constructed. Then, the expert system is used to perform reasoning on each image of the data set collected in step S1. After filtering, the reasoning results are fused with the original annotation content of the image on the annotation item. The fused annotation content is used as the annotation content of the annotation item on the image, such as Figure 4 shown.

[0043] S5. For each annotation item in set B, an expert system is constructed. Then, the expert system is used to perform reasoning on each image of the data set collected in step S1. After filtering, the reasoning result is fused with the original annotation content of the image on the annotation item. The fused annotation content is used as the annotation content of the annotation item on the image, such as Figure 5 shown.

[0044] S6. Train a large multimodal model M using each image of the data set collected in step S1 and the annotation content of each annotation item of the image in set A and set B; then, for each annotation item in set A and set B, perform inference on each image of the data set collected in step S1 based on the large multimodal model M, filter the inference result, and fuse it with the current annotation content of the image on the annotation item, and use the fused annotation content as the annotation content of the annotation item on the image.

[0045] S7. Iterate step S6 multiple times to further improve the quality of the generated annotation items.

[0046] S8. For each labeled item in the set C, a small number of labeled samples corresponding to the labeled item are obtained by manual labeling, and then the multimodal large model M obtained in step S7 is trained using these labeled samples.

[0047] S9. For each annotation item in the set C, use the multimodal large model M obtained in step S8 to perform inference on each image of the data set collected in step S1, filter the inference results, and fuse them with the current annotation content of the image on the annotation item, and use the fused annotation content as the annotation content of the annotation item on the image.

[0048] S10. Use each image of the data set collected in step S1 and the annotation content of each annotation item of the image in set C to train the multimodal large model M obtained in step S9; then for each annotation item in set C, perform inference on each image of the data set collected in step S1 based on the multimodal large model M, filter the inference result, and fuse it with the current annotation content of the image on the annotation item, and use the fused annotation content as the annotation content of the annotation item on the image.

[0049] S11, iterating step S10 multiple times to further improve the quality of the generated annotation items.

[0050] S12. For each image in the data set collected in step S1, the annotation content obtained in step S7 and step S11 is used as the final annotation content of the image, thereby obtaining a multi-granularity and multi-modal optical remote sensing image annotation data set.

[0051] In a feasible embodiment, the types of open source optical remote sensing datasets that need to be collected in step S1 include image classification datasets, target detection datasets, visual grounding datasets, instance segmentation datasets, image description datasets, and VQA datasets.

[0052] In a feasible embodiment, the annotation item set in step S2 includes 7 elements, namely: (1) the category of the entire image, (2) a brief description of the entire image, (3) a description of the relationship between objects in the image, (4) a horizontal object box for each object in the image, (5) a pixel mask for each object in the image, (6) the category name of each object in the image, and (7) a brief description of each object in the image.

[0053] In a possible embodiment, Figure 3 As shown, in step S3, for each annotation item in set L, if the data set collected in step S1 already contains a large number of annotation samples of this type (more than 100,000 annotation samples), the annotation item is divided into set A; if the data set collected in step S1 contains only a small number of annotation samples of this type (less than 100,000 annotation samples), but the current open source multimodal large model can perform high-quality reasoning for the annotation item, the annotation item is divided into set B; if the open source remote sensing data set collected in step S1 contains only a small number of annotation samples of this type (less than 100,000 annotation samples), and the current open source multimodal large model cannot perform high-quality reasoning for the annotation item, the annotation item is divided into set C.

[0054] In a feasible embodiment, the expert system in step S4 is composed of multiple neural network models, and each neural network model is trained based on the original annotation content included in the data set collected in step S1.

[0055] In a feasible embodiment, the reasoning process of the expert system in step S4 includes the following sub-steps:

[0056] S41. Each neural network model in the expert system performs reasoning independently.

[0057] S42: Fusing the inference results of each neural network model based on the voting rule, and outputting the fused inference results.

[0058] In a feasible embodiment, the number of neural network models within the expert system in step S4 is no less than 3.

[0059] In a feasible embodiment, the expert system in step S5 is composed of a high-performance open source multi-modal large model.

[0060] In a feasible embodiment, the filtering method of the inference results in step S4, step S5, step S6, step S9 and step S10 may adopt confidence-based filtering or heuristic rule-based filtering.

[0061] In a feasible embodiment, the fusion method of the annotation contents in step S4, step S5, step S6, step S9 and step S10 is as follows: if the currently targeted annotation item is the annotation item (4) and the current annotation content of the image on the annotation item is empty, then the filtered reasoning result is used as the fused annotation content; if the currently targeted annotation item is the annotation item (4) and the current annotation content of the image on the annotation item is not empty, then the filtered reasoning result and the current annotation content are subjected to non-maximum suppression (NMS) processing, and the processing result is used as the fused annotation content. If the currently targeted annotation item is not the aforementioned annotation item (4), and the current annotation content of the image on this annotation item is not manually annotated, then the filtered reasoning result is used as the fused annotation content; if the currently targeted annotation item is not the aforementioned annotation item (4), and the current annotation content of the image on this annotation item is manually annotated, then the current annotation content of the image is used as the fused annotation content; if the currently targeted annotation item is not the aforementioned annotation item (4), and the filtered reasoning result is empty, then the current annotation content of the image is used as the fused annotation content.

[0062] In a feasible embodiment, the number of iterations of step S6 and step S10 does not exceed 3 times; after a certain number of iterations, the quality improvement process of the annotation items will reach a saturation state, and more iterations will not further improve the quality of the annotation items.

[0063] The present invention also provides Figure 2 The one shown corresponds to Figure 1 An optical remote sensing image annotation engine construction device based on iterative reasoning. At the hardware level, the optical remote sensing image annotation engine construction device based on iterative reasoning includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the following Figure 1 The method for constructing an optical remote sensing image annotation engine based on iterative reasoning. Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0064] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0065] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0066] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0067] For the convenience of description, the above device is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0068] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0073] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0074] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0075] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0076] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0077] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for constructing an optical remote sensing image annotation engine based on iterative reasoning, characterized in that: The following steps are involved: Collect an open source optical remote sensing data set with annotations; set a set L of annotation items for each image in the open source optical remote sensing data set, and divide the set L into a set A, a set B, and a set C; For each annotation item in the set A and the set B, an expert system is constructed respectively, and the expert system is used to perform reasoning on each image in the open source optical remote sensing dataset, and the reasoning results are filtered and fused with the original annotation content of the corresponding image on the annotation item, and the fused annotation content is used as the annotation content of the annotation item on the image; Repeat the following steps: train a multimodal large model M using each image of the open source optical remote sensing dataset and the annotation content of each annotation item of the image in set A and set B; For each annotation item in the set A and the set B, reasoning is performed on each image in the open source optical remote sensing dataset based on the multimodal large model M, and after filtering the reasoning result, it is fused with the current annotation content of the corresponding image on the annotation item, and the fused annotation content is used as the annotation content of the annotation item on the image; After the iteration is completed, for each labeled item in the set C, a small number of labeled samples corresponding to the labeled items are obtained by manual labeling, and the multimodal large model M is trained using the labeled samples; For each annotation item in the set C, the trained multimodal large model M is used to perform reasoning on each image of the open source optical remote sensing dataset, and the reasoning results are filtered and fused with the current annotation content of the corresponding image on the annotation item, and the fused annotation content is used as the annotation content of the annotation item on the image; Repeat the following steps: retrain the trained multimodal large model M using each image of the open source optical remote sensing dataset and the annotation content of each annotation item of the image in the set C; for each annotation item in the set C, perform reasoning on each image in the open source optical remote sensing dataset based on the retrained multimodal large model M, filter the reasoning results, and fuse them with the current annotation content of the corresponding image on the annotation item, and use the fused annotation content as the annotation content of the annotation item on the image; After the iteration is completed, for each image of the open source optical remote sensing data set, the annotation content obtained after two repeated iterations is used as the final annotation content of the image to obtain the optical remote sensing image annotation data set.

2. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 1, characterized in that: The open source optical remote sensing datasets include image classification datasets, target detection datasets, visual grounding datasets, instance segmentation datasets, image description datasets and VQA datasets.

3. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 1, characterized in that: The set L contains 7 elements, which are: image category, a brief description of the image, a description of the relationship between objects in the image, a horizontal object box for each object in the image, a pixel mask for each object in the image, a category name for each object in the image, and a brief description for each object in the image.

4. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 3, characterized in that: The fusion method is specifically as follows: if the currently targeted annotation item is the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is empty, then the filtered inference result is used as the fused annotation content; If the currently targeted annotation item is the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is not empty, then the filtered inference result and the current annotation content are subjected to non-maximum suppression processing, and the processing result is used as the fused annotation content; If the currently targeted annotation item is not the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is not manually annotated, the filtered reasoning result is used as the fused annotation content; If the currently targeted annotation item is not the annotation item of the horizontal target box of each object in the image, and the current annotation content of the image on the annotation item is manually annotated, then the current annotation content of the image is used as the fused annotation content; If the currently targeted annotation item is not the annotation item of the horizontal target box of each object in the image, and the filtered reasoning result is empty, the current annotation content of the image is used as the fused annotation content.

5. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 1, characterized in that: The specific method of dividing the set L into set A, set B and set C is as follows: if the number of labeled samples of the same type of labeled items in the open source optical remote sensing dataset exceeds 100,000, the labeled item is divided into set A; if the number of labeled samples of the same type of labeled items in the open source optical remote sensing dataset is less than 100,000, but the open source multimodal large model can perform inference on the labeled item, the labeled item is divided into set B; if the number of labeled samples of the same type of labeled items in the open source optical remote sensing dataset is less than 100,000, and the open source multimodal large model cannot perform high-quality inference on the labeled item, the labeled item is divided into set C.

6. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 1, characterized in that: For each annotation item in the set A and the set B, an expert system is constructed respectively. The expert system constructed for each annotation item in the set A is composed of multiple neural network models, and each neural network model is trained based on the original annotation content contained in the open source optical remote sensing dataset; the expert system constructed for each annotation item in the set B is composed of an open source multimodal large model.

7. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 1, characterized in that: In the process of using the expert system to perform reasoning on each image in the open source optical remote sensing dataset, the reasoning process of the expert system constructed for each annotation item in the set A includes: Each neural network model in the expert system performs reasoning independently; The inference results of each neural network model are fused based on the voting rule, and the fused inference results are output.

8. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 1, characterized in that: The filtering method adopts confidence-based filtering or heuristic rule-based filtering.

9. The method for constructing an optical remote sensing image annotation engine based on iterative reasoning according to claim 1, characterized in that: The number of iterations of the repeated iteration step does not exceed 3 times.

10. A device for constructing an optical remote sensing image annotation engine based on iterative reasoning, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement a method for constructing an optical remote sensing image annotation engine based on iterative reasoning as described in any one of claims 1 to 9.

Images