Visual understanding method, device and system for remote sensing image, equipment and medium

Abstract

The invention relates to a visual understanding method, device, system and equipment for a remote sensing image, and a medium. Comprising the steps of extracting structured information of a to-be-understood remote sensing image, and generating an image information database corresponding to the to-be-understood remote sensing image based on the structured information; obtaining an initial prompt instruction for the to-be-understood remote sensing image, determining a to-be-retrieved element based on the initial prompt instruction, performing retrieval from the image information database based on the to-be-retrieved element to obtain corresponding target structured information, and generating a target prompt instruction according to the target structured information and the initial prompt instruction; and inputting the target prompt instruction into the fully trained large language model to obtain an understanding result of the to-be-understood remote sensing image. By adopting the method, the accuracy of understanding the remote sensing image can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, system, equipment and medium for visual understanding of remote sensing images. Background Art

[0002] With the rapid development of remote sensing technology, especially the widespread application of earth observation equipment such as high-resolution satellites and drones, massive amounts of remote sensing data continue to be generated. These data are mainly bird's-eye views, containing rich geographical, environmental and object information, such as urban layout, natural resources, transportation networks, etc. However, traditional remote sensing image analysis methods usually rely on single-modality data, such as optical images or SAR (Synthetic Aperture Radar) data. This single-modality method is prone to information loss and understanding bias when understanding complex scenes, and it is difficult to meet the needs of scene diversification and information complexity.

[0003] To solve the above problems, existing technologies mostly use intelligent question-answering systems (such as large language models, LLM) for natural language recognition to analyze and understand remote sensing images. Large language models have powerful language understanding and generation capabilities through large-scale pre-training, and can provide high-quality answers in common scenarios. However, there are still some problems in understanding remote sensing images based on large language models, which are mainly manifested in the following aspects: (1) inaccurate recognition of the number of targets (2) difficulty in identifying the direction and position of targets (3) insufficient recognition of semantic relationships and triples.

[0004] Currently, no effective solution has been proposed for the problem of low recognition accuracy of remote sensing images in existing technologies. Summary of the invention

[0005] Based on this, it is necessary to provide a visual understanding method, device, system, equipment and medium for remote sensing images to address the above technical problems.

[0006] In a first aspect, the present application provides a visual understanding method for remote sensing images. The method comprises: Extracting structured information of the remote sensing image to be understood, and generating an image information database corresponding to the remote sensing image to be understood based on the structured information; Obtaining an initial prompt instruction for the remote sensing image to be understood, determining the element to be retrieved based on the initial prompt instruction, searching from the image information database based on the element to be retrieved, obtaining the corresponding target structured information, and generating the target prompt instruction according to the target structured information and the initial prompt instruction; The target prompt instructions are input into a well-trained large language model to obtain the understanding results of the remote sensing image to be understood.

[0007] In one embodiment, extracting structured information of the acquired remote sensing image to be understood includes: Get a fully trained image analysis model; The remote sensing image to be understood is input into the image analysis model, the elements contained in the remote sensing image to be understood are detected by the image analysis model, and the structural relationship corresponding to the elements is generated; Based on the elements and the structural relationships, structured information corresponding to the remote sensing image to be understood is obtained, wherein the elements and the corresponding structural relationships exist in the form of key-value pairs.

[0008] In one embodiment, generating an image information database corresponding to a remote sensing image to be understood based on structured information includes: A preset text embedding model is obtained, and structured information is embedded based on the text embedding model to obtain structured vector information corresponding to the structured information, and an image information database is generated based on the structured vector information.

[0009] In one embodiment, based on the acquired initial prompt instruction for the remote sensing image to be understood, the corresponding target structured information is retrieved from the image information database, including: Embedding the initial prompt instruction based on the text embedding model to obtain a prompt instruction vector; The corresponding target structure information is retrieved from the image information database based on the hint instruction vector.

[0010] In one embodiment, based on the acquired initial prompt instruction for the remote sensing image to be understood, the corresponding target structured information is retrieved from the image information database, including: Obtaining semantic information of the initial prompt instruction and calculating the similarity between the semantic information and the structured information; At least one target structured information is determined from the structured information according to the similarity calculation result.

[0011] In one embodiment, the target prompt instruction is input into a well-trained large language model to obtain the understanding result of the remote sensing image to be understood, including: The target prompt instruction is input into the large language model, and a visual analysis result corresponding to the target prompt instruction is generated based on the large language model, and based on the visual analysis result, corresponding visual recognition is performed in the remote sensing image to be understood.

[0012] In a second aspect, the present application also provides a visual understanding device for remote sensing images. The device comprises: An acquisition module is used to extract the structured information of the acquired remote sensing image to be understood, and generate an image information database corresponding to the remote sensing image to be understood based on the structured information; A calculation module is used to obtain an initial prompt instruction for the remote sensing image to be understood, determine the element to be retrieved based on the initial prompt instruction, retrieve from the image information database based on the element to be retrieved, obtain the corresponding target structured information, and generate a target prompt instruction according to the target structured information and the initial prompt instruction; The generation module is used to input the target prompt instructions into the well-trained large language model to obtain the understanding results of the remote sensing image to be understood.

[0013] In a third aspect, the present application also provides a visual understanding system for remote sensing images, the system comprising an image analysis module, a database construction module, a retrieval module, a question and answer generation module and a visualization module; An image analysis module is used to extract the structured information of the acquired remote sensing image to be understood and send the structured information to the database construction module; A database construction module is used to integrate structured information and establish an image information database corresponding to the remote sensing image to be understood; A retrieval module is used to obtain the initial prompt instruction, determine the element to be retrieved based on the initial prompt instruction, retrieve the corresponding target structured information from the image information database based on the element to be retrieved, combine the target structured information with the initial prompt instruction, obtain the target prompt instruction, and send the target prompt instruction to the question and answer generation module; A question-answer generation module, used to generate visual analysis results based on target prompt instructions; The visualization module is used to perform visual recognition in the remote sensing image to be understood based on the visual analysis results.

[0014] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and an identifier, the memory stores a computer program, and the identifier implements the following steps when executing the computer program: Extracting structured information of the remote sensing image to be understood, and generating an image information database corresponding to the remote sensing image to be understood based on the structured information; Obtaining an initial prompt instruction for the remote sensing image to be understood, determining the element to be retrieved based on the initial prompt instruction, searching from the image information database based on the element to be retrieved, obtaining the corresponding target structured information, and generating the target prompt instruction according to the target structured information and the initial prompt instruction; The target prompt instructions are input into a well-trained large language model to obtain the understanding results of the remote sensing image to be understood.

[0015] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by the identifier, the following steps are implemented: Extracting structured information of the remote sensing image to be understood, and generating an image information database corresponding to the remote sensing image to be understood based on the structured information; Obtaining an initial prompt instruction for the remote sensing image to be understood, determining the element to be retrieved based on the initial prompt instruction, searching from the image information database based on the element to be retrieved, obtaining the corresponding target structured information, and generating the target prompt instruction according to the target structured information and the initial prompt instruction; The target prompt instructions are input into a well-trained large language model to obtain the understanding results of the remote sensing image to be understood.

[0016] The above-mentioned visual understanding method, device, system, equipment and medium for remote sensing images first obtains the structured information of the remote sensing image to be understood, and generates an image information database based on the structured information; then obtains the initial prompt instruction, determines the element to be retrieved based on the initial prompt instruction, retrieves from the image information database based on the element to be retrieved, obtains the target structured information, generates the target prompt instruction based on the target structured information and the initial prompt instruction; inputs the target prompt instruction into a well-trained large language model to obtain the corresponding understanding result. This application can greatly improve the accuracy of understanding remote sensing images with many elements and complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of a visual understanding method in one embodiment; Figure 2 A data flow diagram of a visual understanding method in one embodiment; Figure 3 is a structural block diagram of a visual understanding device in one embodiment; Figure 4 is a structural block diagram of a visual understanding system in one embodiment; Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] In one embodiment, Figure 1 As shown, a visual understanding method for remote sensing images is provided, comprising the following steps: Step S110 , extracting structured information of the remote sensing image to be understood, and generating an image information database corresponding to the remote sensing image to be understood based on the structured information.

[0020] Specifically, the structured information of the remote sensing image to be understood is extracted, wherein the structured information represents the elements contained in the remote sensing image to be understood and the related information corresponding to the elements, that is, the structured information includes various types of elements in the remote sensing image to be understood and the structured information corresponding to each element, such as the number of each element, the specific position in the image, the relationship between the element and other elements, etc. In some preferred embodiments, after the structured information of the remote sensing image to be understood is obtained, the format of the structured information can be sorted. The following provides a format of the structured information: "Object Category": {Quantity: the specific number of objects of the current category in the image; Position: The specific position of each object of the current category in the image, such as left, right, lower right corner, etc.; Relationship: The relationship between the current category object and other objects.

[0021] The present application includes but is not limited to the above-mentioned format of structured information. It is understandable that a remote sensing image to be understood should be able to extract structured information of multiple object categories.

[0022] Based on the above structured information, an image information database corresponding to the remote sensing image to be understood is generated, wherein one remote sensing image to be understood corresponds to an image information database, and the image information database includes all structured information corresponding to the remote sensing image to be understood. In some preferred embodiments, the structured information of different object categories can be divided into different text blocks for storage.

[0023] Step S120, obtaining an initial prompt instruction for the remote sensing image to be understood, determining the elements to be retrieved based on the initial prompt instruction, searching from the image information database based on the elements to be retrieved, obtaining the corresponding target structured information, and generating the target prompt instruction based on the target structured information and the initial prompt instruction.

[0024] Specifically, an initial prompt instruction for the remote sensing image to be understood is obtained, wherein the initial prompt instruction may be an instruction for image understanding input by the user, such as "query the number and location of all vehicles in the image", or may be a template prompt corresponding to a large language model, such as: "You are a professional remote sensing image question-answering intelligent assistant. Please answer the user's questions based on the structured information in the following image. Please ensure that your answer is accurate and comprehensive. If you don't know, please answer "I don't know" and do not falsify information.

[0025] User question: {user question}".

[0026] After obtaining the initial prompt instruction, the element to be retrieved is determined according to the initial prompt instruction. In this embodiment, preferably, the semantic information in the initial prompt instruction can be captured according to the preset large language model, so as to determine the element to be retrieved based on the initial prompt instruction, wherein the large language model can use the existing Tongyi Qianwen, deepseek, etc. After clarifying the element to be retrieved, the image information database corresponding to the remote sensing image to be understood is searched based on the element to be retrieved to obtain the corresponding target structured information. In some preferred embodiments, the similarity between the element to be retrieved and the structured information in the image information database can be calculated, and several structured information with high similarity can be selected as the above target structured information. For example, the four structured information with the highest similarity can be used as the above target structured information, and the RAG algorithm (Retrieval-Augmented Generation) can be used to complete the retrieval of the target structured information. After retrieving the target structured information, the target structured information and the initial prompt instruction can be combined based on the above large language model to generate the above target prompt instruction. For example, the target prompt instruction can be: "You are a professional remote sensing image question-answering intelligent assistant. Please answer the user's questions based on the structured information in the following image. Please ensure that your answer is accurate and comprehensive. If you don't know, please answer "I don't know" and do not falsify information.

[0027] User question: {User question} Structured information: {object 1 target structured information}, {object 2 target structured information}…{object 4 target structured information}".

[0028] The above is a template for a target prompt instruction. In actual applications, other templates can also be selected for implementation, and this application does not make too many restrictions on this. Further, the above image information database can be established based on any storage method that is convenient for retrieving and querying structured information, so the above target structured information can be in text form, vector form, etc.

[0029] Step S130, inputting the target prompt instruction into the well-trained large language model to obtain the understanding result of the remote sensing image to be understood.

[0030] Specifically, in actual application, the target structured information retrieved is structured information with a high degree of similarity to the element to be retrieved, but it is not necessarily the answer result of the initial prompt instruction. For example, the initial prompt instruction may be to query the number of vehicles in the remote sensing image to be understood, and then the corresponding target structured information retrieved may be the position of each vehicle, the relationship between each vehicle and other objects, and the number of vehicles, etc. At this time, the target structured information can be combined with the initial prompt instruction to obtain the target prompt instruction and input it into the large language model for understanding. The large language model then gives a targeted answer to the prompt instruction input by the user based on the retrieved target structured information, such as answering the number of vehicles in the remote sensing image to be understood. And further, in some embodiments, the retrieved target structured information may be in vector form, and it is difficult for users to directly read the meaning of the target structured information in vector form. Therefore, the large language model is also required to convert the target prompt instruction and output natural language that can be directly read by users.

[0031] Furthermore, the above-mentioned understanding results include but are not limited to the response results corresponding to the target prompt instructions output by the large language model. It can also understand the remote sensing image to be understood based on the target prompt instructions as needed, such as marking the elements to be retrieved in the image, segmenting the area corresponding to the target prompt instructions, etc.

[0032] Through steps S110 to S130, the structured information of the remote sensing image to be understood is extracted. Especially when facing complex scenes (such as buildings and roads connected, vegetation and water bodies interacting, etc.), the present application can comprehensively extract various data such as the number of elements, orientation, contextual relationship, etc. in the remote sensing image to be understood. Furthermore, after obtaining the initial prompt instruction, the corresponding target structured information is retrieved from the image information database, and then the large language model generates the corresponding reply result and the understanding result of the remote sensing image to be understood based on the target structured information and the initial prompt instruction, thereby greatly improving the accuracy of understanding images with many elements and complex scenes.

[0033] In some embodiments, the method further comprises: Get a fully trained image analysis model; The remote sensing image to be understood is input into the image analysis model, the elements contained in the remote sensing image to be understood are detected by the image analysis model, and the structural relationship corresponding to the elements is generated; Based on the elements and the structural relationships, structured information corresponding to the remote sensing image to be understood is obtained, wherein the elements and the corresponding structural relationships exist in the form of key-value pairs.

[0034] Specifically, the above-mentioned image analysis model is obtained. In some preferred embodiments, the image analysis model may be an SGG (Scene Graph Generation) model, including but not limited to a PENET model, etc. Among them, the training of the image analysis model may be to obtain an initial image analysis model and a remote sensing image training set. The remote sensing image may use an open source remote sensing image relationship data set STAR, as well as corresponding annotation boxes, relationship triples, etc. In some preferred embodiments, before the initial image analysis model is trained based on the remote sensing image training set, the remote sensing image training set may be pre-identified, such as cropping, cleaning, denoising, and deduplication of the images in the training set, and then the initial image analysis model is trained based on the above-mentioned pre-identified remote sensing image training set to obtain the above-mentioned fully trained image analysis model.

[0035] The remote sensing image to be understood is input into the image analysis model, and the image analysis model outputs the elements contained in the remote sensing image to be understood, as well as the structural relationship corresponding to each element, the structural relationship including but not limited to the coordinates of the element, the relationship between the element and other elements, etc.

[0036] Finally, the above elements and the corresponding structural relationships are combined to obtain the above structured information. In some preferred embodiments, the structured information can be saved in the form of a dictionary. For example, the category or keyword information of each element is set as the key, and the corresponding value is the quantity, position, and relationship between other objects of the element.

[0037] Through this implementation, the structured information of remote sensing images can be extracted more comprehensively and accurately.

[0038] In some embodiments, the method further comprises: A preset text embedding model is obtained, and structured information is embedded based on the text embedding model to obtain structured vector information corresponding to the structured information, and an image information database is generated based on the structured vector information.

[0039] Specifically, a preset text embedding model is obtained, and the text embedding model can use text2vec-base-Chinese, text2vec-large-Chinese, etc. The structured information is embedded based on the text embedding model to generate a low-dimensional vector representation as the semantic representation of the structured information, thereby obtaining structured vector information. After obtaining the structured vector information, the Milvus library can be used to establish an efficient vector index. In some preferred embodiments, when the structured information is saved in key-value format, the data corresponding to the key can be converted into a vector, the data corresponding to the value can be directly saved in the database, and the data corresponding to the key and the value can be bound one-to-one, thereby completing the embedded representation of the structured information.

[0040] In this embodiment, the structured information is embedded and represented to obtain a low-dimensional vector representation corresponding to the structured information, thereby more efficiently capturing the semantic information of the query and reducing the storage space occupied by the structured information.

[0041] In some embodiments, the method further comprises: Embedding the initial prompt instruction based on the text embedding model to obtain a prompt instruction vector; The corresponding target structure information is retrieved from the image information database based on the hint instruction vector.

[0042] Specifically, when the structured information in the image information database is in the form of vectors, that is, saved as structured vector information, in some preferred embodiments, the acquired initial prompt instruction can also be embedded, and the same text embedding model as that used for embedding and understanding the structured vector is used to ensure that the vector of the initial prompt instruction provided by the user and the vector of the structured data in the database are in the same semantic space. Based on the text embedding model, the initial prompt instruction input by the user can be converted into a dense vector representation, thereby effectively capturing the semantic information of the query. In some preferred embodiments, in order to improve computational efficiency and facilitate subsequent retrieval and recognition, the initial prompt instruction input by the user and the structured information stored in the database can be converted into the same form, such as both can be in text form, or both can be in vector form.

[0043] In some embodiments, the method further comprises: Obtaining semantic information of the initial prompt instruction and calculating the similarity between the semantic information and the structured information; At least one target structured information is determined from the structured information according to the similarity calculation result.

[0044] Specifically, the semantic information of the initial prompt instruction is obtained. The method for extracting the semantic information includes but is not limited to obtaining the semantic features in the initial prompt instruction based on a preset semantic extraction model, such as text2vec-base-Chinese and text2vec-large-Chinese models. It should be noted that the semantic extraction model for extracting the semantic features of the initial prompt instruction in this embodiment is consistent with the text embedding model mentioned above, that is, the semantic extraction model used at this time is the above text embedding model, and the initial prompt instruction is converted into a dense vector representation to obtain the above semantic information. After obtaining the semantic information, the similarity between the semantic information and the structured information in the database is calculated. The similarity between the semantic information and the structured information of the initial prompt instruction can be calculated based on the Milvus index in the database, and the key vector in the database that is close to the query semantics is identified, and based on the similarity calculation results, multiple target structured information is determined, that is, the structured information corresponding to the key is determined. In some preferred embodiments, it can be set to obtain four target structured information from the structured information, that is, to obtain four target structured information that are most similar to the initial prompt instruction from the structured information. In actual applications, the number of target structured information to be obtained can be set by relevant technical personnel. It can be understood that if the number of target structured information obtained is too small, it may not be possible to fully retrieve the results corresponding to the prompt instruction. If the number of target structured information obtained is too large, the calculation speed will decrease. Therefore, in actual applications, the calculation speed and effect can be comprehensively considered to select the most balanced setting.

[0045] In some embodiments, the method further comprises: The target prompt instruction is input into the large language model, and a visual analysis result corresponding to the target prompt instruction is generated based on the large language model, and based on the visual analysis result, corresponding visual recognition is performed in the remote sensing image to be understood.

[0046] Specifically, the target prompt instruction includes an initial prompt instruction and corresponding target structured information. In practical applications, the target structured information and the initial prompt instruction can be integrated into a logically coherent and semantically consistent prompt. The target prompt instruction is input into a large language model, so that a high-quality answer can be generated using the large language model, that is, the visual analysis result is generated. Secondly, the relevant structured information can be visualized in the original remote sensing image to be understood as needed. For example, if the initial prompt instruction input by the user is to query the number of vehicles in the image, the output visual analysis result can be "the number of vehicles is: 36", and the position of each vehicle is marked with a label box in the corresponding remote sensing image to be understood.

[0047] The present application also provides a preferred embodiment of a visual understanding method for remote sensing images. Figure 2A data flow diagram of a visual understanding method in a preferred embodiment.

[0048] Most existing technologies use natural language recognition intelligent question-answering systems (such as large language models, LLM) to analyze and identify remote sensing images. Large language models have powerful language understanding and generation capabilities through large-scale pre-training, and can provide high-quality answers in common scenarios. However, remote sensing images have various types of elements, uneven density, and often cover large areas, have complex semantic relationships, and lack clear contextual references. Direct understanding of remote sensing images through large language models usually results in low accuracy and incomplete recognition.

[0049] In the present application, first, the user terminal 21 detects the remote sensing image to be understood through the preset image analysis model 22, generates structured information, and establishes an image information database corresponding to the remote sensing image to be understood based on the structured information. Secondly, the user terminal 21 inputs the image information database and the preset initial prompt instruction into the text embedding model 23, and the text embedding model 23 embeds the image information database and the initial prompt instruction to respectively obtain an image information database composed of low-latitude structured vector information and a prompt instruction vector. Then, based on the prompt instruction vector, the user terminal 21 retrieves multiple target structured information with the highest similarity in the image information database composed of structured vector information, combines the target structured information with the initial prompt instruction, and obtains the target prompt instruction. Finally, the target prompt instruction is input into the large language model 24, and the visual analysis results and visual recognition results with high comprehension are output based on the large language model 24, and the visual analysis results and visual recognition results are fed back to the user terminal 21.

[0050] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0051] Based on the same inventive concept, the embodiment of the present application also provides a visual understanding device for implementing the visual understanding method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more visual understanding device embodiments provided below can refer to the limitations on the visual understanding method above, and will not be repeated here.

[0052] In one embodiment, Figure 3 As shown, a visual understanding device is provided, including: an acquisition module 31, a calculation module 32 and a generation module 33, wherein: An acquisition module 31 is used to extract the structured information of the acquired remote sensing image to be understood, and generate an image information database corresponding to the remote sensing image to be understood based on the structured information; The calculation module 32 is used to obtain an initial prompt instruction for the remote sensing image to be understood, determine the element to be retrieved based on the initial prompt instruction, retrieve from the image information database based on the element to be retrieved, obtain the corresponding target structured information, and generate the target prompt instruction according to the target structured information and the initial prompt instruction; The generation module 33 is used to input the target prompt instruction into the well-trained large language model to obtain the understanding result of the remote sensing image to be understood.

[0053] Each module in the above-mentioned visual understanding device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the recognizer in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the recognizer can call and execute the operations corresponding to each of the above modules.

[0054] In one embodiment, Figure 4 As shown, a visual understanding system is provided, comprising: Image analysis module, database construction module, retrieval module, question and answer generation module and visualization module; An image analysis module is used to extract the structured information of the acquired remote sensing image to be understood and send the structured information to the database construction module; A database construction module is used to integrate structured information and establish an image information database corresponding to the remote sensing image to be understood; A retrieval module is used to obtain the initial prompt instruction, determine the element to be retrieved based on the initial prompt instruction, retrieve the corresponding target structured information from the image information database based on the element to be retrieved, combine the target structured information with the initial prompt instruction, obtain the target prompt instruction, and send the target prompt instruction to the question and answer generation module; A question-answer generation module, used to generate visual analysis results based on target prompt instructions; The visualization module is used to perform visual recognition in the remote sensing image to be understood based on the visual analysis results.

[0055] Specifically, the image analysis module can extract the structured information of the remote sensing image to be understood, providing assistance for the precise answering and generation of the subsequent large language model.

[0056] The above-mentioned database construction module can use the text embedding model to identify the structured data extracted by the image analysis module into high-order vectors, and then construct them into a searchable vector database.

[0057] The above-mentioned retrieval module is responsible for implementing the retrieval-augmented generation technology (Retrieval-Augmented Generation), receiving the initial prompt instruction input by the user, and then using the above-mentioned text embedding model to convert the initial prompt instruction into a vector form, and retrieve the target structured information with the highest similarity in the database, thereby providing rich contextual information for the subsequent large model generation.

[0058] The question-answer generation module is used to generate a final visual analysis result, which is an accurate answer corresponding to the initial prompt instruction.

[0059] The above-mentioned visualization module is responsible for visually identifying the output results of the big language on the remote sensing image to be understood, such as marking the category and annotation box of the corresponding element.

[0060] The above modular design makes the system highly flexible and scalable. Each module can be upgraded or replaced independently without affecting the functionality of the overall system. For example, the large language model can be updated, the retrieval module can be improved, the image analysis module can be improved, etc. to adapt to the ever-evolving technology and needs.

[0061] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes an identifier, a memory and a network interface connected via a system bus. Among them, the identifier of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store visual understanding related data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the identifier, a visual understanding method is implemented.

[0062] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0063] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a recognizer, the following steps are implemented: Extracting structured information of the remote sensing image to be understood, and generating an image information database corresponding to the remote sensing image to be understood based on the structured information; Obtaining an initial prompt instruction for the remote sensing image to be understood, determining the element to be retrieved based on the initial prompt instruction, searching from the image information database based on the element to be retrieved, obtaining the corresponding target structured information, and generating the target prompt instruction according to the target structured information and the initial prompt instruction; The target prompt instructions are input into a well-trained large language model to obtain the understanding results of the remote sensing image to be understood.

[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0065] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The identifiers involved in each embodiment provided in this application may be universal identifiers, central identifiers, graphic identifiers, digital signal identifiers, programmable logic devices, data recognition logic devices based on quantum computing, etc., but are not limited to this.

[0066] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A visual understanding method for remote sensing images, characterized in that: The method comprises: Extracting structured information of the remote sensing image to be understood, and generating an image information database corresponding to the remote sensing image to be understood based on the structured information; Acquire an initial prompt instruction for the remote sensing image to be understood, determine an element to be retrieved based on the initial prompt instruction, retrieve from the image information database based on the element to be retrieved to obtain corresponding target structured information, and generate a target prompt instruction based on the target structured information and the initial prompt instruction; The target prompt instruction is input into a well-trained large language model to obtain an understanding result of the remote sensing image to be understood.

2. The method according to claim 1, characterized in that The step of extracting the structured information of the remote sensing image to be understood includes: Get a fully trained image analysis model; Inputting the remote sensing image to be understood into the image analysis model, detecting elements contained in the remote sensing image to be understood by the image analysis model, and generating structural relationships corresponding to the elements; The structured information corresponding to the remote sensing image to be understood is obtained based on the elements and the structural relationships, wherein the elements and the corresponding structural relationships exist in the form of key-value pairs.

3. The method according to claim 1, characterized in that: The step of generating an image information database corresponding to the remote sensing image to be understood based on the structured information includes: A preset text embedding model is obtained, and the structured information is embedded and represented based on the text embedding model to obtain structured vector information corresponding to the structured information one by one, and the image information database is generated based on the structured vector information.

4. The method according to claim 3, characterized in that The step of retrieving corresponding target structured information from the image information database based on the acquired initial prompt instruction for the remote sensing image to be understood includes: Embedding the initial prompt instruction based on the text embedding model to obtain a prompt instruction vector; The corresponding target structure information is retrieved from the image information database based on the hint instruction vector.

5. The method according to claim 1, characterized in that The step of retrieving corresponding target structured information from the image information database based on the acquired initial prompt instruction for the remote sensing image to be understood includes: Acquiring semantic information of the initial prompt instruction, and calculating the similarity between the semantic information and the structured information; At least one target structured information is determined from the structured information according to the similarity calculation result.

6. The method according to claim 1, characterized in that The target prompt instruction is input into a well-trained large language model to obtain an understanding result of the remote sensing image to be understood, including: The target prompt instruction is input into the large language model, and a visual analysis result corresponding to the target prompt instruction is generated based on the large language model, and visual recognition is performed correspondingly in the remote sensing image to be understood based on the visual analysis result.

7. A visual understanding device for remote sensing images, characterized in that: The device comprises: An acquisition module, used for extracting structured information of the acquired remote sensing image to be understood, and generating an image information database corresponding to the remote sensing image to be understood based on the structured information; A calculation module, used for obtaining an initial prompt instruction for the remote sensing image to be understood, determining an element to be retrieved based on the initial prompt instruction, searching from the image information database based on the element to be retrieved to obtain corresponding target structured information, and generating a target prompt instruction according to the target structured information and the initial prompt instruction; A generation module is used to input the target prompt instruction into a well-trained large language model to obtain an understanding result of the remote sensing image to be understood.

8. A visual understanding system for remote sensing images, characterized in that: The system includes an image analysis module, a database construction module, a retrieval module, a question and answer generation module and a visualization module; The image analysis module is used to extract the structured information of the acquired remote sensing image to be understood, and send the structured information to the database construction module; The database construction module is used to integrate the structured information and establish an image information database corresponding to the remote sensing image to be understood; The retrieval module is used to obtain the initial prompt instruction, determine the element to be retrieved based on the initial prompt instruction, retrieve the corresponding target structured information from the image information database based on the element to be retrieved, combine the target structured information with the initial prompt instruction, obtain the target prompt instruction, and send the target prompt instruction to the question and answer generation module; The question-answer generation module is used to generate a visual analysis result based on the target prompt instruction; The visualization module is used to perform visual recognition in the remote sensing image to be understood based on the visual analysis result.

9. A computer device comprising a memory and an identifier, wherein the memory stores a computer program, characterized in that: When the identifier executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by an identifier, the steps of the method according to any one of claims 1 to 6 are implemented.

Images