A remote sensing question answer generation method, device, medium and equipment
By combining text and image-based integrated answer presentation in the remote sensing question answer generation process, the limitations caused by the single method of remote sensing data utilization are solved, and the efficiency and accuracy of remote sensing question answering are improved.
Patent Information
- Application Number
- CN202411193875.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-28
AI Technical Summary
In existing technologies, the utilization of remote sensing data focuses on a single mode, which limits the help that remote sensing data can provide to users and restricts the potential value of remote sensing technology.
By acquiring the remote sensing question text input by the user, relevant documents and images are retrieved from a preset database, prompt statements are generated and input into a large language model to generate the question answer text, and a comprehensive answer combining text and images is displayed.
It provides a more intuitive and easier-to-understand way of displaying answers, improves the efficiency and accuracy of generating answers to remote sensing questions, and enables more comprehensive information acquisition.
Smart Images

Figure CN118709766B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method, device, medium and equipment for generating answers to remote sensing questions. Background Art
[0002] At present, with the rapid development of satellite technology, a large amount of remote sensing data is also generated. Since these remote sensing data play an important role in earth observation, environmental monitoring and scientific research, how to efficiently process and utilize these remote sensing data has become an urgent problem to be solved.
[0003] In the field of remote sensing, using large language models to generate answers to remote sensing questions can help users quickly obtain the information they need. However, traditional methods of using remote sensing data often focus on single-modality remote sensing data, which leads to limitations in the help remote sensing data provides to users and restricts the potential value of remote sensing technology.
[0004] To this end, this specification provides a remote sensing question answer generation method, device, medium and equipment. Summary of the invention
[0005] This specification provides a remote sensing question answer generation method, device, medium and equipment to partially solve the above-mentioned problems existing in the prior art.
[0006] This manual adopts the following technical solutions:
[0007] This specification provides a remote sensing question answer generation method, including:
[0008] Get the remote sensing question text entered by the user;
[0009] According to the remote sensing question text, searching a preset number of documents related to the remote sensing question text in a preset database;
[0010] determining a remote sensing image corresponding to at least a portion of the document;
[0011] A prompt sentence is generated based on the remote sensing question text, the documents and the remote sensing image, and the prompt sentence is input into a preset large language model to generate a question answer text for the remote sensing question text through the large language model, and the question answer text and the remote sensing image are displayed to the user.
[0012] Optionally, before searching a preset number of documents related to the remote sensing question text in a preset database, the method further includes:
[0013] Acquire remote sensing images and remote sensing texts, and determine description texts corresponding to the acquired remote sensing images;
[0014] The acquired remote sensing text and the description text corresponding to the remote sensing image are used as segmented documents;
[0015] Splitting the split document into documents of preset length;
[0016] The segmented documents are respectively input into the trained embedding model to obtain vector representations of the segmented documents, and the vector representations of the segmented documents are stored in a database.
[0017] Optionally, searching a preset number of documents related to the remote sensing question text in a preset database specifically includes:
[0018] Inputting the remote sensing question text into the embedding model to obtain a vector representation of the remote sensing question text;
[0019] According to the vector representation of the remote sensing question text, determining in the database a preset number of vector representations of each document that matches the vector representation of the remote sensing question text;
[0020] According to the determined vector representations of the preset number of documents, a preset number of documents matching the remote sensing question text are determined.
[0021] Optionally, before generating a prompt statement according to the remote sensing question text, the documents and the remote sensing image, the method further includes:
[0022] Combining the remote sensing question text with each remote sensing image to obtain each input data;
[0023] For each input data, the input data is input into the trained visual question answering model, so that the visual question answering model generates a visual question text for the remote sensing image in the input data according to the remote sensing question text, and determines an answer text for the input data according to the visual question text;
[0024] The visual question text and the answer text are used as a visual question-answer text pair of the input data;
[0025] Generate a prompt statement according to the remote sensing question text, the documents and the remote sensing image, specifically including:
[0026] The remote sensing question text, the documents, the description text corresponding to the remote sensing image, and the visual question and answer text pair are spliced by using a preset separator symbol to obtain a spliced text;
[0027] The preset prompt word is spliced with the spliced text to obtain a prompt sentence.
[0028] Optionally, generating a prompt statement according to the remote sensing question text, the documents and the remote sensing image specifically includes:
[0029] According to a preset prompt template, determining the filling positions of the remote sensing question text, the description texts corresponding to the documents and the remote sensing image in the prompt template respectively;
[0030] According to the determined filling positions, the remote sensing question text, the documents and the description text corresponding to the remote sensing image are respectively filled into the corresponding filling positions in the prompt template to determine the prompt sentence.
[0031] Optionally, determine the description text corresponding to the acquired remote sensing image, specifically including:
[0032] Input the acquired remote sensing image into the trained feature model to determine the image features;
[0033] The image features are input into the trained image description model to determine the description text corresponding to the acquired remote sensing image.
[0034] Optionally, presenting the question answer text and the remote sensing image to the user specifically includes:
[0035] Performing semantic recognition on the question answer text, and dividing the question answer text according to the semantic recognition result to obtain divided question answer text segments;
[0036] In the case of displaying multiple remote sensing images, for each question answer text segment, according to the semantic recognition result of the question answer text segment, determining the remote sensing image associated with the question answer text segment;
[0037] The answer text segments of each question and the remote sensing images associated with the answer text segments of each question are displayed correspondingly.
[0038] This specification provides a remote sensing question answer generation device, comprising:
[0039] An acquisition module is used to obtain the remote sensing question text input by the user;
[0040] A retrieval module, used to retrieve a preset number of documents related to the remote sensing question text in a preset database according to the remote sensing question text;
[0041] A determination module, used to determine the remote sensing image corresponding to at least part of the document;
[0042] The answer module is used to generate a prompt sentence according to the remote sensing question text, the documents and the remote sensing image, and input the prompt sentence into a preset large language model to generate a question answer text for the remote sensing question text through the large language model, and display the question answer text and the remote sensing image to the user.
[0043] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the remote sensing question answer generation method is implemented.
[0044] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a remote sensing question answer generation method when executing the program.
[0045] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:
[0046] The remote sensing question answer generation method provided in this specification first obtains the remote sensing question text input by the user. According to the remote sensing question text, a preset number of documents related to the remote sensing question text are retrieved in a preset database. The remote sensing image corresponding to at least part of the documents is determined. According to the remote sensing question text, each document and the remote sensing image, a prompt sentence is generated, and the prompt sentence is input into a preset large language model, so as to generate a question answer text for the remote sensing question text through the large language model, and the question answer text and the remote sensing image are displayed to the user.
[0047] Through the remote sensing question text input by the user, the documents related to the remote sensing question text are determined from the database, and then the remote sensing images corresponding to at least part of the documents are determined. According to the determined documents and remote sensing images, prompt sentences are generated for processing by the large language model, so as to obtain the question answer text output by the large language model. After that, the user can be provided with a comprehensive answer combining text and remote sensing images as the answer to the remote sensing question, providing a more intuitive and easy-to-understand answer display method, so that the user can obtain information more comprehensively, and can also improve the efficiency and accuracy of generating answers to remote sensing questions. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:
[0049] Figure 1 A flowchart of a remote sensing question answer generation method provided in an embodiment of this specification;
[0050] Figure 2 A schematic diagram of a remote sensing question answering method provided for this manual;
[0051] Figure 3 A schematic diagram of a process for determining answers to remote sensing questions provided for this specification;
[0052] Figure 4 A schematic diagram showing the answer provided for this manual;
[0053] Figure 5 A schematic diagram of a remote sensing question answer generation device provided in this specification;
[0054] Figure 6 A method corresponding to the Figure 1 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0056] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.
[0057] Figure 1 A flowchart of a remote sensing question answer generation method provided in an embodiment of this specification includes the following steps:
[0058] S100: Obtaining remote sensing question text input by a user.
[0059] In this specification, the process of generating remote sensing question answers involves processing remote sensing data. In the embodiment of this specification, the process of generating remote sensing question answers can be executed by a server. Of course, this specification does not limit which device is used to perform the process of generating remote sensing question answers. For example, a personal computer and a mobile terminal can also be used to generate remote sensing question answers. For the convenience of description, the following description is based on the server as the execution subject.
[0060] In one or more embodiments of the present specification, a graphical user interface is displayed to the user on a terminal used by the user, such as a smart phone, a tablet computer, or a desktop computer. The graphical user interface may include a text input box, allowing the user to click and trigger the input method to enter the user's remote sensing question. The graphical user interface may also include a "submit" or "confirm" button. After the user enters the remote sensing question in the text input box, the user can click this button to send the remote sensing question text to the server through the terminal. Thereby, the server obtains the remote sensing question text related to remote sensing input by the user.
[0061] S102: According to the remote sensing question text, a preset number of documents related to the remote sensing question text are retrieved from a preset database.
[0062] In one or more embodiments of this specification, the server searches for a preset number of documents related to the remote sensing question text in a preset database according to the remote sensing question text. In this specification, the method of searching for text blocks in the database is not limited, and the search method can be set according to the actual situation.
[0063] In one or more embodiments of the present specification, a preset database stores a number of remote sensing images, description texts corresponding to the remote sensing images, and remote sensing texts. Among them, the description text or remote sensing text corresponding to each remote sensing image can be used as a document. These remote sensing images and remote sensing texts can be obtained from data sources or institutions related to remote sensing, such as remote sensing images taken by satellites such as Landsat 8 and Sentinel-2, scientific literature on remote sensing, technical reports, and text information and remote sensing images in online remote sensing knowledge bases.
[0064] In one or more embodiments of the present specification, in order to improve the retrieval efficiency and accuracy of each document related to the remote sensing question text from the database, the server obtains the remote sensing image and the remote sensing text, and determines the description text corresponding to the obtained remote sensing image. In the present specification, there is no limitation on the method for determining the description text corresponding to the remote sensing image. The remote sensing image can be obtained from scientific literature, and the description text corresponding to the remote sensing image in the scientific literature can be determined. The method for determining the description text can be set according to the actual situation.
[0065] The remote sensing text obtained in the above and the description text corresponding to the determined remote sensing image may have a long text length. In order to further improve the retrieval efficiency and facilitate subsequent model processing, the remote sensing text obtained and the description text corresponding to the remote sensing image are used as segmented documents, and the segmented documents are segmented into documents of preset length. Of course, the specific length (preset length) of the segmented documents is not limited in this specification. For example, it can be a document of 512 tokens or a document of less than 512 tokens. The specific length of the segmented document can be determined according to actual conditions. Moreover, each segmented document of the preset length contains complete semantic information.
[0066] Afterwards, the server inputs each segmented document into the trained embedding model respectively, obtains the vector representation of each segmented document, and stores the vector representation of each segmented document in the database. Among them, the vector representation of each document can be stored in a vector database, such as the Weaviate library. The vector representation of each document stored in the vector database can be used to efficiently and quickly retrieve the document by performing similarity retrieval in the vector database. Of course, not only the vector representation of the document is stored in the vector database. When the vector representation of the document is stored in the vector database, the corresponding document is also stored in the vector database. Among them, when the vector representation of the document segmented from the descriptive text corresponding to the remote sensing image is stored in the vector database, the document segmented from the descriptive text corresponding to the corresponding remote sensing image and the address of the remote sensing image are also stored in the vector database.
[0067] In one or more embodiments of the present specification, the server may input the remote sensing question text into the embedding model to obtain the vector representation of the remote sensing question text. Based on the vector representation of the remote sensing question text, the vector representation of a preset number of documents that match the vector representation of the remote sensing question text is determined in the database. Of course, the method of determining a preset number of documents in the database based on the vector representation of the remote sensing question text is not limited in this specification. For example, the K-Nearest Neighbor (KNN) algorithm searches the database for the vector representation of the K documents most relevant to the remote sensing question text. Then, based on the vector representation of the preset number of documents determined, the preset number of documents that match the remote sensing question text is determined. Among them, the embedding model is a trained text embedding model, which is a machine learning model that converts text into numerical vectors. These vectors can capture the semantic information in the text.
[0068] S104: Determine the remote sensing image corresponding to at least part of the document.
[0069] In one or more embodiments of the present specification, since the preset number of documents retrieved by the server from the database may include documents based on remote sensing images, the server determines the remote sensing images corresponding to at least some of the documents. Of course, there are many other ways to determine the remote sensing images corresponding to at least some of the documents. For example, the documents may be semantically analyzed, and then, based on the results of the semantic analysis, remote sensing images that match the documents in terms of content may be searched from a preset remote sensing library as the determined remote sensing images.
[0070] S106: Generate a prompt sentence based on the remote sensing question text, the documents and the remote sensing image, and input the prompt sentence into a preset large language model to generate a question answer text for the remote sensing question text through the large language model, and display the question answer text and the remote sensing image to the user.
[0071] In one or more embodiments of the present specification, the server generates a prompt statement based on the remote sensing question text, each document retrieved from the database, and the determined remote sensing image, and inputs the prompt statement into the preset large language model, so as to generate a question answer text for the remote sensing question text through the large language model, and display the question answer text and the remote sensing image to the user. Among them, the server can also display the retrieved documents and the determined remote sensing images to the user, and no longer operate on the documents and the determined remote sensing images. Of course, in order to make it as convenient as possible for users to read the text, a question answer text for the remote sensing question text is generated through a large language model, and a logically smooth text segment is obtained for user reference. It is worth noting that the preset large language model can be a generative pre-trained transformer (GPT) model, such as GPT3, GPT3.5 and GPT4 models can be used in this solution, and there is no restriction on the GPT model version, which can be set according to actual conditions. Of course, the large language model can also be trained by the user.
[0072] Figure 2 This is a schematic diagram of answering a remote sensing question provided in this manual. Figure 2 As shown, the user inputs the remote sensing question text (remote sensing question), and retrieves the documents related to the remote sensing question text in the database. The remote sensing image related to the remote sensing question text is determined from each document. Based on the remote sensing image and the text remote sensing question text, the question answer text (question answer) and the remote sensing image are determined and sent to the user.
[0073] In one or more embodiments of the present specification, the server may determine the description text corresponding to the remote sensing image. Then, according to the preset prompt template, the corresponding filling positions of the remote sensing question text, the description text corresponding to each document and the remote sensing image in the prompt template are determined. According to the determined filling positions, the remote sensing question text, the description text corresponding to each document and the remote sensing image are filled into the corresponding filling positions in the prompt template to determine the prompt sentence. After that, the prompt sentence can be input into the large language model to obtain the question answer text.
[0074] In one or more embodiments of this specification, the preset prompt template may be:
[0075] "You are a professional intelligent assistant in the field of remote sensing. Please answer the user's questions based on the information provided below. Please ensure that your answers are accurate, comprehensive, and closely integrated with the given context information.
[0076] User question: {Remote sensing question text}
[0077] Related text information:
[0078] {Retrieved Document 1}
[0079] {Retrieved Document 2} ...
[0081] {Retrieved documents n}
[0082] Remote sensing image description:
[0083] Remote sensing image 1: {Description text corresponding to remote sensing image 1}
[0084] Remote sensing image 2: {Description text corresponding to remote sensing image 2} ...
[0086] Remote sensing image m: {description text corresponding to remote sensing image m}
[0087] Comprehensive analysis:
[0088] Based on the information above, please provide a comprehensive and accurate answer explaining how remote sensing data supports the analysis of this question. In your answer, please note:
[0089] 1. Integrate textual and visual information to ensure the completeness of the answer.
[0090] 2. Highlight the important role of remote sensing technology in solving this problem.
[0091] 3. If necessary, explain relevant remote sensing concepts or techniques.
[0092] 4. Identify any potential limitations or uncertainties.
[0093] answer:".
[0094] based on Figure 1 The remote sensing question answer generation method shown in the figure first obtains the remote sensing question text input by the user. According to the remote sensing question text, a preset number of documents related to the remote sensing question text are retrieved in a preset database. The remote sensing image corresponding to at least part of the documents is determined. According to the remote sensing question text, each document and the remote sensing image, a prompt sentence is generated, and the prompt sentence is input into a preset large language model, so as to generate a question answer text for the remote sensing question text through the large language model, and the question answer text and the remote sensing image are displayed to the user.
[0095] Through the remote sensing question text input by the user, the documents related to the remote sensing question text are determined from the database, and then the remote sensing images corresponding to at least part of the documents are determined. According to the determined documents and remote sensing images, prompt sentences are generated for processing by the large language model, so as to obtain the question answer text output by the large language model. After that, the user can be provided with a comprehensive answer combining text and remote sensing images as the answer to the remote sensing question, providing a more intuitive and easy-to-understand answer display method, so that the user can obtain information more comprehensively, and can also improve the efficiency and accuracy of generating answers to remote sensing questions.
[0096] In addition, in one or more embodiments of the present specification, the server may further combine the remote sensing question text with each remote sensing image to obtain each input data. For each input data, the input data is input into a trained visual question answering model, and the visual question answering model generates a visual question text for the remote sensing image in the input data based on the remote sensing question text, and determines the answer text for the input data based on the visual question text. The visual question text and the answer text are used as a visual question answering text pair. Among them, the trained visual question answering model may be a (Visual Question Answer, VQA) model, and the VQA model generates an answer based on the content of the remote sensing image.
[0097] Afterwards, the server determines the description text corresponding to the remote sensing image. The description text corresponding to each document and the remote sensing image and the visual question and answer text pair are spliced by a preset separator to obtain a spliced text. The preset prompt word is spliced with the spliced text to obtain a prompt sentence. The separator can be one of a period, a comma or a semicolon.
[0098] For example, the preset prompt word is spliced with the spliced text, and the prompt sentence obtained can be expressed as: "Please answer the question based on the information provided below: spliced text". Among them, "Please answer the question based on the information provided below:" is the prompt word. The spliced text can be expressed as: "Remote sensing question text. Document 1. Document 2. Document 3. Description text 1 corresponding to the remote sensing image. Description text 2 corresponding to the remote sensing image. Visual question and answer text pair 1. Visual question and answer text pair 2.".
[0099] In one or more embodiments of the present specification, the server may determine the document corresponding to the remote sensing image from various documents as the description text corresponding to the remote sensing image. The remote sensing image may also be input into a trained feature model to determine the image features. The image features are input into a trained image description model to determine the description text corresponding to the remote sensing image.
[0100] In one or more embodiments of the present specification, the server determines the document corresponding to the remote sensing image from the retrieved documents as the description text corresponding to the remote sensing image.
[0101] According to the preset prompt template, the prompt sentence is determined according to the remote sensing question text, each document, the description text corresponding to the remote sensing image, and each visual question and answer text pair.
[0102] In one or more embodiments of this specification, the preset prompt template may also be:
[0103] "You are a professional intelligent assistant in the field of remote sensing. Please answer the user's questions based on the information provided below. Please ensure that your answers are accurate, comprehensive, and closely integrated with the given context information.
[0104] User question: {Remote sensing question text}
[0105] Related text information:
[0106] {Retrieved Document 1}
[0107] {Retrieved Document 2} ...
[0109] {Retrieved documents n}
[0110] Remote sensing image description:
[0111] Remote sensing image 1: {Description text corresponding to remote sensing image 1}
[0112] Remote sensing image 2: {Description text corresponding to remote sensing image 2} ...
[0114] Remote sensing image m: {description text corresponding to remote sensing image m}
[0115] Visual Question Answering Results:
[0116] Problem 1: {Visual Problem Text 1 for Remote Sensing Image 1}
[0117] Answer 1: {VQA model’s answer to visual question text 1}
[0118] Problem 2: {Visual Problem Text 2 for Remote Sensing Image 2}
[0119] Answer 2: {VQA model’s answer to visual question text 2} ...
[0121] Question m: {visual question text m for remote sensing image m}
[0122] answer m: {the answer given by the VQA model to the visual question text m}
[0123] Comprehensive analysis:
[0124] Based on the information above, please provide a comprehensive and accurate answer explaining how remote sensing data supports the analysis of this question. In your answer, please note:
[0125] 1. Integrate textual and visual information to ensure the completeness of the answer.
[0126] 2. Highlight the important role of remote sensing technology in solving this problem.
[0127] 3. If necessary, explain relevant remote sensing concepts or techniques.
[0128] 4. Identify any potential limitations or uncertainties.
[0129] answer:".
[0130] Figure 3 A flowchart for determining the answer to a remote sensing question is provided in this manual. Figure 3 As shown, Figure 3 The data collection in the upper left corner is responsible for collecting remote sensing images and remote sensing text information, determining image descriptions from the collected remote sensing images (images), segmenting image descriptions from remote sensing text information (text), and storing the segmented documents in the database through an embedding model for retrieval. When the user enters the remote sensing question text, the embedding model converts the remote sensing question text and retrieves the document in the middle of the database. The remote sensing image is determined based on the retrieved document. The remote sensing image and the remote sensing question text are obtained through the visual question answering model to obtain the result of the visual question answering. The result of the visual question answering and the retrieved document are combined into a prompt sentence, so that the large language model can obtain the question answer text based on the prompt sentence. Finally, based on the question answer text and the remote sensing image, the answer to the remote sensing question text is returned to the user.
[0131] In one or more embodiments of the present specification, there may be a case where the relevant text information listed in the prompt template and the remote sensing image description contain documents that are identical to the description text corresponding to the remote sensing image. In this case, the server may also omit the documents in the relevant text information that are identical to the description text corresponding to the remote sensing image and not list them in the prompt statement.
[0132] In one or more embodiments of the present specification, the server obtains remote sensing images and remote sensing texts. The obtained remote sensing images and remote sensing texts are preprocessed, and the preprocessing includes at least one of image correction and removal of useless texts. The image correction includes at least atmospheric correction, geometric correction, and radiation correction of the remote sensing images.
[0133] In one or more embodiments of the present specification, the server inputs the acquired remote sensing image into a trained feature model to determine image features. The trained feature model may be a residual network model, such as a residual network 50-layer model (ResNet-50 model), to extract image features of the remote sensing image.
[0134] The image features are input into the trained image description model to determine the description text corresponding to the acquired remote sensing image. The trained image description model can be a Transformer-based image description generation model, such as a Contrastive Language-Image Pre-training (CLIP) model, which converts the image features of the remote sensing image into description text, that is, the description text corresponding to the remote sensing image.
[0135] In one or more embodiments of the present specification, the server may perform semantic recognition on the question answer text, and divide the question answer text according to the semantic recognition result to obtain divided question answer text segments.
[0136] In the case of displaying a plurality of remote sensing images, for each question answer text segment, a remote sensing image associated with the question answer text segment is determined according to a semantic recognition result of the question answer text segment.
[0137] The answer text segments of each question and the remote sensing images associated with the answer text segments of each question are displayed correspondingly.
[0138] Figure 4 This is a schematic diagram provided in this manual to show the answer. Figure 4 As shown, the left side is the user interface when the question answer text is not semantically recognized and the question answer text is divided. The right side is the user interface when the question answer text is semantically recognized and the question answer text is divided. Figure 4The dotted lines in the display interfaces on the left and right sides indicate that the user needs to swipe up the screen or scroll the interface with the mouse to display the content below the dotted line. On the left, the display method of the question answer text and remote sensing images is not convenient for users to understand the reading method of the question answer text based on the remote sensing images, and the interface displayed to the user cannot display all the question answer text and remote sensing images in one interface. On the right, the question answer text is divided and displayed together with the associated remote sensing images, which is convenient for users to understand and can reasonably distribute the question answer text segments and remote sensing images on the interface.
[0139] The above is a remote sensing question answer generation method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding remote sensing question answer generation device, such as Figure 5 shown.
[0140] Figure 5 A schematic diagram of a remote sensing question answer generation device provided in this specification specifically includes:
[0141] The acquisition module 500 is used to acquire the remote sensing question text input by the user;
[0142] A retrieval module 502 is used to retrieve a preset number of documents related to the remote sensing question text in a preset database according to the remote sensing question text;
[0143] A determination module 504, configured to determine a remote sensing image corresponding to at least part of the document;
[0144] The answer module 506 is used to generate a prompt sentence based on the remote sensing question text, the documents and the remote sensing image, and input the prompt sentence into a preset large language model to generate a question answer text for the remote sensing question text through the large language model, and display the question answer text and the remote sensing image to the user.
[0145] Optionally, the retrieval module 502 is also used to obtain remote sensing images and remote sensing texts, and determine the descriptive texts corresponding to the acquired remote sensing images, use the acquired remote sensing texts and the descriptive texts corresponding to the remote sensing images as segmented documents, segment the segmented documents into documents of preset length, input each segmented document into the trained embedding model respectively, obtain the vector representation of each segmented document, and store the vector representation of each segmented document in a database.
[0146] Optionally, the retrieval module 502 is also used to input the remote sensing question text into the embedding model to obtain the vector representation of the remote sensing question text, determine the vector representation of a preset number of documents in the database that match the vector representation of the remote sensing question text based on the vector representation of the remote sensing question text, and determine the preset number of documents that match the remote sensing question text based on the determined vector representation of the preset number of documents.
[0147] Optionally, the answer module 506 is further used to combine the remote sensing question text with each remote sensing image respectively to obtain each input data, and for each input data, input the input data into a trained visual question answering model, so as to generate a visual question text for the remote sensing image in the input data according to the remote sensing question text through the visual question answering model, and determine the answer text of the input data according to the visual question text, and use the visual question text and the answer text as a visual question answer text pair for the input data; splice the remote sensing question text, the documents, the description text corresponding to the remote sensing image, and the visual question answer text pair through a preset separator to obtain a spliced text, and splice a preset prompt word with the spliced text to obtain a prompt sentence.
[0148] Optionally, the answer module 506 is also used to determine the filling positions corresponding to the remote sensing question text, the description texts corresponding to the documents and the remote sensing images in the prompt template according to a preset prompt template, and fill the remote sensing question text, the description texts corresponding to the documents and the remote sensing images into the corresponding filling positions in the prompt template according to the determined filling positions to determine the prompt statement.
[0149] Optionally, the retrieval module 502 is further used to input the acquired remote sensing image into a trained feature model to determine image features, and input the image features into a trained image description model to determine a description text corresponding to the acquired remote sensing image.
[0150] Optionally, the answer module 506 is further used to perform semantic recognition on the question answer text, and divide the question answer text according to the semantic recognition result to obtain divided question answer text segments. When displaying multiple remote sensing images, for each question answer text segment, based on the semantic recognition result of the question answer text segment, the remote sensing image associated with the question answer text segment is determined, and each question answer text segment and the remote sensing image associated with the question answer text segment are displayed accordingly.
[0151] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1A remote sensing question answering generation method is provided.
[0152] This manual also provides Figure 6 The schematic structure diagram of the electronic device shown in FIG. Figure 6 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The remote sensing question answering generation method.
[0153] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0154] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0155] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.
[0156] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0157] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0158] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0159] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0162] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0163] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0164] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0165] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0166] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0167] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0168] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0169] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.
Claims
1. A remote sensing question answering generation method, It is characterized in that include: Get the remote sensing question text entered by the user; According to the remote sensing question text, searching a preset number of documents related to the remote sensing question text in a preset database; determining a remote sensing image corresponding to at least a portion of the document; A prompt sentence is generated based on the remote sensing question text, the documents and the remote sensing image, and the prompt sentence is input into a preset large language model to generate a question answer text for the remote sensing question text through the large language model, and the question answer text and the remote sensing image are displayed to the user.
2. The method according to claim 1, It is characterized in that Before searching a preset number of documents related to the remote sensing question text in a preset database, the method further includes: Acquire remote sensing images and remote sensing texts, and determine description texts corresponding to the acquired remote sensing images; The acquired remote sensing text and the description text corresponding to the remote sensing image are used as segmented documents; Splitting the split document into documents of preset length; The segmented documents are respectively input into the trained embedding model to obtain vector representations of the segmented documents, and the vector representations of the segmented documents are stored in a database.
3. The method according to claim 2, It is characterized in that Retrieving a preset number of documents related to the remote sensing problem text in a preset database, specifically including: Inputting the remote sensing question text into the embedding model to obtain a vector representation of the remote sensing question text; According to the vector representation of the remote sensing question text, determining in the database a preset number of vector representations of each document that matches the vector representation of the remote sensing question text; According to the determined vector representations of the preset number of documents, a preset number of documents matching the remote sensing question text are determined.
4. The method according to claim 1, It is characterized in that Before generating a prompt statement according to the remote sensing question text, the documents and the remote sensing image, the method further includes: Combining the remote sensing question text with each remote sensing image to obtain input data; For each input data, the input data is input into the trained visual question answering model, so that the visual question answering model generates a visual question text for the remote sensing image in the input data according to the remote sensing question text, and determines an answer text for the input data according to the visual question text; The visual question text and the answer text are used as a visual question-answer text pair of the input data; Generate a prompt statement according to the remote sensing question text, the documents and the remote sensing image, specifically including: The remote sensing question text, the documents, the description text corresponding to the remote sensing image, and the visual question and answer text pair are spliced by using a preset separator symbol to obtain a spliced text; The preset prompt word is spliced with the spliced text to obtain a prompt sentence.
5. The method according to claim 1, It is characterized in that Generate a prompt statement according to the remote sensing question text, the documents and the remote sensing image, specifically including: According to a preset prompt template, determining the filling positions of the remote sensing question text, the description texts corresponding to the documents and the remote sensing image in the prompt template respectively; According to the determined filling positions, the remote sensing question text, the documents and the description text corresponding to the remote sensing image are respectively filled into the corresponding filling positions in the prompt template to determine the prompt sentence.
6. The method according to claim 2, It is characterized in that Determine the description text corresponding to the acquired remote sensing image, including: Input the acquired remote sensing image into the trained feature model to determine the image features; The image features are input into the trained image description model to determine the description text corresponding to the acquired remote sensing image.
7. The method according to claim 1, It is characterized in that The question answer text and the remote sensing image are displayed to the user, specifically comprising: Performing semantic recognition on the question answer text, and dividing the question answer text according to the semantic recognition result to obtain divided question answer text segments; In the case of displaying multiple remote sensing images, for each question answer text segment, according to the semantic recognition result of the question answer text segment, determining the remote sensing image associated with the question answer text segment; The answer text segments of each question and the remote sensing images associated with the answer text segments of each question are displayed correspondingly.
8. A remote sensing question answer generation device, It is characterized in that include: An acquisition module is used to obtain the remote sensing question text input by the user; A retrieval module, used to retrieve a preset number of documents related to the remote sensing question text in a preset database according to the remote sensing question text; A determination module, used to determine the remote sensing image corresponding to at least part of the document; The answer module is used to generate a prompt sentence according to the remote sensing question text, the documents and the remote sensing image, and input the prompt sentence into a preset large language model to generate a question answer text for the remote sensing question text through the large language model, and display the question answer text and the remote sensing image to the user.
9. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.
