Remote sensing image processing method, system and equipment based on knowledge graph and medium

Abstract

The embodiment of the invention provides a remote sensing image processing method, system and equipment based on a knowledge graph and a medium, and the method comprises the steps: obtaining a high-resolution image sample and corresponding spatio-temporal information, a query text and an answer label, carrying out the retrieval in the knowledge graph based on the query text and the spatio-temporal information, obtaining a knowledge text, and carrying out the processing of the knowledge text. The query text, the knowledge text, the high-resolution image sample and a low-resolution image obtained by down-sampling based on the high-resolution image sample are respectively coded, and the visual language model obtains a target mark based on a text mark obtained by coding, a high-resolution image mark and a low-resolution image mark; the large-scale language model obtains the first prediction answer based on the target mark, performs model training based on the first prediction answer and the answer label to obtain the target large-scale language model, and performs prediction based on the input image and the input query text by using the model to obtain the second prediction answer, so that the accuracy of remote sensing knowledge retrieval by the model is improved.

Description

Technical Field

[0001] The present application relates to the fields of remote sensing technology and artificial intelligence technology, and in particular to a remote sensing image processing method, system, device and medium based on a knowledge graph. Background Art

[0002] In the field of remote sensing technology and artificial intelligence, large-scale language models combined with retrieval-enhanced generation technology have become an effective means to improve the model's understanding and generation capabilities. This paradigm usually involves extracting keywords from the input text question, and then using these keywords to efficiently search in a large knowledge base or knowledge graph to obtain background knowledge and information related to the question. In this way, the model can combine the retrieved knowledge to generate more accurate and rich answers or explanations.

[0003] However, in the specific application scenario of remote sensing tasks, traditional large-scale language models combined with retrieval-enhanced generation technology face significant challenges. Remote sensing task problems often focus on the description and analysis of visual content rather than simple language expression. Such problems usually contain fewer extractable keywords, and these keywords are often not precise enough to accurately reflect the core visual features and requirements of the problem. In addition, the information in remote sensing images is complex and diverse, containing a large amount of professional knowledge such as geography, landforms, and meteorology. The extraction and interpretation of this information places extremely high demands on the accuracy and comprehensiveness of keywords.

[0004] Due to the above reasons, keywords directly extracted from remote sensing task questions often cannot effectively retrieve relevant and accurate knowledge in the knowledge base or knowledge graph, which greatly limits the accuracy and effect of the application of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models. Therefore, how to optimize keyword extraction strategies and improve retrieval accuracy and efficiency based on the characteristics of remote sensing tasks has become a key issue that needs to be solved in the current research on remote sensing visual language models. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a remote sensing image processing method, system, device and medium based on a knowledge graph, aiming to use a spatial index knowledge graph to replace traditional keyword extraction, to solve the problem that large-scale language models are difficult to query remote sensing knowledge, and to improve the accuracy of remote sensing knowledge retrieval in large-scale language models.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a remote sensing image processing method based on a knowledge graph, the method comprising:

[0007] Acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples;

[0008] Based on the query text and the spatiotemporal information, searching in a preset knowledge graph to obtain knowledge text;

[0009] Based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag;

[0010] Downsampling the high-resolution image samples to obtain low-resolution image samples;

[0011] Inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image tag and a low-resolution image tag;

[0012] Obtaining a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag;

[0013] Based on the target tag, obtaining a first predicted answer through a large-scale language model;

[0014] Based on the first predicted answer and the answer label, training the large-scale language model to obtain a target large-scale language model;

[0015] The target large-scale language model is used to perform prediction based on the input image and the input query text to obtain a second predicted answer.

[0016] Through the method provided in the first aspect, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in knowledge graphs, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval.

[0017] In a possible implementation, based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag includes:

[0018] The knowledge text is sorted according to the time accuracy, and is spliced ​​with the query text to obtain a spliced ​​text, and the spliced ​​text is input into the text encoder for text encoding to obtain a text tag.

[0019] In one possible implementation, the visual language model includes a self-attention module, a flexible attention module, and a projection layer;

[0020] The obtaining a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag comprises:

[0021] Inputting the low-resolution image tag and the text tag into a self-attention module to perform self-attention calculation to obtain a first tag;

[0022] Obtaining a second tag through a flexible attention module based on the first tag, the high-resolution image tag, the low-resolution image tag, and the text tag;

[0023] The projection layer analyzes and processes the second mark to obtain a target mark.

[0024] In a possible implementation, the knowledge text includes ROI data, and the flexible attention module includes a hierarchical self-attention module and a high-resolution marker selection module;

[0025] Based on the first tag, the high-resolution image tag, the low-resolution image tag, and the text tag, obtaining a second tag by a flexible attention module includes:

[0026] Perform coordinate calculation based on the ROI data to obtain coordinate marks;

[0027] Adding the coordinate mark and the first mark to obtain a fusion mark;

[0028] Based on the fusion mark, the high-resolution image mark, the low-resolution image mark and the text mark in the hierarchical self-attention module, the high-resolution mark selection module performs multiple rounds of high-resolution mark screening to obtain a target high-resolution mark;

[0029] Based on the target high-resolution label, the low-resolution image label and the text label, a second label is obtained through a hierarchical self-attention module.

[0030] In one possible implementation, the hierarchical self-attention module includes a plurality of hierarchical self-attention layers;

[0031] Based on the fusion mark, the high-resolution image mark, the low-resolution image mark and the text mark in the hierarchical self-attention module, multiple rounds of high-resolution mark screening are performed by the high-resolution mark selection module to obtain the target high-resolution mark, including:

[0032] The fusion mark is screened by the high-resolution mark selection module at the first layer of the hierarchical self-attention layer to obtain the selected high-resolution mark;

[0033] Based on the selected high-resolution mark, the low-resolution image mark and the text mark, a calculation is performed through the next layer of the hierarchical self-attention layer, and the new fusion mark is obtained by adding the coordinate mark to the selected high-resolution mark;

[0034] Based on the new fusion mark, the high-resolution mark selection module performs high-resolution mark screening layer by layer in the multiple hierarchical self-attention layers to finally obtain the target high-resolution mark.

[0035] In a possible implementation, training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model includes:

[0036] A loss function is determined based on the first predicted answer and the answer label, and parameters of the large-scale language model are adjusted based on the loss function to obtain a target large-scale language model.

[0037] In a possible implementation manner, the step of adjusting parameters of the large-scale language model based on the loss function includes:

[0038] The parameters of the large-scale language model are calculated based on the loss function and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimization algorithm.

[0039] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present application proposes a remote sensing image processing system based on a knowledge graph, the system comprising:

[0040] A training sample acquisition module is used to acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples;

[0041] Retrieval module: used to search in a preset knowledge graph based on the query text and the spatiotemporal information to obtain knowledge text;

[0042] A text encoding module: used for performing text encoding through a text encoder based on the query text and the knowledge text to obtain a text tag;

[0043] Downsampling module: used for downsampling the high-resolution image samples to obtain low-resolution image samples;

[0044] Image encoding module: used for inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image mark and a low-resolution image mark;

[0045] A target tag acquisition module: used to obtain a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag;

[0046] Model prediction module: used to obtain a first prediction answer based on the target tag through a large-scale language model;

[0047] Model training module: training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model;

[0048] Data processing module: used to use the target large-scale language model to make predictions based on the input image and the input query text to obtain a second prediction answer.

[0049] Through the system provided in the second aspect, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in the knowledge graph, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval.

[0050] In a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the remote sensing image processing method based on the knowledge graph is implemented as described in any possible implementation method in the first aspect.

[0051] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the remote sensing image processing method based on the knowledge graph is implemented as described in any possible implementation method in the first aspect.

[0052] It can be seen from the technical solutions provided by one or more embodiments of the above specification that the remote sensing image processing method based on the knowledge graph provided by the embodiments of the present application obtains high-resolution image samples and their corresponding spatiotemporal information, query text, and answer labels, and searches in a preset knowledge graph based on the query text and spatiotemporal information to obtain knowledge text, respectively encodes the query text and the knowledge text, and performs image encoding on the high-resolution image samples and the low-resolution images obtained by downsampling the high-resolution image samples, obtains target labels through a visual language model based on the obtained text labels, high-resolution image labels, and low-resolution image labels, obtains a first predicted answer through a large-scale language model based on the target labels, performs model training based on the first predicted answer and the answer label to obtain a target large-scale language model, and uses the target large-scale language model to perform prediction based on the input image and the input query text to obtain a second predicted answer, thereby improving the accuracy of the model in retrieving remote sensing knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the description of one or more embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 It is a flowchart of a remote sensing image processing method based on a knowledge graph provided in an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of the system architecture of the visual language model provided in an embodiment of the present application;

[0056] Figure 3 This is an architecture diagram of a remote sensing image processing system based on a knowledge graph provided for an embodiment of the present application;

[0057] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the one or more embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0062] Figure 1 is an optional flowchart of the remote sensing image processing method based on the knowledge graph provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S900.

[0063] First, as Figure 1 As shown, a remote sensing image processing method based on a knowledge graph is provided, and the method includes:

[0064] S100: Acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples.

[0065] S200: Based on the query text and spatiotemporal information, search in a preset knowledge graph to obtain knowledge text.

[0066] It should be noted that the spatiotemporal information is composed of the time information and spatial information of the high-resolution image samples, and the time information and spatial information include the time and geographic coordinates of the high-resolution image samples. The time and geographic coordinates of the high-resolution image samples are added on the basis of the query text, that is, the spatial index method is adopted to perform knowledge retrieval in the preset knowledge graph to obtain the knowledge text, wherein the knowledge text includes the land use information, administrative district information and ROI information of the surface within the image coverage range. By adopting the form of spatial index knowledge graph to replace the traditional keyword extraction, the limitations of the existing large-scale language model in querying remote sensing knowledge can be overcome, so that the large-scale language model can effectively retrieve relevant and accurate knowledge in the knowledge graph, improve the application accuracy and effect of the large-scale language model combined with the retrieval enhancement generation technology in the remote sensing visual language model, and improve the accuracy and efficiency of knowledge retrieval. In addition, it can be understood that the embodiment of the present application does not limit the type of knowledge text.

[0067] It should also be noted that by combining query text and spatiotemporal information, searching in the preset knowledge graph can help the model understand the user's query intention more accurately, capture the user's query intention and information needs more comprehensively, and find the remote sensing knowledge that best matches it, making full use of the visual features of remote sensing problems and improving the accuracy and efficiency of retrieval.

[0068] S300: Based on the query text and the knowledge text, text encoding is performed by a text encoder to obtain a text tag.

[0069] S400 , down-sampling the high-resolution image samples to obtain low-resolution image samples.

[0070] S500: Input high-resolution image samples and low-resolution image samples into an image encoder respectively to obtain high-resolution image tags and low-resolution image tags.

[0071] S600, based on the high-resolution image tag, the low-resolution image tag and the text tag, a target tag is obtained through a visual language model.

[0072] S700: Based on the target tag, obtain a first predicted answer through a large-scale language model.

[0073] S800: Based on the first predicted answer and the answer label, train a large-scale language model to obtain a target large-scale language model.

[0074] S900: Use the target large-scale language model to make predictions based on the input image and the input query text to obtain a second prediction answer.

[0075] Through the method provided in the first aspect, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in knowledge graphs, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval.

[0076] In a possible implementation, based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag includes:

[0077] The knowledge text is sorted according to the time accuracy, and is spliced ​​with the query text to obtain a spliced ​​text, and the spliced ​​text is input into the text encoder for text encoding to obtain a text tag.

[0078] It should be noted that the knowledge text includes land use information, administrative district information, and ROI (Region of Interest) information on the surface within the image coverage area. Sorting the retrieved knowledge text by time accuracy can help the large-scale language model understand the text content more comprehensively, so that the large-scale language model can understand and process the knowledge text more accurately. In addition, it can be understood that the embodiment of the present application does not limit the type of knowledge text.

[0079] like Figure 2 As shown, Figure 2 A schematic diagram of the system architecture of the visual language model provided for the embodiment of the present application. In one possible implementation, the visual language model includes a self-attention module, a flexible attention module, and a projection layer; the obtaining of a target tag through the visual language model based on the high-resolution image tag, the low-resolution image tag, and the text tag includes: inputting the low-resolution image tag and the text tag into the self-attention module for self-attention calculation to obtain a first tag; based on the first tag, the high-resolution image tag, the low-resolution image tag, and the text tag, obtaining a second tag through the flexible attention module; the projection layer analyzes and processes the second tag to obtain a target tag.

[0080] Among them, in some embodiments, the introduction of a self-attention module in the visual language model can enhance the model's ability to capture and understand the complex relationship between remote sensing images and texts. It enables the model to dynamically allocate attention to key visual features in the image and core semantic information in the text, thereby more accurately interpreting the image content and generating text descriptions or answers that match it. At the same time, the self-attention module promotes the deep fusion between the two modal data of image and text, allowing the model to show higher accuracy when processing cross-modal tasks. In order to improve computational efficiency and reduce the amount of calculation, only low-resolution image tags and text tags are input into the self-attention module for self-attention calculation to obtain the first tag, so as to roughly understand the entire image and improve the model's ability to understand the image. In order to perceive more details of the image more efficiently, a flexible attention module is introduced in the visual language model, which can perform weighted averaging operations based on the context of the input features, and dynamically determine the attention weights by calculating the similarity function between related pixel pairs. This mechanism enables the model to flexibly focus on different areas in the image, thereby capturing more feature information.

[0081] In one possible implementation, the knowledge text includes ROI data, and the flexible attention module includes a hierarchical self-attention module and a high-resolution tag selection module; based on the first tag, the high-resolution image tag, the low-resolution image tag and the text tag, obtaining the second tag through the flexible attention module includes: performing coordinate calculation based on the ROI data to obtain a coordinate tag; adding the coordinate tag and the first tag to obtain a fusion tag; based on the fusion tag, the high-resolution image tag, the low-resolution image tag and the text tag, performing multiple rounds of high-resolution tag screening in the hierarchical self-attention module through the high-resolution tag selection module to obtain a target high-resolution tag; based on the target high-resolution tag, the low-resolution image tag and the text tag, obtaining the second tag through the hierarchical self-attention module.

[0082] It should be noted that the current mainstream large-scale language models were designed to process images of regular sizes. Their input sizes are usually limited by multiple factors such as model architecture, computing resources, and training efficiency, and are generally small. This size mismatch has become a major obstacle to directly applying advanced visual language models to remote sensing image analysis tasks. By introducing ROI data in the hierarchical self-attention module, multiple rounds of high-resolution tag selection are performed in the hierarchical self-attention module to obtain a target high-resolution tag. Based on the target high-resolution tag, low-resolution image tag, and text tag, a second tag is obtained through the hierarchical self-attention module. The projection layer obtains the target tag by analyzing and processing the second tag. Based on the target tag, the large-scale language model is used to predict the answer. In surface research, ROI data refers to the area with the most significant changes in surface morphology and cover type or the most research value. By introducing ROI data in the hierarchical self-attention module, the limitations of existing large-scale language models in processing large-format images can be overcome. While maintaining the integrity of remote sensing image information, large-scale language models can be used to process large-format images, significantly improving the analysis efficiency and depth of remote sensing data, so that the rich information in the images can be captured and interpreted more comprehensively and accurately, promoting the widespread application of remote sensing technology in key areas such as environmental monitoring, urban planning, and disaster warning, providing more detailed and reliable data support for decision-making, and helping to accelerate the data processing process and improve the level of automation and intelligence, thereby reducing the cost and error of manual interpretation to a certain extent.

[0083] In one possible implementation, the hierarchical self-attention module includes multiple hierarchical self-attention layers; based on the fusion mark, the high-resolution image mark, the low-resolution image mark and the text mark, the hierarchical self-attention module performs multiple rounds of high-resolution mark screening through the high-resolution mark selection module to obtain the target high-resolution mark, including: the fusion mark is screened for high-resolution marks in the first layer of the hierarchical self-attention layer by the high-resolution mark selection module to obtain the selected high-resolution mark; based on the selected high-resolution mark, the low-resolution image mark and the text mark, the calculation is performed through the next layer of the hierarchical self-attention layer, and added to the coordinate mark to obtain a new fusion mark; based on the new fusion mark, the high-resolution mark selection module is used to screen high-resolution marks layer by layer in the multiple hierarchical self-attention layers to finally obtain the target high-resolution mark.

[0084] It should be noted that, in some embodiments, the flexible attention module includes a high-resolution marker selection module and a hierarchical self-attention module. The high-resolution marker selection module is used to perform multiple rounds of high-resolution image marker screening in multiple hierarchical self-attention layers in the hierarchical self-attention module, and the high-resolution markers selected in each layer are input into the next hierarchical self-attention layer, and calculated with the low-resolution image markers and text markers. The calculated markers are added to the coordinate markers to obtain new fusion markers, which are used for high-resolution marker selection of the next hierarchical self-attention layer until the target high-resolution marker is selected. The purpose of high-resolution screening is to reduce the overall computing cost by reducing unnecessary computing overhead while improving the model's ability to process high-resolution images, thereby achieving more efficient and economical visual language model applications.

[0085] It should also be noted that the coordinate marks are obtained by calculating the coordinates based on the ROI data. In each layer of hierarchical self-attention layer, the marks calculated based on the selected high-resolution marks, low-resolution marks and text marks are added to the coordinate marks as the input of the next layer of hierarchical self-attention layer, and are used to screen the high-resolution marks. By introducing ROI data, the limitations of existing large-scale language models in processing large-format images can be overcome. While maintaining the integrity of remote sensing image information, large-scale language models can be used to process large-format images, significantly improving the analysis efficiency and depth of remote sensing data, so that the rich information in the images can be captured and interpreted more comprehensively and accurately, promoting the widespread application of remote sensing technology in key fields such as environmental monitoring, urban planning, and disaster warning, providing more detailed and reliable data support for decision-making, and helping to accelerate the data processing process and improve the level of automation and intelligence, thereby reducing the cost and error of manual interpretation to a certain extent.

[0086] In a possible implementation, training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model includes:

[0087] A loss function is determined based on the first predicted answer and the answer label, and parameters of the large-scale language model are adjusted based on the loss function to obtain a target large-scale language model.

[0088] In a possible implementation manner, the step of adjusting parameters of the large-scale language model based on the loss function includes:

[0089] The parameters of the large-scale language model are calculated based on the loss function and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimization algorithm.

[0090] Among them, it should be noted that the loss function is an important indicator to measure the difference between the model's prediction results and the actual results. By calculating the loss value, you can intuitively understand the performance of the model on the training data. When the loss value gradually decreases, it means that the large-scale language model is gradually approaching the distribution of the real data. At this time, the change in the loss value can be used to guide the adjustment of the parameters. The back propagation algorithm calculates the gradient between each layer in the network, and propagates the error signal from the output layer to the input layer layer by layer, thereby obtaining the gradient of each parameter. After obtaining the gradient of each parameter, the optimizer is used to optimize and update the parameters. Through multiple iterations of back propagation and parameter update, the large-scale language model can better fit the training data, optimize the performance of the large-scale language model, improve the generalization ability of the model, accelerate the model training process, and improve the efficiency of the model in processing complex tasks.

[0091] To achieve the above purpose, Figure 3 As shown, the second aspect of the embodiment of the present application proposes a remote sensing image processing system based on a knowledge graph, the system comprising:

[0092] Training sample acquisition module: used to acquire training samples, wherein the training samples include high-resolution image samples and the spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples.

[0093] Retrieval module: used to search in a preset knowledge graph based on the query text and the spatiotemporal information to obtain the knowledge text.

[0094] Among them, it should be noted that the spatiotemporal information is composed of the time information and spatial information of the high-resolution image samples, and the time information and spatial information include the time and geographic coordinates of the high-resolution image samples. The time and geographic coordinates of the high-resolution image samples are added on the basis of the query text, that is, the spatial index method is adopted to perform knowledge retrieval in the preset knowledge graph to obtain the knowledge text, wherein the knowledge text includes the land use information, administrative district information and ROI information of the surface within the image coverage range. By adopting the form of spatial index knowledge graph to replace the traditional keyword extraction, the limitations of the existing large-scale language model in querying remote sensing knowledge can be overcome, so that the large-scale language model can effectively retrieve relevant and accurate knowledge in the knowledge graph, improve the application accuracy and effect of the large-scale language model combined with the retrieval enhancement generation technology in the remote sensing visual language model, and improve the accuracy and efficiency of knowledge retrieval. In addition, it can be understood that the embodiment of the present application does not limit the type of knowledge text.

[0095] It should also be noted that by combining query text and spatiotemporal information, searching in the preset knowledge graph can help the model understand the user's query intention more accurately, capture the user's query intention and information needs more comprehensively, and find the remote sensing knowledge that best matches it, making full use of the visual features of remote sensing problems and improving the accuracy and efficiency of retrieval.

[0096] Text encoding module: used to perform text encoding based on the query text and the knowledge text through a text encoder to obtain text tags.

[0097] Down-sampling module: used to down-sample the high-resolution image samples to obtain low-resolution image samples.

[0098] Image encoding module: used for inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image tag and a low-resolution image tag.

[0099] A target tag acquisition module is used to obtain a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag.

[0100] It should be noted that the visual language model includes a self-attention module, which includes multiple hierarchical self-attention layers. The knowledge text includes ROI data, and coordinate calculation is performed based on the ROI data to obtain coordinate marks; multiple rounds of iterative high-resolution mark screening are performed in multiple hierarchical self-attention layers based on the coordinate marks, high-resolution image marks, low-resolution image marks and text marks to obtain target high-resolution marks; hierarchical self-attention calculation is performed based on the target high-resolution mark, low-resolution image mark and text mark to obtain the target mark.

[0101] Model prediction module: used to obtain a first prediction answer based on the target tag through a large-scale language model.

[0102] Model training module: Based on the first predicted answer and the answer label, train the large-scale language model to obtain a target large-scale language model.

[0103] Data processing module: used to use the target large-scale language model to make predictions based on the input image and the input query text to obtain a second prediction answer.

[0104] Through the system provided in the second aspect, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in the knowledge graph, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval.

[0105] The present application also provides an electronic device, such as Figure 4 As shown, the electronic device 1400 includes:

[0106] one or more processors 1410;

[0107] The memory 1420 stores one or more programs. When the one or more programs are executed by one or more processors 1410, the one or more processors 1410 implement the remote sensing image processing system based on the knowledge graph provided in any embodiment of the present application.

[0108] The memory 1420, as a non-transient network system, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 1420 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1420 may optionally include a memory 1420 remotely arranged relative to the processor 1410, and these remote memories 1420 may be connected to the processor 1410 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0109] The memory 1420 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1420 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1420, and the processor 1410 calls and executes the methods of the embodiments of this application.

[0110] The processor 1410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0111] In some embodiments, the electronic device further comprises:

[0112] Input / output interface, used to realize information input and output;

[0113] Communication interface, used to realize communication interaction between this device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0114] A bus that transmits information between the various components of the device (e.g., the processor 1410, the memory 1420, the input / output interface, and the communication interface);

[0115] The processor 1410 , the memory 1420 , the input / output interface and the communication interface can be connected to each other in communication within the device via a bus.

[0116] An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, which are used to execute a remote sensing image processing system based on a knowledge graph provided by any embodiment of the present application.

[0117] An embodiment of the present application also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the remote sensing image processing system based on the knowledge graph provided by any embodiment of the present application.

[0118] The system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0119] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0120] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0121] The above describes some embodiments of the present application with reference to the accompanying drawings, but does not limit the scope of the present invention. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present application.

[0122] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0123] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0124] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0127] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A remote sensing image processing method based on knowledge graph, characterized in that: The method comprises: Acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples; Based on the query text and the spatiotemporal information, searching in a preset knowledge graph to obtain knowledge text; Based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag; Downsampling the high-resolution image samples to obtain low-resolution image samples; Inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image tag and a low-resolution image tag; Obtaining a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag; Based on the target tag, obtaining a first predicted answer through a large-scale language model; Based on the first predicted answer and the answer label, training the large-scale language model to obtain a target large-scale language model; The target large-scale language model is used to perform prediction based on the input image and the input query text to obtain a second predicted answer.

2. The method according to claim 1, characterized in that Based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag includes: The knowledge text is sorted according to the time accuracy, and is spliced ​​with the query text to obtain a spliced ​​text, and the spliced ​​text is input into the text encoder for text encoding to obtain a text tag.

3. The method according to claim 1, characterized in that The visual language model includes a self-attention module, a flexible attention module and a projection layer; The obtaining a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag comprises: Inputting the low-resolution image tag and the text tag into a self-attention module to perform self-attention calculation to obtain a first tag; Obtaining a second tag through a flexible attention module based on the first tag, the high-resolution image tag, the low-resolution image tag, and the text tag; The projection layer analyzes and processes the second mark to obtain a target mark.

4. The method according to claim 3, characterized in that The knowledge text includes ROI data, and the flexible attention module includes a hierarchical self-attention module and a high-resolution marker selection module; The obtaining a second tag through a flexible attention module based on the first tag, the high-resolution image tag, the low-resolution image tag and the text tag comprises: Perform coordinate calculation based on the ROI data to obtain coordinate marks; Adding the coordinate mark and the first mark to obtain a fusion mark; Based on the fusion mark, the high-resolution image mark, the low-resolution image mark and the text mark in the hierarchical self-attention module, the high-resolution mark selection module performs multiple rounds of high-resolution mark screening to obtain a target high-resolution mark; Based on the target high-resolution label, the low-resolution image label and the text label, a second label is obtained through a hierarchical self-attention module.

5. The method according to claim 4, characterized in that The hierarchical self-attention module includes a plurality of hierarchical self-attention layers; The step of performing multiple rounds of high-resolution tag screening in the hierarchical self-attention module based on the fusion tag, the high-resolution image tag, the low-resolution image tag and the text tag through the high-resolution tag selection module to obtain a target high-resolution tag includes: The fusion mark is screened by the high-resolution mark selection module at the first layer of the hierarchical self-attention layer to obtain the selected high-resolution mark; Based on the selected high-resolution mark, the low-resolution image mark and the text mark, a calculation is performed through the next layer of the hierarchical self-attention layer, and the new fusion mark is obtained by adding the coordinate mark to the selected high-resolution mark; Based on the new fusion mark, the high-resolution mark selection module performs high-resolution mark screening layer by layer in the multiple hierarchical self-attention layers to finally obtain the target high-resolution mark.

6. The method according to claim 1, characterized in that Based on the first predicted answer and the answer label, training the large-scale language model to obtain a target large-scale language model includes: A loss function is determined based on the first predicted answer and the answer label, and parameters of the large-scale language model are adjusted based on the loss function to obtain a target large-scale language model.

7. The method according to claim 6, characterized in that The step of adjusting the parameters of the large-scale language model based on the loss function comprises: The parameters of the large-scale language model are calculated based on the loss function and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimization algorithm.

8. A remote sensing image processing system based on knowledge graph, characterized in that: The system comprises: A training sample acquisition module is used to acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples; Retrieval module: used to search in a preset knowledge graph based on the query text and the spatiotemporal information to obtain knowledge text; A text encoding module: used for performing text encoding through a text encoder based on the query text and the knowledge text to obtain a text tag; Downsampling module: used for downsampling the high-resolution image samples to obtain low-resolution image samples; Image encoding module: used for inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image mark and a low-resolution image mark; A target tag acquisition module: used to obtain a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag; Model prediction module: used to obtain a first prediction answer based on the target tag through a large-scale language model; Model training module: training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model; Data processing module: used to use the target large-scale language model to make predictions based on the input image and the input query text to obtain a second prediction answer.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the remote sensing image processing method based on the knowledge graph as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the remote sensing image processing method based on the knowledge graph described in any one of claims 1 to 7 is implemented.

Images