A remote sensing image processing method, system, device and medium based on knowledge graph
By using the combination of spatial indexed knowledge graphs and visual language models in remote sensing tasks, the accuracy and efficiency of traditional large-scale language models in search of remote sensing knowledge in remote sensing tasks are solved, and more efficient remote sensing knowledge retrieval and processing are achieved.
Patent Information
- Application Number
- CN202411900339.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-08-29
- Estimated Expiration
- 2044-12-23
AI Technical Summary
Traditional large-scale language models are difficult to accurately retrieve remote sensing knowledge in remote sensing tasks, resulting in inaccuracy and inefficiency of remote sensing visual language models.
The spatial index knowledge graph is used to replace traditional keyword extraction. By obtaining high-resolution image samples and their spatio-temporal information, query text and answer labels, combining text encoding and image encoding, using visual language models and large-scale language models for training, optimize keyword extraction strategies to improve retrieval accuracy.
It improves the application accuracy and efficiency of large-scale language models in remote sensing visual language models, and can more accurately retrieve remote sensing knowledge in the knowledge graph, improving the accuracy and efficiency of knowledge retrieval.
Smart Images
Figure CN119782479B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of remote sensing technology and artificial intelligence technology, and in particular to a remote sensing image processing method, system, device and medium based on a knowledge graph. Background Art
[0002] In the fields of remote sensing and artificial intelligence, combining large-scale language models with retrieval-augmented generation techniques has become an effective means of improving model understanding and generation capabilities. This paradigm typically involves extracting keywords from input text questions and then using these keywords to efficiently search through a vast knowledge base or knowledge graph to obtain relevant background knowledge and information. In this way, the model can combine the retrieved knowledge to generate more accurate and rich answers or explanations.
[0003] However, in the specific application scenario of remote sensing, traditional large-scale language models combined with retrieval-enhanced generation techniques face significant challenges. Remote sensing problems often focus on the description and analysis of visual content, rather than simple linguistic expression. These problems typically contain relatively few extractable keywords, and these keywords are often imprecise, failing to accurately reflect the core visual features and requirements of the problem. Furthermore, the information in remote sensing imagery is complex and diverse, encompassing a wealth of specialized knowledge on geography, landforms, meteorology, and other topics. Extracting and interpreting this information places extremely high demands on the accuracy and comprehensiveness of keywords.
[0004] For these reasons, keywords extracted directly from remote sensing task questions often fail to effectively retrieve relevant and accurate knowledge from knowledge bases or knowledge graphs. This significantly limits the accuracy and effectiveness of large-scale language models combined with retrieval-enhanced generation techniques in remote sensing visual language models. Therefore, optimizing keyword extraction strategies based on the characteristics of remote sensing tasks and improving retrieval accuracy and efficiency has become a key issue that needs to be addressed in current remote sensing visual language model research. Summary of the Invention
[0005] The main purpose of the embodiments of this application is to propose a remote sensing image processing method, system, device and medium based on knowledge graph, aiming to use spatial index knowledge graph to replace traditional keyword extraction, to solve the problem that large-scale language models are difficult to query remote sensing knowledge, and to improve the accuracy of remote sensing knowledge retrieval in large-scale language models.
[0006] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a remote sensing image processing method based on a knowledge graph, the method comprising:
[0007] Acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples;
[0008] Based on the query text and the spatiotemporal information, searching in a preset knowledge graph to obtain knowledge text;
[0009] Based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag;
[0010] Downsampling the high-resolution image samples to obtain low-resolution image samples;
[0011] Inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image label and a low-resolution image label;
[0012] Obtaining a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag, and the text tag;
[0013] Based on the target tag, obtaining a first predicted answer through a large-scale language model;
[0014] Training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model;
[0015] The target large-scale language model is used to perform prediction based on the input image and the input query text to obtain a second predicted answer.
[0016] Through the method provided in the first aspect, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in knowledge graphs, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval.
[0017] In a possible implementation, performing text encoding by a text encoder based on the query text and the knowledge text to obtain a text tag includes:
[0018] The knowledge texts are sorted according to the time accuracy and spliced with the query text to obtain spliced texts, and the spliced texts are input into the text encoder for text encoding to obtain text tags.
[0019] In one possible implementation, the visual language model includes a self-attention module, a flexible attention module, and a projection layer;
[0020] The obtaining of a target tag by a visual language model based on the high-resolution image tag, the low-resolution image tag, and the text tag comprises:
[0021] Inputting the low-resolution image tag and the text tag into a self-attention module to perform self-attention calculation to obtain a first tag;
[0022] Obtaining a second label based on the first label, the high-resolution image label, the low-resolution image label, and the text label through a flexible attention module;
[0023] The projection layer analyzes and processes the second mark to obtain a target mark.
[0024] In one possible implementation, the knowledge text includes ROI data, and the flexible attention module includes a hierarchical self-attention module and a high-resolution marker selection module;
[0025] Obtaining a second label based on the first label, the high-resolution image label, the low-resolution image label, and the text label by a flexible attention module includes:
[0026] Performing coordinate calculation based on the ROI data to obtain coordinate marks;
[0027] Adding the coordinate mark and the first mark to obtain a fusion mark;
[0028] Based on the fusion mark, the high-resolution image mark, the low-resolution image mark and the text mark in the hierarchical self-attention module, the high-resolution mark selection module performs multiple rounds of high-resolution mark screening to obtain a target high-resolution mark;
[0029] A second label is obtained through a hierarchical self-attention module based on the target high-resolution label, the low-resolution image label and the text label.
[0030] In one possible implementation, the hierarchical self-attention module includes a plurality of hierarchical self-attention layers;
[0031] Based on the fusion mark, the high-resolution image mark, the low-resolution image mark, and the text mark in the hierarchical self-attention module, the high-resolution mark selection module performs multiple rounds of high-resolution mark screening to obtain target high-resolution marks, including:
[0032] The fusion mark is screened by the high-resolution mark selection module at the first layer of the hierarchical self-attention layer to obtain a selected high-resolution mark;
[0033] Based on the selected high-resolution mark, the low-resolution image mark and the text mark, a calculation is performed through the next layer of the hierarchical self-attention layer, and the new fusion mark is added to the coordinate mark to obtain a new fusion mark;
[0034] Based on the new fusion mark, the high-resolution mark selection module is used to screen high-resolution marks layer by layer in the multiple hierarchical self-attention layers, and finally obtain the target high-resolution mark.
[0035] In one possible implementation, training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model includes:
[0036] A loss function is determined based on the first predicted answer and the answer label, and parameters of the large-scale language model are adjusted based on the loss function to obtain a target large-scale language model.
[0037] In one possible implementation, the step of adjusting parameters of the large-scale language model based on the loss function includes:
[0038] The parameters of the large-scale language model are calculated based on the loss function and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimization algorithm.
[0039] To achieve the above objectives, a second aspect of an embodiment of the present application proposes a remote sensing image processing system based on a knowledge graph, the system comprising:
[0040] A training sample acquisition module is used to acquire training samples, wherein the training samples include high-resolution image samples and the spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples;
[0041] Retrieval module: used to search in a preset knowledge graph based on the query text and the spatiotemporal information to obtain knowledge text;
[0042] A text encoding module: configured to perform text encoding based on the query text and the knowledge text through a text encoder to obtain text tags;
[0043] Downsampling module: used for downsampling the high-resolution image samples to obtain low-resolution image samples;
[0044] Image encoding module: used for inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image label and a low-resolution image label;
[0045] A target tag acquisition module is configured to obtain a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag, and the text tag;
[0046] Model prediction module: used to obtain a first prediction answer based on the target tag through a large-scale language model;
[0047] Model training module: training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model;
[0048] Data processing module: used to use the target large-scale language model to make predictions based on the input image and the input query text to obtain a second prediction answer.
[0049] The system provided in the second aspect can overcome the limitations of existing large-scale language models in querying remote sensing knowledge, enabling large-scale language models to effectively retrieve relevant and accurate knowledge in knowledge graphs, improving the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improving the accuracy and efficiency of knowledge retrieval.
[0050] In a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the remote sensing image processing method based on the knowledge graph as described in any possible implementation method in the first aspect is implemented.
[0051] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the remote sensing image processing method based on the knowledge graph is implemented as described in any possible implementation method in the first aspect.
[0052] It can be seen from the technical solutions provided by one or more embodiments of the above specification that the remote sensing image processing method based on the knowledge graph provided by the embodiments of the present application obtains high-resolution image samples and their corresponding spatiotemporal information, query text, and answer labels, and searches in a preset knowledge graph based on the query text and spatiotemporal information to obtain knowledge text, performs text encoding on the query text and the knowledge text respectively, performs image encoding on the high-resolution image samples and the low-resolution images obtained by downsampling the high-resolution image samples, obtains target labels through a visual language model based on the obtained text labels, high-resolution image labels, and low-resolution image labels, obtains a first predicted answer through a large-scale language model based on the target labels, performs model training based on the first predicted answer and the answer label to obtain a target large-scale language model, uses the target large-scale language model to perform prediction based on the input image and input query text, and obtains a second predicted answer, thereby improving the accuracy of the model retrieval of remote sensing knowledge. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the description of one or more embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0054] Figure 1 This is a flow chart of a remote sensing image processing method based on a knowledge graph provided in an embodiment of the present application;
[0055] Figure 2 This is a schematic diagram of the system architecture of the visual language model provided in the embodiment of the present application;
[0056] Figure 3 This is an architecture diagram of a remote sensing image processing system based on a knowledge graph provided in an embodiment of the present application;
[0057] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0058] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the one or more embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.
[0059] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.
[0061] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0062] Figure 1 This is an optional flowchart of the remote sensing image processing method based on the knowledge graph provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S900.
[0063] First, as Figure 1 As shown, a remote sensing image processing method based on a knowledge graph is provided, the method comprising:
[0064] S100: Acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples.
[0065] S200: Based on the query text and spatiotemporal information, search in a preset knowledge graph to obtain knowledge text.
[0066] It should be noted that the spatiotemporal information is composed of the time information and spatial information of the high-resolution image samples, and the time information and spatial information include the time and geographic coordinates of the high-resolution image samples. The time and geographic coordinates of the high-resolution image samples are added to the query text, that is, the spatial index method is adopted to perform knowledge retrieval in the preset knowledge graph to obtain the knowledge text, wherein the knowledge text includes land use information, administrative district information and ROI information of the surface within the image coverage area. By adopting the form of spatial index knowledge graph to replace traditional keyword extraction, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in the knowledge graph, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval. In addition, it can be understood that the embodiments of the present application do not limit the type of knowledge text.
[0067] It should also be noted that by combining query text and spatiotemporal information and searching in the preset knowledge graph, the model can more accurately understand the user's query intention, more comprehensively capture the user's query intention and information needs, and find the remote sensing knowledge that best matches it, making full use of the visual features of remote sensing problems and improving the accuracy and efficiency of retrieval.
[0068] S300: Based on the query text and the knowledge text, perform text encoding through a text encoder to obtain a text tag.
[0069] S400 , down-sampling the high-resolution image samples to obtain low-resolution image samples.
[0070] S500 , inputting high-resolution image samples and low-resolution image samples into an image encoder respectively to obtain high-resolution image labels and low-resolution image labels.
[0071] S600, based on the high-resolution image tag, the low-resolution image tag and the text tag, a target tag is obtained through a visual language model.
[0072] S700: Based on the target tag, obtain a first predicted answer through a large-scale language model.
[0073] S800: Based on the first predicted answer and the answer label, train a large-scale language model to obtain a target large-scale language model.
[0074] S900: Use the target large-scale language model to perform prediction based on the input image and the input query text to obtain a second predicted answer.
[0075] Through the method provided in the first aspect, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in knowledge graphs, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval.
[0076] In a possible implementation, performing text encoding by a text encoder based on the query text and the knowledge text to obtain a text tag includes:
[0077] The knowledge texts are sorted according to the time accuracy and spliced with the query text to obtain spliced texts, and the spliced texts are input into the text encoder for text encoding to obtain text tags.
[0078] It should be noted that the knowledge text includes land use information, administrative district information, and ROI (Region of Interest) information on the surface within the image coverage area. Sorting the retrieved knowledge text by time accuracy can help the large-scale language model more comprehensively understand the text content, thereby enabling the large-scale language model to more accurately understand and process the knowledge text. In addition, it should be understood that the embodiments of this application do not limit the type of knowledge text.
[0079] like Figure 2 As shown, Figure 2 Schematic diagram of the system architecture of the visual language model provided in an embodiment of the present application. In one possible implementation, the visual language model includes a self-attention module, a flexible attention module, and a projection layer; obtaining a target tag through the visual language model based on the high-resolution image tag, the low-resolution image tag, and the text tag includes: inputting the low-resolution image tag and the text tag into the self-attention module for self-attention calculation to obtain a first tag; obtaining a second tag through the flexible attention module based on the first tag, the high-resolution image tag, the low-resolution image tag, and the text tag; and the projection layer analyzing and processing the second tag to obtain a target tag.
[0080] In some embodiments, the introduction of a self-attention module into the visual language model can enhance the model's ability to capture and understand the complex relationship between remote sensing images and text. It enables the model to dynamically allocate attention to key visual features in the image and core semantic information in the text, thereby more accurately interpreting the image content and generating matching text descriptions or answers. At the same time, the self-attention module promotes the deep fusion of image and text modal data, allowing the model to perform higher accuracy when handling cross-modal tasks. To improve computational efficiency and reduce the amount of computation, only low-resolution image tags and text tags are input into the self-attention module for self-attention calculation to obtain the first tag, which is used to roughly understand the entire image and improve the model's ability to understand the image. To more efficiently perceive more image details, a flexible attention module is introduced into the visual language model. It can perform a weighted average operation based on the context of the input features and dynamically determine the attention weight by calculating the similarity function between related pixel pairs. This mechanism enables the model to flexibly focus on different areas in the image, thereby capturing more feature information.
[0081] In one possible implementation, the knowledge text includes ROI data, and the flexible attention module includes a hierarchical self-attention module and a high-resolution tag selection module; based on the first tag, the high-resolution image tag, the low-resolution image tag and the text tag, obtaining the second tag through the flexible attention module includes: performing coordinate calculation based on the ROI data to obtain a coordinate tag; adding the coordinate tag and the first tag to obtain a fusion tag; based on the fusion tag, the high-resolution image tag, the low-resolution image tag and the text tag, performing multiple rounds of high-resolution tag screening in the hierarchical self-attention module through the high-resolution tag selection module to obtain a target high-resolution tag; based on the target high-resolution tag, the low-resolution image tag and the text tag, obtaining the second tag through the hierarchical self-attention module.
[0082] It should be noted that the current mainstream large-scale language models were primarily designed to process images of regular size. Their input size is usually limited by multiple factors such as model architecture, computing resources, and training efficiency, and is generally small. This size mismatch has become a major obstacle to directly applying advanced visual language models to remote sensing image analysis tasks. However, by introducing ROI data into the hierarchical self-attention module, multiple rounds of high-resolution tag selection are performed in the hierarchical self-attention module to obtain a target high-resolution tag. Based on the target high-resolution tag, low-resolution image tag, and text tag, a second tag is obtained through the hierarchical self-attention module. The projection layer analyzes and processes the second tag to obtain the target tag. Based on this target tag, the large-scale language model is used to predict the answer. In surface research, ROI data refers to the areas where the changes in surface morphology and cover types are most significant or are of the greatest research value. By introducing ROI data into the hierarchical self-attention module, the limitations of existing large-scale language models in processing large-format images can be overcome. While maintaining the integrity of remote sensing image information, large-scale language models can be enabled to process large-format images, significantly improving the analysis efficiency and depth of remote sensing data, allowing the rich information in the images to be captured and interpreted more comprehensively and accurately, promoting the widespread application of remote sensing technology in key areas such as environmental monitoring, urban planning, and disaster warning, providing more detailed and reliable data support for decision-making, and helping to accelerate data processing processes, improve automation and intelligence levels, thereby reducing the cost and errors of manual interpretation to a certain extent.
[0083] In one possible implementation, the hierarchical self-attention module includes multiple hierarchical self-attention layers; based on the fusion mark, the high-resolution image mark, the low-resolution image mark and the text mark, the hierarchical self-attention module performs multiple rounds of high-resolution mark screening through the high-resolution mark selection module to obtain the target high-resolution mark, including: the fusion mark is screened for high-resolution marks in the first layer of the hierarchical self-attention layer by the high-resolution mark selection module to obtain a selected high-resolution mark; based on the selected high-resolution mark, the low-resolution image mark and the text mark, the next layer of the hierarchical self-attention layer is calculated and added to the coordinate mark to obtain a new fusion mark; based on the new fusion mark, the high-resolution mark selection module is used to screen high-resolution marks layer by layer in the multiple hierarchical self-attention layers to finally obtain the target high-resolution mark.
[0084] It should be noted that, in some embodiments, the flexible attention module includes a high-resolution marker selection module and a hierarchical self-attention module. The high-resolution marker selection module is used to perform multiple rounds of high-resolution image marker screening in multiple hierarchical self-attention layers in the hierarchical self-attention module, and input the high-resolution markers selected in each layer into the next hierarchical self-attention layer, and calculate them with the low-resolution image markers and text markers. The calculated markers are added to the coordinate markers to obtain new fusion markers, which are used for high-resolution marker selection in the next hierarchical self-attention layer until the target high-resolution marker is selected. The purpose of high-resolution screening is to improve the model's ability to process high-resolution images while reducing unnecessary computational overhead to reduce the overall computational cost, thereby achieving more efficient and economical visual language model applications.
[0085] It should also be noted that the coordinate marks are obtained by coordinate calculation based on ROI data. In each layer of hierarchical self-attention layer, the marks calculated based on the selected high-resolution marks, low-resolution marks and text marks are added to the coordinate marks as the input of the next layer of hierarchical self-attention layer and used to screen the high-resolution marks. By introducing ROI data, the limitations of existing large-scale language models in processing large-format images can be overcome. While maintaining the integrity of remote sensing image information, large-scale language models can be used to process large-format images, significantly improving the analysis efficiency and depth of remote sensing data, so that the rich information in the images can be captured and interpreted more comprehensively and accurately, promoting the widespread application of remote sensing technology in key fields such as environmental monitoring, urban planning, and disaster warning, providing more detailed and reliable data support for decision-making, and helping to accelerate the data processing process, improve the level of automation and intelligence, thereby reducing the cost and error of manual interpretation to a certain extent.
[0086] In one possible implementation, training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model includes:
[0087] A loss function is determined based on the first predicted answer and the answer label, and parameters of the large-scale language model are adjusted based on the loss function to obtain a target large-scale language model.
[0088] In one possible implementation, the step of adjusting parameters of the large-scale language model based on the loss function includes:
[0089] The parameters of the large-scale language model are calculated based on the loss function and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimization algorithm.
[0090] It's important to note that the loss function is an important metric for measuring the difference between a model's predictions and the actual results. By calculating the loss value, one can intuitively understand the model's performance on the training data. As the loss value gradually decreases, it indicates that the large-scale language model is gradually approaching the distribution of the real data. At this point, the change in the loss value can be used to guide parameter adjustments. The backpropagation algorithm calculates the gradients between each layer in the network, propagating the error signal back from the output layer to the input layer layer by layer to obtain the gradient of each parameter. After obtaining the gradient of each parameter, the optimizer optimizes and updates the parameters. Through multiple iterations of backpropagation and parameter updates, the large-scale language model can better fit the training data, optimize its performance, improve its generalization ability, accelerate the model training process, and increase its efficiency in handling complex tasks.
[0091] To achieve the above purpose, Figure 3 As shown, the second aspect of the embodiment of the present application proposes a remote sensing image processing system based on a knowledge graph, the system comprising:
[0092] Training sample acquisition module: used to acquire training samples, wherein the training samples include high-resolution image samples and the spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples.
[0093] Retrieval module: used to search in a preset knowledge graph based on the query text and the spatiotemporal information to obtain knowledge text.
[0094] Among them, it should be noted that the spatiotemporal information is composed of the time information and spatial information of the high-resolution image samples, and the time information and spatial information include the time and geographic coordinates of the high-resolution image samples. The time and geographic coordinates of the high-resolution image samples are added to the query text, that is, the spatial index method is adopted to perform knowledge retrieval in the preset knowledge graph to obtain the knowledge text, wherein the knowledge text includes land use information, administrative district information and ROI information of the surface within the image coverage area. By adopting the form of spatial index knowledge graph to replace traditional keyword extraction, the limitations of existing large-scale language models in querying remote sensing knowledge can be overcome, so that large-scale language models can effectively retrieve relevant and accurate knowledge in the knowledge graph, improve the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improve the accuracy and efficiency of knowledge retrieval. In addition, it can be understood that the embodiments of the present application do not limit the type of knowledge text.
[0095] It should also be noted that by combining query text and spatiotemporal information and searching in the preset knowledge graph, the model can more accurately understand the user's query intention, more comprehensively capture the user's query intention and information needs, and find the remote sensing knowledge that best matches it, making full use of the visual features of remote sensing problems and improving the accuracy and efficiency of retrieval.
[0096] Text encoding module: used to perform text encoding based on the query text and the knowledge text through a text encoder to obtain text tags.
[0097] Downsampling module: used to downsample the high-resolution image samples to obtain low-resolution image samples.
[0098] Image encoding module: used for inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image label and a low-resolution image label.
[0099] A target tag acquisition module is configured to obtain a target tag through a visual language model based on the high-resolution image tag, the low-resolution image tag and the text tag.
[0100] It should be noted that the visual language model includes a self-attention module, which includes multiple hierarchical self-attention layers. The knowledge text includes ROI data, and coordinate calculation is performed based on the ROI data to obtain coordinate tags; based on the coordinate tags, high-resolution image tags, low-resolution image tags and text tags, multiple rounds of iterative high-resolution tag screening are performed in multiple hierarchical self-attention layers to obtain target high-resolution tags; hierarchical self-attention calculation is performed based on the target high-resolution tags, low-resolution image tags and text tags to obtain the target tags.
[0101] Model prediction module: used to obtain a first prediction answer based on the target tag through a large-scale language model.
[0102] Model training module: Based on the first predicted answer and the answer label, train the large-scale language model to obtain a target large-scale language model.
[0103] Data processing module: used to use the target large-scale language model to make predictions based on the input image and the input query text to obtain a second prediction answer.
[0104] The system provided in the second aspect can overcome the limitations of existing large-scale language models in querying remote sensing knowledge, enabling large-scale language models to effectively retrieve relevant and accurate knowledge in knowledge graphs, improving the application accuracy and effect of large-scale language models combined with retrieval enhancement generation technology in remote sensing visual language models, and improving the accuracy and efficiency of knowledge retrieval.
[0105] The present application also provides an electronic device, such as Figure 4 As shown, the electronic device 1400 includes:
[0106] one or more processors 1410;
[0107] The memory 1420 stores one or more programs. When the one or more programs are executed by one or more processors 1410, the one or more processors 1410 implement the remote sensing image processing system based on the knowledge graph provided in any embodiment of the present application.
[0108] The memory 1420 is a non-transient network system that can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 1420 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1420 may optionally include a memory 1420 remotely located relative to the processor 1410, and these remote memories 1420 may be connected to the processor 1410 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0109] The memory 1420 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1420 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1420 and is called by the processor 1410 to execute the methods of the embodiments of this application.
[0110] The processor 1410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.
[0111] In some embodiments, the electronic device further comprises:
[0112] Input / output interface, used to realize information input and output;
[0113] Communication interface, used to enable communication between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);
[0114] A bus that transmits information between various components of the device (e.g., the processor 1410, memory 1420, input / output interfaces, and communication interfaces);
[0115] The processor 1410 , the memory 1420 , the input / output interface, and the communication interface can be communicatively connected to each other within the device via a bus.
[0116] An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, which are used to execute the knowledge graph-based remote sensing image processing system provided by any embodiment of the present application.
[0117] An embodiment of the present application also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. The processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the remote sensing image processing system based on the knowledge graph provided by any embodiment of the present application.
[0118] The system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.
[0119] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0120] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0121] The above description of some embodiments of the present application with reference to the accompanying drawings does not limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present application.
[0122] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.
[0123] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0124] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0125] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.
[0126] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0127] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.
Claims
1. A remote sensing image processing method based on knowledge graph, characterized in that: The method comprises: Acquire training samples, wherein the training samples include high-resolution image samples and spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples; Based on the query text and the spatiotemporal information, searching in a preset knowledge graph to obtain knowledge text; Based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag; Downsampling the high-resolution image samples to obtain low-resolution image samples; Inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image label and a low-resolution image label; Acquire a visual language model, wherein the visual language model includes a self-attention module, a flexible attention module, and a projection layer; Inputting the low-resolution image label and the text label into a self-attention module for self-attention calculation to obtain a first label; obtaining a second label through a flexible attention module based on the first label, the high-resolution image label, the low-resolution image label, and the text label; and analyzing and processing the second label by the projection layer to obtain a target label; Based on the target tag, obtaining a first predicted answer through a large-scale language model; Training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model; The target large-scale language model is used to perform prediction based on the input image and the input query text to obtain a second predicted answer.
2. The method according to claim 1, characterized in that Based on the query text and the knowledge text, performing text encoding by a text encoder to obtain a text tag includes: The knowledge texts are sorted according to the time accuracy and spliced with the query text to obtain spliced texts, and the spliced texts are input into the text encoder for text encoding to obtain text tags.
3. The method according to claim 1, characterized in that The knowledge text includes ROI data, and the flexible attention module includes a hierarchical self-attention module and a high-resolution marker selection module; The obtaining, by a flexible attention module, a second tag based on the first tag, the high-resolution image tag, the low-resolution image tag, and the text tag comprises: Perform coordinate calculation based on the ROI data to obtain coordinate marks; Adding the coordinate mark and the first mark to obtain a fusion mark; Based on the fusion mark, the high-resolution image mark, the low-resolution image mark and the text mark in the hierarchical self-attention module, the high-resolution mark selection module performs multiple rounds of high-resolution mark screening to obtain a target high-resolution mark; A second label is obtained through a hierarchical self-attention module based on the target high-resolution label, the low-resolution image label and the text label.
4. The method according to claim 3, characterized in that The hierarchical self-attention module includes a plurality of hierarchical self-attention layers; The step of performing multiple rounds of high-resolution marker screening in the hierarchical self-attention module based on the fusion marker, the high-resolution image marker, the low-resolution image marker, and the text marker by the high-resolution marker selection module to obtain a target high-resolution marker includes: The fusion mark is screened by the high-resolution mark selection module at the first layer of the hierarchical self-attention layer to obtain a selected high-resolution mark; Based on the selected high-resolution mark, the low-resolution image mark and the text mark, a calculation is performed through the next layer of the hierarchical self-attention layer, and the new fusion mark is added to the coordinate mark to obtain a new fusion mark; Based on the new fusion mark, the high-resolution mark selection module is used to screen high-resolution marks layer by layer in the multiple hierarchical self-attention layers, and finally obtain the target high-resolution mark.
5. The method according to claim 1, wherein Training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model includes: A loss function is determined based on the first predicted answer and the answer label, and parameters of the large-scale language model are adjusted based on the loss function to obtain a target large-scale language model.
6. The method according to claim 5, characterized in that The step of adjusting parameters of the large-scale language model based on the loss function includes: The parameters of the large-scale language model are calculated based on the loss function and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimization algorithm.
7. A remote sensing image processing system based on knowledge graph, characterized in that: The system comprises: A training sample acquisition module is used to acquire training samples, wherein the training samples include high-resolution image samples and the spatiotemporal information, query text, and answer labels corresponding to the high-resolution image samples; Retrieval module: used to search in a preset knowledge graph based on the query text and the spatiotemporal information to obtain knowledge text; A text encoding module: configured to perform text encoding based on the query text and the knowledge text through a text encoder to obtain text tags; Downsampling module: used for downsampling the high-resolution image samples to obtain low-resolution image samples; Image encoding module: used for inputting the high-resolution image sample and the low-resolution image sample into an image encoder respectively to obtain a high-resolution image label and a low-resolution image label; A visual language model acquisition module is used to acquire a visual language model, wherein the visual language model includes a self-attention module, a flexible attention module, and a projection layer; Target label acquisition module: used to input the low-resolution image label and the text label into the self-attention module for self-attention calculation to obtain a first label; based on the first label, the high-resolution image label, the low-resolution image label and the text label, obtain a second label through the flexible attention module; the projection layer analyzes and processes the second label to obtain a target label; Model prediction module: used to obtain a first prediction answer based on the target tag through a large-scale language model; Model training module: training the large-scale language model based on the first predicted answer and the answer label to obtain a target large-scale language model; Data processing module: used to use the target large-scale language model to make predictions based on the input image and the input query text to obtain a second prediction answer.
8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the remote sensing image processing method based on the knowledge graph described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the remote sensing image processing method based on the knowledge graph according to any one of claims 1 to 6 is implemented.