Remote sensing image searching method and device and computer readable storage medium

Abstract

The invention provides a remote sensing image search method and device and a computer readable storage medium, a wide area remote sensing image and a language instruction describing a search target are acquired, and cross-modal semantic information of the wide area remote sensing image and the language instruction is fused to realize semantic alignment, so that the search process is guided by using the language instruction, and the search efficiency is improved. The target sub-region discovery efficiency and accuracy under mass data and a complex environment are remarkably improved, and a basis is provided for efficient positioning of a target. The to-be-queried sequences of a plurality of sub-regions in the wide-area remote sensing image are dynamically planned under the constraint of limited resources, high-value regions are preferentially explored, and search and utilization are effectively balanced. When a sequence to be queried is executed, corresponding sub-regions in a wide-area remote sensing image are queried in sequence, a sub-region set which is matched with a language instruction and contains a target is finally output, the response speed and the intelligent level of a system are improved on the whole through the multi-modal information cooperation capability, and the method is suitable for space-based earth observation and other resource-limited scenes.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a remote sensing image search method, device and computer-readable storage medium. Background Technology

[0002] With the continuous improvement of spatial, temporal, and spectral resolution, the scale of image data generated by Earth observation systems has increased dramatically. This poses a significant challenge to the traditional "space-based sensing, ground-based computing, and human decision-making" data processing model: the pressure on network transmission and ground information processing systems has increased dramatically, resulting in long mission response cycles, low data utilization, and insufficient intelligence in mission management, ultimately limiting the system's application effectiveness in rapidly changing environments. Large-scale data loses its timeliness due to transmission and processing delays, causing a huge waste of resources. At the same time, existing observation systems lack autonomy and adaptability, and mission planning and execution are significantly lagging, especially in responding to emergencies such as earthquakes and debris flows, where the inability to obtain the latest image data in a timely manner often leads to irreparable losses. Advances in space technology have created an urgent need for the "space-based sensing, space-based computing, and space-based decision-making" model. This model directly deploys information processing modules on an intelligent satellite constellation, transforming "ground-based" into "space-based," and actively executes image processing algorithms on-orbit through intelligent algorithms, significantly reducing the transmission load on the space-ground network and ultimately improving the flexibility and response speed of mission execution. However, how to quickly locate key areas in massive wide-area remote sensing images has become one of the key technical challenges of space-based Earth observation systems. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this specification provides remote sensing image search methods, devices, and computer-readable storage media.

[0004] Firstly, a remote sensing image search method is provided, the method comprising: Acquire wide-area remote sensing images and language instructions describing the search targets; By fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions, a query sequence of multiple sub-regions in the wide-area remote sensing image is dynamically planned under preset resource constraints. The query sequence is executed, and the corresponding sub-regions in the wide-area remote sensing image are queried in turn. The set of sub-regions containing the target that match the language command is output.

[0005] According to the remote sensing image search method provided in this application, the step of dynamically planning a query sequence of multiple sub-regions in the wide-area remote sensing image under preset resource constraints by fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions includes: Semantic alignment is performed on the wide-area remote sensing image and the language instructions to obtain a fused multimodal feature representation; Based on the multimodal feature representation, historical search results, and remaining resources, the current search state is constructed; Under preset resource constraints, a query sequence of multiple sub-regions in the wide-area remote sensing image is generated based on the current search state.

[0006] According to the remote sensing image search method provided in this application, the step of generating a query sequence of multiple sub-regions in the wide-area remote sensing image based on the current search state is achieved by a pre-trained search strategy model; The search strategy model is configured to be trained through reinforcement learning to maximize cumulative rewards under resource constraints, the rewards being determined based on whether the target exists within the queried sub-region.

[0007] According to the remote sensing image search method provided in this application, the method further includes: Based on the semantic features of multiple sub-regions in the wide-area remote sensing image, the image space is clustered to construct a graph model representing the relationship between the clustered regions; Graph-guided features for macroscopic search are generated based on the graph model; The construction of the current search state includes: The current search state is constructed based on the multimodal feature representation, the graph-guided features, historical search results, and remaining resources.

[0008] According to the remote sensing image search method provided in this application, the nodes of the graph model represent multiple clustering regions obtained by clustering the image space, and the edges of the graph model represent the association relationships between the clustering regions.

[0009] According to the remote sensing image search method provided in this application, the feature representation of the nodes in the graph model is dynamically updated based on the historical search results during the query process.

[0010] According to the remote sensing image search method provided in this application, the resource constraints include query count constraints and / or movement cost constraints between sub-regions.

[0011] According to a remote sensing image search method provided in this application, the mobility cost constraint is determined based on the Manhattan distance between sub-regions.

[0012] In a second aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the remote sensing image search method as described in the first aspect.

[0013] Thirdly, a computer-readable storage medium is provided, on which a remote sensing image search program is stored, wherein the remote sensing image search program, when executed, implements any of the remote sensing image search methods described in the first aspect above.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the remote sensing image search method as described in any of the above.

[0015] The remote sensing image search method, device, and computer-readable storage medium described in this specification have the following advantages compared to the current difficulty in quickly locating specific targets in wide-area remote sensing images: Firstly, it acquires wide-area remote sensing images and language instructions describing the search target. By fusing cross-modal semantic information of the wide-area remote sensing images and language instructions, semantic alignment is achieved. This allows the language instructions to guide the search process, significantly improving the efficiency and accuracy of target sub-region discovery in massive data and complex environments, and providing a foundation for efficient target localization.

[0016] Secondly, under limited resource constraints, the system dynamically plans query sequences for multiple sub-regions within a wide-area remote sensing image, prioritizing the exploration of high-value areas and effectively balancing search and utilization. When executing the query sequence, the system sequentially queries the corresponding sub-regions within the wide-area remote sensing image, ultimately outputting a set of sub-regions containing the target that matches the verbal command. Through multimodal information collaboration capabilities, the system's overall response speed and intelligence level are improved, making it suitable for resource-constrained scenarios such as space-based Earth observation.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0019] Figure 1 This is a flowchart illustrating a remote sensing image search method according to an exemplary embodiment of this specification.

[0020] Figure 2 This is a flowchart illustrating a first embodiment of a remote sensing image search method according to an exemplary embodiment of this specification.

[0021] Figure 3 This is a flowchart illustrating a second embodiment of a remote sensing image search method according to an exemplary embodiment of this specification.

[0022] Figure 4This is a schematic diagram of a remote sensing image search device illustrated in this specification according to an exemplary embodiment.

[0023] Figure 5 This is a schematic block diagram of a remote sensing image search device according to an exemplary embodiment of this specification. Detailed Implementation

[0024] The technical solutions in the embodiments (or "implementations") of this application will be clearly and completely described herein with reference to the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0025] If the embodiments of this application contain terms relating to directional indications or positional relationships (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the attached figures); if the specific posture changes, the directional indications or positional relationships will also change accordingly. Furthermore, the terms "first" and "second" used in the embodiments of this application are only for descriptive convenience and should not be construed as indicating or implying relative importance.

[0026] This application provides a remote sensing image search method, apparatus, and computer-readable storage medium. The application will be described in detail below with reference to the accompanying drawings. The features described in the embodiments and implementations can be combined with each other.

[0027] Wide-area remote sensing images typically cover a wide range of ground features, for example, with a size of 30,000*30,000 pixels. Quickly locating specific targets (such as vehicles, buildings, or disaster areas) in wide-area remote sensing images is an extremely challenging task.

[0028] Existing non-short-sighted search techniques utilize deep reinforcement learning-based visual active exploration frameworks for wide-area geospatial exploration, incorporating meta-learning methods to enhance the model's adaptability and efficiency in new tasks. However, these methods rely on visual cues, limiting their exploration capabilities in complex environments and resulting in low search efficiency.

[0029] To address the aforementioned technical problems, this specification provides a remote sensing image search method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a remote sensing image search method provided in the embodiments of this specification.

[0030] The aim is to guide the visual search process with language commands and dynamically program the search path through reinforcement learning, so as to achieve efficient target localization in wide-area remote sensing images under resource constraints.

[0031] In other words, by using VLAS (Visual-Language Active Search) technology, guided by specific language instructions, sub-regions in a wide-area remote sensing image are sequentially screened according to priority, maximizing the coverage of as many target objects as possible while overcoming budget or resource constraints. Furthermore, the inherent spatial correlations of neighboring sub-regions can also provide important clues for the search process. This method balances exploration (improving model efficiency) and utilization (discovering more targets) under resource constraints by integrating machine learning models that predict target labels with customized algorithmic strategies.

[0032] It should be noted that the remote sensing image search scheme in this paper is directly deployed on the embodied intelligent satellite constellation, transforming it from "ground-based" to "space-based". Through intelligent algorithms, the remote sensing image search algorithm is actively executed in orbit, which significantly reduces the transmission load of the space-ground network and ultimately improves the flexibility and response speed of mission execution.

[0033] This application provides a first embodiment of a remote sensing image search method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a remote sensing image search method provided in this specification.

[0034] Specifically, this includes the following steps 101 to 103: In step 101, a wide-area remote sensing image and language instructions describing the search target are acquired.

[0035] First, initialize the search scenario and parameters. This includes the following steps: Acquire wide-area remote sensing images, divide the wide-area remote sensing images into grid cells, and obtain multiple sub-regions. ,in, The total number of grid cells into which the image is divided, each This represents a sub-region in a wide-area image. Subsequently, key sub-regions containing the target are selected from multiple sub-regions to achieve efficient target localization.

[0036] At the same time, obtain language instructions used to describe the search target. .

[0037] The language instructions described in this paper are used to clearly define the visual targets to be searched. Their semantic content includes, but is not limited to, descriptions of the target object category, its optional visual attributes, spatial relationships, or scene context, such as "identify the area where large vehicles exist." In remote sensing impact search, this language instruction is used as prior knowledge to guide the semantic understanding and decision-making direction of the entire subsequent visual search process.

[0038] As an example, the language instructions may be, but are not limited to, standard instructions generated from text, speech, structured templates such as forms, etc.

[0039] Furthermore, preset resource constraints are set for the search.

[0040] As an example, the resource constraints include query count constraints and / or movement cost constraints between sub-regions.

[0041] Preset total query budget and initialize historical search results. Remaining query budget and time step The query budget needs to comprehensively consider the number of query executions and the movement costs between the preceding and following sub-regions to be queried. The movement cost constraint is determined based on the Manhattan distance between the sub-regions to be queried.

[0042] By setting the above parameters, it is possible to prioritize the exploration of high-value areas under limited resources and budget constraints, and effectively balance exploration and utilization.

[0043] Step 102: By fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions, a query sequence of multiple sub-regions in the wide-area remote sensing image is dynamically planned under preset resource constraints.

[0044] Faced with tens to hundreds of terabytes of data generated daily by global satellite networks, the VLAS presented in this paper, through multimodal information fusion, eliminates the need for random searches. By simply inputting specific language commands, the system integrates visual observations from UAVs or remote sensing satellites with human textual cues, prioritizing the exploration of high-probability areas. This allows the team to focus on the sub-regions most likely to discover targets. In other words, it can quickly locate key sub-regions in complex wide-area images with massive amounts of data and limited computing power, significantly optimizing resource allocation for subsequent tasks (such as target detection).

[0045] In some embodiments, the step of dynamically planning a query sequence for multiple sub-regions in the wide-area remote sensing image under preset resource constraints by fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions includes the following steps 1021 to 1023: In step 1021, the wide-area remote sensing image and the language instructions are semantically aligned to obtain a fused multimodal feature representation.

[0046] As an example, via CLIP or other basic cross-modal encoders Image features and language instruction features are extracted separately, and multimodal semantic fusion is achieved through feature alignment.

[0047] Specifically, regarding images and language instructions Encode: Image Encoding as image features Preserve spatial location information.

[0048] Language instructions Encoding as instruction features ,in For language feature dimensions.

[0049] In step 1022, the current search state is constructed based on the multimodal feature representation, historical search results, and remaining resources.

[0050] Fusion image features Instruction characteristics Historical search results and remaining budget Generate the initial state .

[0051] The historical search results use a three-value label to record three states of the sub-region: searched and the target exists, searched but the target does not exist, and not searched.

[0052] In step 1023, under preset resource constraints, a query sequence of multiple sub-regions in the wide-area remote sensing image is generated based on the current search state.

[0053] In some embodiments, the generation of a query sequence of multiple sub-regions in the wide-area remote sensing image based on the current search state is achieved through a pre-trained search strategy model; The search strategy model described in the text outputs an action based on the current search state to select the sub-region to be queried. For ease of description, the search strategy model using VLAS will be referred to as the controller. express.

[0054] For example, at time step Through the controller Based on the current state Generate Actions Select the sub-region number to be queried. This represents the number of sub-regions queried in each query. It can be a decision neural network that takes the current state as input and outputs the action, i.e., the action. .

[0055] Step 103: Execute the query sequence, sequentially query the corresponding sub-regions in the wide-area remote sensing image, and output a set of sub-regions containing the target that match the language command.

[0056] The actions in the previously obtained query sequence In sub-region selection operations, based on the sub-region numbers given in the query sequence, the image data of each specified sub-region in the sequence is processed sequentially. For example, for each sub-region, a visual perception model is invoked to analyze it, determining whether its content matches the search target described by the natural language instruction, and generating a binary query result (target exists / target does not exist). For example, the visual perception model can be, but is not limited to, object detection models (such as YOLO, Faster R-CNN) or image classification models.

[0057] After the entire query sequence has been executed, the system integrates all successful query results and generates a set of location information for all sub-regions that hit the target, which serves as the final output of this active search task. This set of location information may, but is not limited to, a list of coordinates or a set of sub-region bounding boxes.

[0058] In some embodiments, the search strategy model is configured to be trained by reinforcement learning to maximize cumulative reward under resource constraints, the reward being determined based on whether a target exists within the queried subregion.

[0059] Actions When applied to a sub-region selection, perform the following operations: a. Obtain instant rewards. Based on the presence of a target marker. Calculate rewards .in, This indicates that the target exists. This indicates that it does not exist.

[0060] b. Update historical search results. For explored sub-regions, i.e., their serial numbers... ,set up The new historical search results are .

[0061] That is, the sub-regions corresponding to the target that are confirmed to exist after the query. When it is confirmed that it does not exist If no search is performed .

[0062] c. Update the remaining budget. (Based on Manhattan distance) Calculate movement costs and update the remaining budget. ,in The sub-region selected in the previous step.

[0063] d. Update the current search status. Generate a new status. .

[0064] Next, training data is collected. Transitional tuples are recorded. , for targeting Optimization.

[0065] The search process described above terminates when a termination condition is met. As an example, if the remaining budget... If the search fails, the search process will terminate; otherwise, the search will terminate. Then return to step 102 to continue the iteration.

[0066] In some embodiments, the search path is optimized using a reinforcement learning strategy.

[0067] As an example, a controller is jointly trained using reinforcement learning (RL) and supervised learning (SL).

[0068] First, calculate the loss value of the loss function.

[0069] Based on collected transition tuples Calculate reinforcement learning loss and supervised learning loss Weighted sum: ,in This is a hyperparameter.

[0070] Next, update the controller. parameter.

[0071] Based on the calculated loss value, the tactical controller is updated using the backpropagation algorithm. and basic backbone network The parameters are as follows. The gradient is calculated as... To maximize accumulated rewards.

[0072] Finally, by iteratively performing sub-region selection, strategy optimization, and controller parameter updates, the final output is the target maximization discovery result that meets the budget constraints.

[0073] That is, output serialization sub-region selection. ,in Indicates the first The selected set of sub-regions ultimately achieves the maximum discovery of the target object under budget constraints.

[0074] Through the above embodiments, wide-area remote sensing images and language instructions describing the search target are obtained. Semantic alignment is achieved by mapping image sub-regions and language instructions to the same semantic space. Based on this, a reinforcement learning agent (controller) is constructed. This system determines which sub-region to query next based on the fused multimodal state (image features, language features, historical search state, and remaining budget), and continuously optimizes the search strategy through reward signals. Ultimately, it achieves rapid localization of the target sub-region. Simultaneously, the fusion of cross-modal semantic information from wide-area remote sensing images and language commands significantly enhances adaptability to complex scenes, enabling efficient target localization in complex wide-area images.

[0075] This application provides a second embodiment of a remote sensing image search method, referring to... Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of a remote sensing image search method provided in this specification.

[0076] The proposed method improves upon the basic Visual-Language Active Search (VLAS) method by implementing a graph-augmented Visual-Language Active Search (PAGE) method. It achieves efficient target sub-region localization in complex scenes by introducing a hierarchical controller architecture and a dynamic graph model. For ease of description, the search strategy model employing VLAS will be referred to as the controller. express.

[0077] Specifically, this includes the following steps 201 to 203: Step 201: Obtain wide-area remote sensing images and language instructions describing the search target.

[0078] Receive and segment aerial or satellite images Input language commands ,in The total number of grid cells into which the image is divided, each It represents a sub-region in a wide-area image.

[0079] Set the preset total query budget and initialize historical search results. Remaining query budget Initial Queryed Sub-Region Features and time step The query budget needs to take into account both the number of query executions and the Manhattan distance between the preceding and following sub-regions.

[0080] Step 202: By fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions, a query sequence of multiple sub-regions in the wide-area remote sensing image is dynamically planned under preset resource constraints.

[0081] First, through CLIP or other basic cross-modal encoders For images and language instructions Encode: Image Encoding as image features Preserve spatial location information.

[0082] Language instructions Encoding as instruction features ,in For language feature dimensions.

[0083] Next, a knowledge graph based on visual language data is constructed. , where nodes Represents the mean characteristics of clustered sub-regions, and the edges It represents the feature similarity between classes. Knowledge graphs can model the commonalities among features in sub-regions, thereby improving the accuracy of proactive search.

[0084] In some embodiments, the image space is clustered according to the semantic features of multiple sub-regions in the wide-area remote sensing image to construct a graph model representing the relationship between clustered regions; graph-guided features for macro-search are generated based on the graph model; and the current search state is constructed based on the multimodal feature representation, the graph-guided features, historical search results, and remaining resources.

[0085] In this graph model, nodes represent multiple clustered regions obtained by clustering the image space, and edges represent the relationships between the clustered regions.

[0086] Specifically, the construction process employs clustering region partitioning: the K-Means algorithm is used to cluster the semantic features of all sub-regions of all images in the training set, thus partitioning them. There are 1 cluster region, whose numbers are as follows: .

[0087] Graph Node Defined as each cluster region Average image features of all sub-regions and average language features Composition. Among them, The mean feature is obtained by extracting pixels from sub-regions and encoding them using a cross-modal encoder. Extracted from sub-region categories via a cross-modal encoder.

[0088] Simultaneously, the adjacency probability between clustered regions is calculated using the Sinkhorn-Knopp algorithm, i.e., the average probability that sub-regions are physically adjacent, thereby generating a normalized adjacency matrix, which is ultimately defined as the edge of the graph model. Ultimately, based on the graph Constructing graph convolutional networks Using convolutional networks to process the graph structure, based on the features of the queried sub-region. Extracting graph-guided features .

[0089] Subsequently, image features are fused. Instruction characteristics Graph-guided features Historical search results and remaining budget Generate the current search status .

[0090] Under preset resource constraints, a query sequence of multiple sub-regions in the wide-area remote sensing image is generated based on the current search state.

[0091] As an example, we first calculate the features of the sub-regions.

[0092] At time step , will the The set of selected sub-regions All sub-regions are based on Encode and calculate the feature mean to obtain the sub-region features. .

[0093] Then identify the corresponding node numbers of the current and target sub-regions in the graph.

[0094] The current area is ,in Euclidean distance; target area is ,in It is a one-hot vector representing the category of the target object to be searched in the language instructions.

[0095] At time step Hierarchical controller Based on the current search status Generate Actions The process involves selecting the sub-region numbers to be queried, thus forming a query sequence. This represents the number of sub-regions queried each time.

[0096] also, It is a decision neural network that can be used through graph convolutional networks. Computational graph features Then calculate the sub-region selection action according to the VLAS method. .

[0097] Step 203: Execute the query sequence, sequentially query the corresponding sub-regions in the wide-area remote sensing image, and output a set of sub-regions containing the target that match the language command.

[0098] Actions When applied to a sub-region selection, perform the following operations: Get an instant reward. Based on the presence of a target marker. Calculate rewards .in, This indicates that the target exists. This indicates that it does not exist.

[0099] Update historical search results. For explored sub-regions, i.e., their sequence numbers. ,set up The new historical search results are In summary, the sub-regions where the target is confirmed to exist after the query correspond to... When it is confirmed that it does not exist If no search is performed .

[0100] Update remaining budget. Based on Manhattan distance. Calculate movement costs and update the remaining budget. ,in The sub-region selected in the previous step.

[0101] Update state, generate new state .

[0102] The feature representations of the nodes in the graph model are dynamically updated based on historical search results during the query process. Specifically, this is done based on the features of the selected sub-region. The average image features of the cluster centers to which the current sub-region belongs are adjusted in real time. .in To adjust the hyperparameters, they are generally set to a positive real number close to 0.

[0103] If the remaining budget If the search fails, the search process will terminate; otherwise, the search will terminate. Then return to step S301 to continue the iteration.

[0104] During the search process, training data is collected. Transitional tuples are recorded. , for targeting Optimization.

[0105] The specific optimization process is as follows: Based on collected transition tuples Calculate reinforcement learning loss and supervised learning loss weighted sum ,in This is a hyperparameter.

[0106] Update the controller using the backpropagation algorithm. and basic backbone network The parameters and gradient calculation are as follows: To maximize accumulated rewards.

[0107] Through the above process, the serialized sub-region selection is output. ,in Indicates the first The selected set of sub-regions ultimately achieves the maximum discovery of the target object under budget constraints.

[0108] Through the above embodiments, graph structure modeling and hierarchical controllers are introduced on top of VLAS. Macro-regional graphs are constructed through clustering, enhancing semantic understanding and strategic planning capabilities for complex scenes. More detailed decision-making is achieved in complex wide-area images, significantly improving the efficiency and accuracy of target localization.

[0109] This application provides a remote sensing image search method, device, and computer-readable storage medium, which has the following advantages compared with the current difficulty in quickly locating specific targets in wide-area remote sensing images: Firstly, it acquires wide-area remote sensing images and language instructions describing the search target. By fusing cross-modal semantic information of the wide-area remote sensing images and language instructions, semantic alignment is achieved. This allows the language instructions to guide the search process, significantly improving the efficiency and accuracy of target sub-region discovery in massive data and complex environments, and providing a foundation for efficient target localization.

[0110] Secondly, under limited resource constraints, the system dynamically plans query sequences for multiple sub-regions within a wide-area remote sensing image, prioritizing the exploration of high-value areas and effectively balancing search and utilization. When executing the query sequence, the system sequentially queries the corresponding sub-regions within the wide-area remote sensing image, ultimately outputting a set of sub-regions containing the target that matches the verbal command. Through multimodal information collaboration capabilities, the system's overall response speed and intelligence level are improved, making it suitable for resource-constrained scenarios such as space-based Earth observation.

[0111] Based on the same concept as the methods described above, this application also proposes a remote sensing image search device. For example... Figure 4 As shown, the device includes: The information acquisition module is used to acquire wide-area remote sensing images and language instructions describing the search target; The feature fusion module is used to dynamically plan the query sequence of multiple sub-regions in the wide-area remote sensing image under preset resource constraints by fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions. The target search module is used to execute the query sequence, sequentially query the corresponding sub-regions in the wide-area remote sensing image, and output a set of sub-regions containing the target that match the language command.

[0112] The specific implementation process of the functions and roles of each module / submodule / unit in the above device can be found in the implementation process of the corresponding steps in the above method, which can achieve the same technical effect, and will not be repeated here.

[0113] This application also provides a vehicle controller for implementing the remote sensing image search method described above.

[0114] Figure 5 An example is a schematic diagram of the physical structure of a remote sensing image search device, such as... Figure 5 As shown, the remote sensing image search device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions stored in the memory 530 to execute the remote sensing image search method.

[0115] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the remote sensing image search methods provided by the above methods.

[0117] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the remote sensing image search methods provided by the methods described above.

[0118] It should be noted that the technical solutions or features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A remote sensing image search method, characterized in that, The method includes: Acquire wide-area remote sensing images and language instructions describing the search targets; By fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions, a query sequence of multiple sub-regions in the wide-area remote sensing image is dynamically planned under preset resource constraints. The query sequence is executed, and the corresponding sub-regions in the wide-area remote sensing image are queried in turn. The set of sub-regions containing the target that match the language command is output.

2. The remote sensing image search method as described in claim 1, characterized in that, The step of dynamically planning a query sequence for multiple sub-regions in the wide-area remote sensing image under preset resource constraints by fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions includes: Semantic alignment is performed on the wide-area remote sensing image and the language instructions to obtain a fused multimodal feature representation; Based on the multimodal feature representation, historical search results, and remaining resources, the current search state is constructed; Under preset resource constraints, a query sequence of multiple sub-regions in the wide-area remote sensing image is generated based on the current search state.

3. The remote sensing image search method as described in claim 2, characterized in that, The generation of query sequences for multiple sub-regions in the wide-area remote sensing image based on the current search state is achieved through a pre-trained search strategy model; The search strategy model is configured to be trained through reinforcement learning to maximize cumulative rewards under resource constraints, the rewards being determined based on whether the target exists within the queried sub-region.

4. The remote sensing image search method as described in claim 2, characterized in that, The method further includes: Based on the semantic features of multiple sub-regions in the wide-area remote sensing image, the image space is clustered to construct a graph model representing the relationship between the clustered regions; Graph-guided features for macroscopic search are generated based on the graph model; The construction of the current search state includes: The current search state is constructed based on the multimodal feature representation, the graph-guided features, historical search results, and remaining resources.

5. The remote sensing image search method as described in claim 4, characterized in that, The nodes of the graph model represent multiple clustered regions obtained by clustering the image space, and the edges of the graph model represent the relationships between the clustered regions.

6. The remote sensing image search method as described in claim 5, characterized in that, The feature representations of the nodes in the graph model are dynamically updated based on the historical search results during the query process.

7. The remote sensing image search method as described in claim 1, characterized in that, The resource constraints include query count constraints and / or movement cost constraints between sub-regions.

8. The remote sensing image search method as described in claim 7, characterized in that, The movement cost constraint is determined based on the Manhattan distance between sub-regions.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the remote sensing image search method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a remote sensing image search program, which, when executed, implements the remote sensing image search method as described in any one of claims 1 to 8.

Images