Remote sensing image searching method, device and computer readable storage medium

Abstract

The application provides a remote sensing image search method, device and computer readable storage medium, acquires a wide-area remote sensing image and a language instruction describing a search target, realizes semantic alignment by fusing cross-modal semantic information of the wide-area remote sensing image and the language instruction, thereby guiding the search process by using the language instruction, significantly improving the target sub-region discovery efficiency and accuracy in massive data and complex environment, and providing a basis for realizing efficient positioning of the target. Under the constraint of limited resources, a to-be-queried sequence of multiple sub-regions in the wide-area remote sensing image is dynamically planned, a high-value region is preferentially explored, and search and utilization are effectively balanced. When the to-be-queried sequence is executed, corresponding sub-regions in the wide-area remote sensing image are sequentially queried, and finally a sub-region set containing the target matched with the language instruction is output, the overall system response speed and intelligent level are improved through the multi-modal information cooperation capability, and the application is suitable for resource-constrained scenes such as space-based earth observation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a remote sensing image search method, device and computer readable storage medium. BACKGROUND

[0002] With the continuous improvement of spatial, temporal and spectral resolution, the image data generated by the earth observation system has grown dramatically in size. This has brought great challenges to the traditional "sky sensing, ground computing, manual decision-making" (sky-based sensing, ground-based computing, manual decision-making) data processing mode: the network transmission and ground information processing system pressure increases dramatically, resulting in long task response cycle, low data utilization rate, and insufficient intelligent level of task management and control, ultimately limiting the application efficiency of the system in the rapidly changing environment. Large-scale data loses timeliness due to transmission and processing delay, resulting in great waste of resources. At the same time, the existing observation system lacks autonomy and adaptability, and there is a significant lag in task planning and execution. In particular, when dealing with emergencies such as earthquakes and mudslides, the inability to obtain the latest image data in a timely manner often leads to irreparable losses. The progress of space technology has given rise to the urgent need for a "sky sensing, sky computing, sky decision-making" mode. This mode directly deploys information processing modules on the intelligent satellite cluster, changes the "ground" to "sky", and actively executes image processing algorithms on orbit through intelligent algorithms, significantly reducing the network transmission load between sky and earth, and ultimately improving the flexibility and response speed of task execution. However, how to quickly locate the key area in the vast area of remote sensing images has become one of the key technical challenges of the sky-based earth observation system. SUMMARY

[0003] To overcome the problems in the related art, the present specification provides a remote sensing image search method, device and computer readable storage medium.

[0004] In a first aspect, a remote sensing image search method is provided, the method comprising:

[0005] obtaining a wide-area remote sensing image and a language instruction describing a search target;

[0006] by fusing the cross-modal semantic information of the wide-area remote sensing image and the language instruction, dynamically planning a to-be-queried sequence of a plurality of sub-regions in the wide-area remote sensing image under a preset resource constraint;

[0007] executing the to-be-queried sequence, sequentially querying the corresponding sub-regions in the wide-area remote sensing image, and outputting a sub-region set containing the target that matches the language instruction.

[0008] According to a remote sensing image search method provided by the application, the cross-modal semantic information of the wide-area remote sensing image and the language instruction is fused, a to-be-queried sequence of a plurality of sub-regions in the wide-area remote sensing image is dynamically planned under a preset resource constraint, and the method comprises the following steps:

[0009] The wide-area remote sensing image and the language instruction are semantically aligned to obtain a fused multi-modal feature representation.

[0010] A current search state is constructed based on the multi-modal feature representation, a historical search result and a remaining resource.

[0011] Under the preset resource constraint, the to-be-queried sequence of the plurality of sub-regions in the wide-area remote sensing image is generated according to the current search state.

[0012] According to a remote sensing image search method provided by the application, the to-be-queried sequence of the plurality of sub-regions in the wide-area remote sensing image is generated according to the current search state by using a pre-trained search strategy model.

[0013] The search strategy model is configured to be trained by reinforcement learning to maximize cumulative rewards under a resource constraint, and the rewards are determined according to whether a target exists in the queried sub-region.

[0014] According to a remote sensing image search method provided by the application, the method further comprises:

[0015] An image space is clustered according to semantic features of a plurality of sub-regions in the wide-area remote sensing image to construct a graph model representing the relationship between clustered regions.

[0016] A graph-guided feature for macro search is generated based on the graph model.

[0017] The current search state is constructed, and the method comprises the following steps:

[0018] The current search state is constructed based on the multi-modal feature representation, the graph-guided feature, a historical search result and a remaining resource.

[0019] According to a remote sensing image search method provided by the application, the nodes of the graph model represent a plurality of clustered regions obtained by clustering the image space, and the edges of the graph model represent the association relationship between the clustered regions.

[0020] According to a remote sensing image search method provided by the application, the feature representation of the nodes in the graph model is dynamically updated according to a historical search result in a query process.

[0021] According to a remote sensing image search method provided by the application, the resource constraint comprises a query frequency constraint and / or a movement cost constraint between sub-regions.

[0022] According to the remote sensing image search method provided in the present application, the mobile cost constraint is determined based on the Manhattan distance between sub-regions.

[0023] In a second aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the remote sensing image search method according to the first aspect is implemented.

[0024] In a third aspect, a computer readable storage medium is provided, wherein the computer readable storage medium stores a remote sensing image search program, and when the remote sensing image search program is executed, any one of the remote sensing image search methods according to the first aspect is implemented.

[0025] The present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the remote sensing image search method according to any one of the above aspects.

[0026] The remote sensing image search method, device and computer readable storage medium provided in the embodiments of the present application have the following beneficial effects compared with the current difficulty in quickly locating a specific target in a wide-area remote sensing image:

[0027] In a first aspect, a wide-area remote sensing image and a language instruction describing a search target are obtained, semantic alignment is achieved by fusing cross-modal semantic information of the wide-area remote sensing image and the language instruction, so as to guide the search process by using the language instruction, and the efficiency and accuracy of target sub-region discovery in a large amount of data and a complex environment are significantly improved, thereby providing a basis for efficient positioning of the target.

[0028] In a second aspect, a sequence of sub-regions to be queried in the wide-area remote sensing image is dynamically planned under limited resource constraints, high-value regions are preferentially explored, and search and utilization are effectively balanced. When the sequence of sub-regions to be queried is executed, the corresponding sub-regions in the wide-area remote sensing image are sequentially queried, and finally a set of sub-regions containing the target that matches the language instruction is output. Through the multi-modal information collaboration capability, the overall system response speed and intelligent level are improved, and the method is suitable for resource-constrained scenarios such as space-based earth observation.

[0029] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present specification. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present specification and, together with the specification, serve to explain the principles of the present specification.

[0031] Figure 1 FIG. 1 is a flowchart of a remote sensing image search method according to an exemplary embodiment of the present specification.

[0032] Figure 2 FIG. 1 is a flowchart of a first embodiment of a remote sensing image search method according to an example embodiment of the present specification.

[0033] Figure 3 FIG. 2 is a flowchart of a second embodiment of a remote sensing image search method according to an example embodiment of the present specification.

[0034] Figure 4 FIG. 3 is a schematic diagram of a remote sensing image search device according to an example embodiment of the present specification.

[0035] Figure 5 FIG. 4 is a schematic block diagram of a remote sensing image search device according to an example embodiment of the present specification. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments (or “modes of implementation”) of the present application will be described clearly and completely in conjunction with the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated.

[0037] If the present application embodiments involve directional indications or positional relationships (such as up, down, left, right, front, back, inner, outer, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationship, movement, etc. between the components in a certain posture (as shown in the drawings); if the specific posture changes, the directional indication or positional relationship will also change accordingly. In addition, the terms “first”, “second”, etc. in the present application embodiments are only used for convenience of description, and cannot be understood as indicating or implying relative importance.

[0038] The present application provides a remote sensing image search method, device and computer readable storage medium. The present application will be described in detail below in conjunction with the accompanying drawings. The features in the following embodiments and modes of implementation can be combined with each other.

[0039] Wide-area remote sensing images usually cover a wide range of ground objects, for example, with a size of 30000*30000 pixels. It is a very challenging task to quickly locate a specific target (such as a vehicle, a building, or a disaster area) in a wide-area remote sensing image.

[0040] Existing non-short-sighted search techniques use a deep reinforcement learning-based visual active exploration framework for wide-area geographic spatial exploration, and introduce a meta-learning method to improve the adaptability and efficiency of the model in new tasks. However, this method relies on visual cues and has limited exploration ability in complex environments, resulting in low search efficiency.

[0041] To solve the above technical problems, the present specification provides a remote sensing image search method, referring to Figure 1 , Figure 1 is a flowchart of a remote sensing image search method provided by an embodiment of the present specification.

[0042] It aims to guide the visual search process with language instructions and dynamically plan the search path through reinforcement learning to achieve efficient positioning of targets in wide-area remote sensing images under resource constraints.

[0043] In other words, through the VLAS (Visual-Language Active Search) technology, under the guidance of specific language instructions, the sub-regions in the wide-area remote sensing image are sequentially screened according to priority, which can maximize the coverage of as many target objects as possible while overcoming budget or resource constraints. In addition, the spatial correlation inherent in adjacent sub-regions can also provide important clues for the search process. This method balances exploration (improving model performance) and utilization (finding more targets) under resource constraints by integrating machine learning models that predict target labels and customized algorithm strategies.

[0044] It should be noted that the remote sensing image search scheme in this paper is directly deployed on the embodied intelligent satellite cluster, changing "ground-based" to "space-based", and actively executing remote sensing image search algorithms on-orbit through intelligent algorithms, which significantly reduces the transmission load of space-earth networks and ultimately improves the flexibility and response speed of task execution.

[0045] The present application provides a first embodiment of a remote sensing image search method, referring to Figure 2 , Figure 2 is a flowchart of a first embodiment of a remote sensing image search method provided by an embodiment of the present specification.

[0046] Specifically, the following steps 101 to 103 are included:

[0047] In step 101, a wide-area remote sensing image and a language instruction describing a search target are obtained.

[0048] First, the search scene and parameters are initialized. This includes the following processes:

[0049] The wide-area remote sensing image is obtained, and the wide-area remote sensing image is divided into grid cells to obtain a plurality of sub-regions. , wherein is the total number of grid cells for image division, and each represents a sub-region in the wide-area image. Subsequently, key sub-regions containing targets are selected from the plurality of sub-regions to achieve efficient positioning of the targets.

[0050] Meanwhile, a language instruction for describing the search target is acquired .

[0051] The language instruction described herein is used to explicitly define the visual target to be searched, and its semantic content includes but is not limited to the description of the target object category, its optional visual attribute, spatial relationship or scene context, such as "identify the area where large vehicles exist". In remote sensing search, this language instruction is used as prior knowledge to guide the semantic understanding and decision direction of the subsequent visual search process.

[0052] As an example, the language instruction can be but is not limited to a standard instruction in the form of text, voice, structured template such as form, etc.

[0053] Further, a preset resource constraint for searching is set.

[0054] As an example, the resource constraint includes a query frequency constraint and / or a movement cost constraint between sub-regions.

[0055] A total query budget is preset , and a historical search result is initialized , a remaining query budget , and a time step . The query budget needs to consider the execution frequency of the query and the movement cost of the sub-regions to be queried before and after. The movement cost constraint is determined based on the Manhattan distance between the sub-regions to be queried.

[0056] Through the above setting, high-value areas can be preferentially explored under the constraints of limited resources and budget, and exploration and utilization can be effectively balanced.

[0057] Step 102, by fusing the cross-modal semantic information of the wide-area remote sensing image and the language instruction, a to-be-queried sequence of a plurality of sub-regions in the wide-area remote sensing image is dynamically planned under the preset resource constraint.

[0058] In the face of tens of TB to hundreds of TB of data generated by the global satellite network every day, the VLAS of the present application can integrate the visual observation of unmanned aerial vehicles or remote sensing satellites and human text prompts through multi-modal information fusion, without random search. Only by inputting a specific language instruction, the system can preferentially explore high-probability areas, so that the team can focus on sub-regions where the target is most likely to be found. That is, the key sub-regions in the complex wide-area image can be quickly locked under the condition of massive data and limited computing power, and the resource allocation of the subsequent task (such as target detection) can be significantly optimized.

[0059] In some embodiments, the dynamic planning of the query sequence of the plurality of sub-regions in the wide-area remote sensing image under the preset resource constraint by fusing the cross-modal semantic information of the wide-area remote sensing image and the language instruction comprises the following steps 1021 to 1023:

[0060] In step 1021, the wide-area remote sensing image and the language instruction are semantically aligned to obtain a fused multi-modal feature representation.

[0061] As an example, the CLIP or other basic cross-modal encoder is used to encode the image and the language instruction. The image feature and the language instruction feature are extracted respectively, and multi-modal semantic fusion is achieved through feature alignment.

[0062] Specifically, the image and the language instruction are encoded as follows:

[0063] The image is encoded into an image feature , which retains spatial position information.

[0064] The language instruction is encoded into an instruction feature , where is the language feature dimension.

[0065] In step 1022, the current search state is constructed based on the multi-modal feature representation, the historical search result, and the remaining resources.

[0066] The image feature , the instruction feature , the historical search result , and the remaining budget are fused to generate an initial state .

[0067] The historical search result is recorded in three values to record the three states of the sub-region, i.e., searched and target exists, searched and target does not exist, and not searched.

[0068] In step 1023, the query sequence of the plurality of sub-regions in the wide-area remote sensing image is generated according to the current search state under the preset resource constraint.

[0069] In some embodiments, the generation of the query sequence of the plurality of sub-regions in the wide-area remote sensing image according to the current search state is achieved by a pre-trained search strategy model.

[0070] The search strategy model described in the text outputs an action according to the current search state, and selects a sub-region to be queried. For ease of description, the search strategy model using VLAS will be referred to as a controller .

[0071] Exemplarily, at time step , the controller generates an action according to the current state , and selects the number of sub-regions to be queried. Among them is the number of sub-regions queried each time, may be a decision neural network with the current state as input and the action as output, i.e., the action .

[0072] Step 103: Execute the sequence to be queried, sequentially query the corresponding sub-regions in the wide-area remote sensing image, and output a set of sub-regions containing the target that match the language instruction.

[0073] Apply the action obtained in the foregoing to the sub-region selection operation, and sequentially process the image data of each sub-region specified in the sequence according to the sub-region number given by the sequence to be queried. Exemplarily, for each sub-region, a visual perception model is called to analyze whether its content matches the search target described by the natural language instruction, and a binary query result (target exists / target does not exist) is generated. Exemplarily, the visual perception model may be, but is not limited to, a target detection model (such as YOLO, Faster R-CNN) or an image classification model.

[0074] After the entire query sequence is executed, the system integrates all successful query results to generate a set of location information of sub-regions containing all hit targets as the final output of this active search task. Among them, the set of location information may be, but is not limited to, a coordinate list or a set of sub-region bounding boxes.

[0075] In some embodiments, the search strategy model is configured to be trained by reinforcement learning to maximize the cumulative reward under resource constraints, and the reward is determined according to whether the target exists in the queried sub-region.

[0076] Apply the action to the sub-region selection, and perform the following operations:

[0077] a. Obtain the immediate reward. According to the target existence flag , calculate the reward . Among them, represents that the target exists, represents that the target does not exist.

[0078] b. Update history search results. For the explored sub-region, i.e., the sub-region with the sequence number , set , and the new history search result is .

[0079] That is, if the query confirms the existence of the target in the sub-region, the sub-region is , if the query confirms the non-existence of the target in the sub-region, the sub-region is , and if the query does not search the sub-region, the sub-region is .

[0080] c. Update the remaining budget. According to the Manhattan distance , the moving cost is calculated, and the remaining budget is updated as , where is the sub-region selected in the previous step.

[0081] d. Update the current search state. The new state is generated.

[0082] After that, the training data is collected. The transition tuple is recorded, which is used for optimization of .

[0083] The above search path process terminates the search process when the termination condition is met. As an example, if the remaining budget , the search process is terminated; otherwise, let and return to step 102 to continue iteration.

[0084] In some embodiments, the search path is optimized by a reinforcement learning strategy.

[0085] As an example, the controller is trained jointly using reinforcement learning (RL) and supervised learning (SL).

[0086] First, the loss value of the loss function is calculated.

[0087] Based on the collected transition tuple , the weighted sum of the reinforcement learning loss and the supervised learning loss is calculated:

[0088] , where is a hyperparameter.

[0089] Next, the parameters of the controller are updated.

[0090] Based on the loss value calculated as described above, the tactical controller and the basic backbone network parameters. Wherein, the gradient is calculated as to maximize the cumulative reward.

[0091] Finally, by iteratively performing sub-region selection, policy optimization and controller parameter updating, the final output is the maximized discovery result of the target under the budget constraint.

[0092] that is, the output serialized sub-region selection wherein represents the sub-region set selected in the i-th iteration, and finally realizes the maximized discovery of the target object under the budget constraint.

[0093] Through the above embodiment, the wide-area remote sensing image and the language instruction describing the search target are obtained, and by mapping the image sub-region and the language instruction to the same semantic space, semantic alignment is realized. On this basis, a reinforcement learning intelligent agent (controller ) is constructed, which decides which sub-region to query next according to the fused multi-modal state (image feature, language feature, historical search state, remaining budget), and continuously optimizes the search strategy through the reward signal. Finally, the target sub-region is quickly located. At the same time, the fusion of cross-modal semantic information of the wide-area remote sensing image and the language instruction significantly enhances the adaptability to complex scenes, and can efficiently locate the target in complex wide-area images.

[0094] The present application provides a second embodiment of a remote sensing image search method, referring to Figure 3 , Figure 3 is a flowchart of a second embodiment of a remote sensing image search method provided by the embodiments of the present application.

[0095] The method is improved on the basis of the basic visual-linguistic active search (VLAS) method, and is a graph-enhanced visual-linguistic active search (PAGE) method. It realizes efficient target sub-region positioning in complex scenes by introducing a hierarchical controller architecture and a dynamic graph model. In order to facilitate description, the search strategy model using VLAS is referred to as controller .

[0096] Specifically, the following steps 201 to 203 are included:

[0097] Step 201, acquiring a wide-area remote sensing image and a language instruction describing a search target.

[0098] receiving and blocking an aerial or satellite image , inputting a language instruction wherein is the total number of grid units divided by the image, and each represents a sub-region in the wide-area image. ​

[0099] setting a preset total query budget , and initializing a history search result , a remaining query budget , an initial queried sub-region feature , and a time step . The query budget needs to comprehensively consider the execution times of the query and the Manhattan distance of the front and rear to-be-queried sub-regions.

[0100] In step 202, a to-be-queried sequence of multiple sub-regions in the wide-area remote sensing image is dynamically planned under a preset resource constraint by fusing the cross-modal semantic information of the wide-area remote sensing image and the language instruction.

[0101] First, the image and the language instruction are encoded by a CLIP or other basic cross-modal encoder :

[0102] The image is encoded into an image feature , which retains spatial position information.

[0103] The language instruction is encoded into an instruction feature , where is the language feature dimension.

[0104] Next, a knowledge graph based on visual language data is constructed , where the nodes represent the cluster sub-region mean features, and the edges represent the feature similarity between classes. The knowledge graph can model the commonality between sub-region features, thereby improving the accuracy of active search.

[0105] In some embodiments, the image space is clustered according to the semantic features of multiple sub-regions in the wide-area remote sensing image, a graph model representing the relationship between the clustered regions is constructed, graph-guided features for macro search are generated based on the graph model, and the current search state is constructed based on the multi-modal feature representation, the graph-guided features, the history search result, and the remaining resources.

[0106] The nodes of the graph model represent multiple clustered regions obtained by clustering the image space, and the edges of the graph model represent the association relationship between the clustered regions.

[0107] Specifically, the construction process adopts cluster region division: the K-Means algorithm is used to cluster the semantic features of all sub-regions of all images in the training set, and clustered regions are divided, numbered .

[0108] graph nodes defined as the average image feature of all sub-regions in each cluster region and average language feature consist of. Wherein, the mean feature is extracted by the pixel of the sub-region and calculated after encoding by the cross-modal encoder; the sub-region category is extracted by the cross-modal encoder.

[0109] At the same time, the Sinkhorn-Knopp algorithm is used to calculate the adjacency probability between the cluster regions, that is, the average probability of whether the sub-regions are physically adjacent, so as to generate a normalized adjacency matrix, which is finally defined as the edge of the graph model . Finally, the graph convolution network is constructed based on the graph . The graph-guided feature is extracted by using the convolution network to process the graph structure according to the queried sub-region feature .

[0110] Thereafter, the image feature , instruction feature , graph-guided feature , historical search results and remaining budget are fused to generate the current search state .

[0111] Under the preset resource constraint, a to-be-queried sequence of the plurality of sub-regions in the wide-area remote sensing image is generated according to the current search state.

[0112] As an example, the sub-region feature is calculated first.

[0113] At time step , all sub-regions in the sub-region set selected in the first step are respectively encoded based on and the feature mean is calculated to obtain the sub-region feature .

[0114] The corresponding node serial number of the current and target sub-regions in the graph is identified.

[0115] The current region is , wherein is the Euclidean distance; and the target region is , wherein is the one-hot vector representing the to-be-searched target object category in the language instruction.

[0116] At time step​​ Hierarchical controller Based on the current search status Generate Actions The process involves selecting the sub-region numbers to be queried, thus forming a query sequence. This represents the number of sub-regions queried each time.

[0117] also, It is a decision neural network that can be used through graph convolutional networks. Computational graph features Then calculate the sub-region selection action according to the VLAS method. .

[0118] Step 203: Execute the query sequence, sequentially query the corresponding sub-regions in the wide-area remote sensing image, and output a set of sub-regions containing the target that match the language command.

[0119] Actions When applied to a sub-region selection, perform the following operations:

[0120] Get an instant reward. Based on the presence of a target marker. Calculate rewards .in, This indicates that the target exists. This indicates that it does not exist.

[0121] Update historical search results. For explored sub-regions, i.e., their sequence numbers. ,set up The new historical search results are In summary, the sub-regions where the target is confirmed to exist after the query correspond to... When it is confirmed that it does not exist If no search is performed .

[0122] Update remaining budget. Based on Manhattan distance. Calculate movement costs and update the remaining budget. ,in The sub-region selected in the previous step.

[0123] Update state, generate new state .

[0124] The feature representations of the nodes in the graph model are dynamically updated based on historical search results during the query process. Specifically, this is done based on the features of the selected sub-region. The average image features of the cluster centers to which the current sub-region belongs are adjusted in real time. .in To adjust the hyperparameters, they are generally set to a positive real number close to 0.

[0125] If the remaining budget , then terminate the search process; otherwise, let and return to step S301 to continue iteration.

[0126] During the search process, training data is collected. Transition tuples are recorded for optimization of .

[0127] The specific optimization process is as follows:

[0128] Based on the collected transition tuples , the weighted sum of reinforcement learning loss and supervised learning loss is calculated , where is a hyperparameter.

[0129] The parameters of the controller and the base backbone network are updated by a backpropagation algorithm, and the gradient calculation is to maximize the cumulative reward.

[0130] Through the above process, the serialized sub-region selection is output, where denotes the sub-region set selected in the th step, and finally the maximization of the target object under the budget constraint is realized.

[0131] Through the above embodiment, the graph structure modeling and hierarchical controller are introduced on the basis of VLAS, the macro area graph is constructed through clustering, and the semantic understanding and strategic planning ability for complex scenes are enhanced. More detailed decision making is realized in complex wide-area images, and the efficiency and accuracy of target positioning are significantly improved.

[0132] The application provides a remote sensing image search method and device and computer readable storage medium. Compared with the current difficulty in quickly positioning a specific target in a wide-area remote sensing image, the application has the following beneficial effects:

[0133] In the first aspect, a wide-area remote sensing image and a language instruction describing a search target are acquired. Semantic alignment is realized by fusing cross-modal semantic information of the wide-area remote sensing image and the language instruction, so as to guide the search process by using the language instruction, and the efficiency and accuracy of sub-region discovery of the target in a large amount of data and a complex environment are significantly improved, thereby providing a basis for efficient positioning of the target.

[0134] In a second aspect, a sequence of sub-regions to be queried in a wide-area remote sensing image is dynamically planned under limited resource constraints, high-value regions are preferentially explored, and search and utilization are effectively balanced. In executing the sequence of sub-regions to be queried, corresponding sub-regions in the wide-area remote sensing image are sequentially queried, and a set of sub-regions containing a target that matches a language instruction is finally output. Through multi-modal information collaboration capability, the system response speed and intelligent level are overall improved, and the system is suitable for resource-constrained scenarios such as space-based earth observation.

[0135] Based on the same application concept as the above method, the embodiments of the present application also propose a remote sensing image search device. As shown in Figure 4 The device comprises:

[0136] An information acquisition module is configured to acquire a wide-area remote sensing image and a language instruction describing a search target.

[0137] A feature fusion module is configured to dynamically plan a sequence of sub-regions to be queried in the wide-area remote sensing image by fusing cross-modal semantic information of the wide-area remote sensing image and the language instruction under a preset resource constraint.

[0138] A target search module is configured to execute the sequence of sub-regions to be queried, sequentially query corresponding sub-regions in the wide-area remote sensing image, and output a set of sub-regions containing the target that matches the language instruction.

[0139] The implementation process of the functions and roles of each module / submodule / unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and the same technical effects can be achieved, and thus will not be described here.

[0140] The present application also provides a whole vehicle controller for implementing the remote sensing image search method described above.

[0141] Figure 5 An example of an entity structure diagram of a remote sensing image search device is shown in Figure 5 The remote sensing image search device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540. The processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can invoke logical instructions in the memory 530 to execute the remote sensing image search method.

[0142] In addition, the logic instructions in the memory 530 described above can be implemented in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0143] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the remote sensing image search method provided by the above-mentioned methods.

[0144] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the remote sensing image search method provided by the above-mentioned methods.

[0145] It should be noted that the technical solutions or technical features described in the above embodiments can be combined or supplemented with each other without conflict, and the scope of protection of the present application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A remote sensing image search method, characterized in that, The method includes: Acquire wide-area remote sensing images and language instructions describing the search targets; By fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions, a query sequence of multiple sub-regions in the wide-area remote sensing image is dynamically planned under preset resource constraints. Execute the query sequence, sequentially query the corresponding sub-regions in the wide-area remote sensing image, and output a set of sub-regions containing the target that match the language command; The step of dynamically planning a query sequence for multiple sub-regions in the wide-area remote sensing image under preset resource constraints by fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions includes: Semantic alignment is performed on the wide-area remote sensing image and the language instructions to obtain a fused multimodal feature representation; The image space is clustered based on the semantic features of multiple sub-regions in the wide-area remote sensing image, and a graph model representing the relationships between the clustered regions is constructed. Graph-guided features for macroscopic search are generated based on the graph model. The current search state is constructed based on the multimodal feature representation, the graph-guided features, historical search results, and remaining resources. Nodes in the graph model represent multiple clustered regions obtained by clustering the image space, and edges in the graph model represent the relationships between the clustered regions. Under preset resource constraints, a query sequence of multiple sub-regions in the wide-area remote sensing image is generated based on the current search state.

2. The remote sensing image search method as described in claim 1, characterized in that, The step of dynamically planning a query sequence for multiple sub-regions in the wide-area remote sensing image under preset resource constraints by fusing cross-modal semantic information of the wide-area remote sensing image and the language instructions includes: Semantic alignment is performed on the wide-area remote sensing image and the language instructions to obtain a fused multimodal feature representation; Based on the multimodal feature representation, historical search results, and remaining resources, the current search state is constructed; Under preset resource constraints, a query sequence of multiple sub-regions in the wide-area remote sensing image is generated based on the current search state.

3. The remote sensing image search method as described in claim 2, characterized in that, The generation of query sequences for multiple sub-regions in the wide-area remote sensing image based on the current search state is achieved through a pre-trained search strategy model; The search strategy model is configured to be trained through reinforcement learning to maximize cumulative rewards under resource constraints, the rewards being determined based on whether the target exists within the queried sub-region.

4. The remote sensing image search method as described in claim 1, characterized in that, The feature representations of the nodes in the graph model are dynamically updated based on the historical search results during the query process.

5. The remote sensing image search method as described in claim 1, characterized in that, The resource constraints include query count constraints and / or movement cost constraints between sub-regions.

6. The remote sensing image search method as described in claim 5, characterized in that, The movement cost constraint is determined based on the Manhattan distance between sub-regions.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the remote sensing image search method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a remote sensing image search program, which, when executed, implements the remote sensing image search method as described in any one of claims 1 to 6.

Images