Remote sensing image open-vocabulary object detection method
By constructing an incrementally scalable remote sensing knowledge graph and collaborating with a visual language model, the problem of readily usable remote sensing target detection in open scenarios is solved, improving the accuracy and interpretability of detection and realizing knowledge-enhanced target detection of remote sensing images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING GEOMATICS & REMOTE SENSING CENT
- Filing Date
- 2026-05-19
- Publication Date
- 2026-07-21
AI Technical Summary
Existing remote sensing target detection technologies cannot be used immediately in open scenarios, lack the utilization of prior knowledge in the remote sensing field, have insufficient adaptability of visual language models, and are disconnected from the knowledge graph and detection task, resulting in limited detection capabilities and unreliable results.
We construct an incrementally scalable remote sensing knowledge graph and deeply collaborate with a visual language model. The knowledge graph provides prior information for adjusting model parameters and semantic embedding. We combine knowledge-enhanced visual-text input pairs for cross-modal alignment and consistency verification to achieve open-vocabulary target detection.
It enables open-vocabulary, knowledge-enhanced, and interpretable target detection of remote sensing imagery, improving application flexibility and detection accuracy, reducing false detection rate, and providing traceable decision-making basis.
Smart Images
Figure CN122223312B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing image target detection technology, and in particular relates to a method for target detection of open vocabulary in remote sensing images. Background Technology
[0002] With the continuous improvement of the spatiotemporal resolution of remote sensing satellites, the volume of high-resolution, multispectral, and multi-platform remote sensing images is growing exponentially. However, the powerful remote sensing data acquisition capabilities are not matched with the intelligent interpretation and knowledge service capabilities of remote sensing big data. Current mainstream remote sensing target detection technologies still face the following key bottlenecks:
[0003] First, the limitations of the closed-set detection paradigm are amplified, making it difficult to handle open scenarios. Existing methods heavily rely on labeled data and are trained and detected on a fixed set of categories. However, the closed-set training paradigm limits the model to recognizing only preset categories. When detecting objects of entirely new categories, new category labels need to be added for retraining, which fails to achieve the "ready-to-use" capability for open-vocabulary detection. This severely restricts its application in dynamically changing scenarios and emergency response tasks.
[0004] Second, there is a lack of utilization of prior knowledge in the remote sensing field. Most existing models only learn apparent features from image pixels, failing to effectively model and utilize prior knowledge in the remote sensing field, such as typical spectral features of ground objects, inherent spatial distribution and symbiotic relationships between targets (e.g., airports must be adjacent to runways), and semantic constraints that conform to geographical common sense. This results in limited semantic understanding ability of the models in complex scenarios, making them prone to false detections that violate domain knowledge.
[0005] Third, the remote sensing adaptability of emerging visual language models is insufficient. Recently, visual language models pre-trained on large-scale internet images and text have demonstrated powerful open-world perception and natural language reasoning capabilities, providing new ideas for open vocabulary detection. However, directly applying general VLMs to the remote sensing field has significant shortcomings: First, their pre-training corpora lack specialized remote sensing data, leading to biases in the semantic understanding of key remote sensing features such as spectrum, texture, and spatial layout, and making them prone to "embedding drift" when facing new categories; Second, the models themselves lack structured knowledge and cannot identify and correct detection results that violate spatial-semantic common sense, resulting in insufficient reliability and interpretability of the results.
[0006] Fourth, traditional knowledge graphs are disconnected from detection tasks. While knowledge graphs can store domain knowledge in a structured form, traditional construction methods are mostly static and offline, making it impossible to dynamically expand and update them in real time based on new concepts input by users during detection tasks. They are often isolated from visual perception models, failing to form a "perception-knowledge" closed loop, and therefore struggle to play their core role of "real-time knowledge supply and verification" in open vocabulary detection.
[0007] Therefore, there is an urgent need for an open vocabulary target detection method that combines the advantages of incremental knowledge graph structured reasoning with the open perception capabilities of visual language models. Summary of the Invention
[0008] To address the aforementioned problems in the prior art, this invention employs a remote sensing image open vocabulary target detection method, comprising:
[0009] S1. Obtain publicly available remote sensing datasets and construct an initial remote sensing knowledge graph with remotely sensed ground objects as entities based on the publicly available remote sensing datasets; wherein, the initial remote sensing knowledge graph includes the attributes of entities and the relationships between entities;
[0010] S2. When the user inputs the remote sensing image to be detected and multiple target categories i, the current remote sensing knowledge graph is obtained, and the current remote sensing knowledge graph is queried to determine whether each target category i is a known category or a newly added category; the key knowledge of each known category is extracted from the current remote sensing knowledge graph, and the current remote sensing knowledge graph is incremented according to each newly added category to obtain the latest remote sensing knowledge graph, and the key knowledge of each newly added category is extracted from the latest remote sensing knowledge graph.
[0011] S3. Combine the key knowledge of each target category i with the remote sensing image to be detected to generate knowledge-enhanced visual-text input pairs for each target category i.
[0012] S4. Obtain the visual language model, and generate category concept semantic embedding vectors for each target category i based on the visual language model and the latest remote sensing knowledge graph; combine the category concept semantic embedding vectors for each target category i to obtain a dynamic detection vocabulary.
[0013] S5. Based on the visual language model, dynamic detection vocabulary, and knowledge-enhanced visual-text input pair, perform visual-semantic cross-modal alignment and open vocabulary detection on each target category i to obtain the detection box set for each target category i.
[0014] S6. Based on the latest remote sensing knowledge graph, perform spatial and semantic consistency verification on the detection box set of each target category i to obtain the detection results of each target category i.
[0015] Beneficial effects:
[0016] 1. This invention constructs an online, incrementally expandable remote sensing knowledge graph and deeply integrates it with a visual language model. This enables open-vocabulary, knowledge-enhanced, and interpretable target detection of remote sensing images, capable of understanding and instantly responding to any new category input by the user without retraining, greatly improving application flexibility. 2. This invention simultaneously applies the prior information of the knowledge graph to two key stages: at the task level, the task instruction text, incorporating prior information from the knowledge graph, is encoded into a task conditional semantic vector. The parameters of the visual language model are adjusted based on this vector, guiding the model on "how to see" the target; at the conceptual level, the prior information from the knowledge graph is input into the visual language model to construct new categories. A stable category concept semantic embedding vector is built to define "what it is". For new categories with incomplete information, a semantic completion mechanism is activated to integrate its basic definition text, graph hierarchy relationship and related context information to generate semantic embeddings, so as to make up for the incompleteness of automatically acquired information and ensure that new categories can also obtain high-quality semantic representation. The two work together to effectively solve the semantic drift and adaptation problem of general visual speech models in the field of remote sensing and the problem of incomplete information of new categories, which can significantly enhance the accuracy of detection and adaptability to professional scenarios. 3. This invention uses the logical rules contained in the knowledge graph to perform spatial and semantic consistency verification of the detection box, effectively reducing the false detection rate and providing traceable decision basis. Attached Figure Description
[0017] Figure 1 A flowchart of a remote sensing image open vocabulary target detection method provided in an embodiment of the present invention;
[0018] Figure 2 A flowchart for constructing a remote sensing knowledge graph provided in an embodiment of the present invention;
[0019] Figure 3 This is a schematic diagram illustrating the dual-path knowledge collaboration of steps S3 and S4 provided in an embodiment of the present invention;
[0020] Figure 4 A flowchart of visual-semantic cross-modal alignment and open vocabulary detection provided in an embodiment of the present invention;
[0021] Figure 5 A flowchart for spatial and semantic consistency verification provided for embodiments of the present invention. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] like Figure 1 As shown, this embodiment of the invention provides a method for open-vocabulary target detection in remote sensing images. By combining the advantages of incremental knowledge graph structured reasoning with the open perception capabilities of visual language models, it achieves open-vocabulary, knowledge-enhanced, and interpretable target detection in remote sensing images. Specifically, it includes the following steps:
[0024] S1. Obtain publicly available remote sensing datasets and construct an initial remote sensing knowledge graph with remotely sensed ground objects as entities based on the publicly available remote sensing datasets; wherein, the initial remote sensing knowledge graph includes the attributes of entities and the relationships between entities; the attributes include: visual attributes, spatial context attributes, functional attributes, and core text attributes, and the relationships include: spatial relationships and semantic relationships between entities;
[0025] like Figure 2 As shown, constructing an initial remote sensing knowledge graph with remotely sensed ground features as entities includes:
[0026] The logical structure of a knowledge graph includes a schema layer and a data layer. It is constructed using a combination of top-down and bottom-up approaches, and is designed to meet the needs of open vocabulary target detection, clearly defining the organizational form of knowledge.
[0027] The model layer uses remotely sensed ground objects as the core entities, corresponding to detectable remotely sensed target categories, and defines attributes and relationships between entities for each entity, specifically including the following information:
[0028] Visual attributes include: shape description, spectral features, texture description, typical colors, layout features, and size range (related to different resolutions); spatial context attributes include: typical spatial descriptions (describing which other entities the entity often co-occurs with or has positional relationships with), such as proximity to rivers, location in suburban areas, and typical geographical environments (mountainous areas, farmland, etc.); functional attributes include: transportation, agricultural planting, and usage status, which can provide high-level semantic understanding for the model. Attributes with a clear value range (interval range or enumerated values) (such as typical spectral index range, possible geographical environments, etc.) are common-sense attributes.
[0029] Relationships are divided into spatial relationships and semantic relationships, which connect different remotely sensed ground objects. Spatial relationships may include adjacency, co-occurrence, and orientation. Semantic relationships may include category inheritance, part-whole, synonymy, or generalized relationships.
[0030] The core text attributes are either comprehensive descriptive text or basic definition text. The comprehensive descriptive text is a natural language text that integrates other attributes (i.e., visual attributes, spatial context attributes, and functional attributes) and relationships. It serves as the core semantic representation of the entity and is pre-generated and stored during the initialization of the remote sensing knowledge graph for direct reading in subsequent steps. The basic definition text is a short core definition text (such as a searched encyclopedia summary) added when incrementally building the remote sensing knowledge graph. It is used to quickly obtain the core semantics of new categories during incremental construction.
[0031] Within the schema layer framework, a high-frequency list is obtained using publicly available remote sensing datasets as the primary data source, along with other multi-source information (such as professional databases, encyclopedic data, geographic information data, etc.). Entities, attributes, and relationships in the high-frequency list are extracted to form triples. Through knowledge fusion, these triples are stored in a graph database (such as Neo4j) for storage domain management, resulting in an initial remote sensing knowledge graph.
[0032] S2. When the user inputs the remote sensing image to be detected and multiple target categories i, obtain the current remote sensing knowledge graph. Query the current remote sensing knowledge graph Determine whether each target category i is a known class or a newly added class; in the current remote sensing knowledge graph Extract key knowledge from each known class k, and then apply it to the current remote sensing knowledge graph based on each newly added class j. Incremental processing is performed to obtain the latest remote sensing knowledge graph. From the latest remote sensing knowledge graph Extract the key knowledge of each newly added class j;
[0033] like Figure 2 As shown, this relates to the current remote sensing knowledge graph. Incremental processing includes:
[0034] The system automatically retrieves information for each newly added class j by calling multi-source information acquisition interfaces (such as encyclopedia APIs, geographic information data, domain knowledge bases / standards, etc.). Under the framework constraints of the schema layer, it performs knowledge extraction and entity creation (such as triple extraction, cleaning, and fusion) on the information of each newly added class j to obtain the entities (corresponding to target category j) and their basic definition text and relations for each newly added class j. The entities, their basic definition text, and relations for each newly added class j are then added to the current remote sensing knowledge graph as new nodes and edges. To obtain the latest remote sensing knowledge graph Among them, the latest remote sensing knowledge graph The core text attribute of the entity corresponding to the known class is the comprehensive description text, and the core text attribute of the entity corresponding to the newly added class is the basic definition text.
[0035] When adding new classes, automatic incremental construction typically only obtains basic definition text and basic relationships, while detailed visual attributes, spatial context attributes, and other information are usually difficult to extract automatically from general knowledge sources. This invention allows for the gradual improvement of these attributes through manual review or semi-automatic completion. After improvement, the core text attributes are replaced with a comprehensive descriptive text constructed from the improved visual attributes, spatial context attributes, and other information.
[0036] Extracting key knowledge of a target category from a knowledge graph includes: extracting relevant attributes and relationships from the knowledge graph as key knowledge for the target category to be detected.
[0037] Preferably, the most typical attributes and relationships are extracted. The most typical attributes and relationships are determined based on domain knowledge. Generally, the most typical visual attributes are shape, texture, color, layout, etc., and the most typical spatial context attributes are descriptions of the spatial environment, such as "often located in flat and open areas" or "often adjacent to roads".
[0038] S3. Combine the key knowledge of each target category i with the remote sensing image to be detected to generate knowledge-enhanced visual-text input pairs for each target category i.
[0039] like Figure 3 As shown, constructing a knowledge-enhanced visual-text input pair for target category i includes:
[0040] S31. Standardize the remote sensing image to be detected to obtain multiple standardized remote sensing image tensors.
[0041] Remote sensing images are processed and analyzed using remote sensing image processing tools (such as GDAL). According to preset adaptation rules, necessary radiometric normalization and band selection / rearrangement are performed to generate a standardized image tensor with a fixed number of channels. Based on the global size of the image, the typical scale of the target object (some information can be derived from knowledge graphs), and the optimal input size of the model, the large-format image is cut into a series of standardized remote sensing image tensors to adapt to the visual language model.
[0042] S32. Combine the key knowledge of target category i with the metadata of the remote sensing image to be detected to generate the task instruction text for target category i.
[0043] The metadata of the remote sensing image to be detected includes data such as resolution and sensor; therefore, a template example of the task instruction text is as follows: "Please detect the target category in this {sensor} image with {resolution} meters. In this type of image, this target usually possesses {key knowledge}."
[0044] S33. Combine multiple standardized remote sensing image tensors with the task instruction text of target category i to obtain the knowledge-enhanced visual-text input pair of target category i.
[0045] S4. Obtain the visual language model, and then use the visual language model and the latest remote sensing knowledge graph. Generate category concept semantic embedding vectors for each target category; combine the category concept semantic embedding vectors of each target category to obtain a dynamic detection vocabulary;
[0046] Given that a class has a complete comprehensive description text in the core text attributes of the knowledge graph, this comprehensive description text can be directly encoded using a visual language model. When adding a new class, only basic definitions and aliases are typically obtained during automatic incremental construction. Using incomplete basic definition text results in poor quality semantic embeddings. Therefore, the generated category concept semantic embedding vector for a target category i includes:
[0047] If target category i is a known class, then in the latest remote sensing knowledge graph Extracting the core text attributes of the entity corresponding to target category i, and inputting the extracted core text attributes into the text encoder of the visual language model, yields the category concept semantic embedding vector of target category i; where, in the latest remote sensing knowledge graph The core text attribute of the entity corresponding to the known class is the comprehensive description text;
[0048] If target category i is a newly added category, then based on the latest remote sensing knowledge graph... A visual language model is used to perform semantic completion and embedding on target category i, resulting in a category concept semantic embedding vector for target category i. The core text attributes of the entities corresponding to the newly added class in the latest remote sensing knowledge graph are used as the basis for defining the text.
[0049] For newly added categories with incomplete information, a semantic completion mechanism is activated. This mechanism integrates the basic definition text, hierarchical relationships in the graph, and related contextual information to generate semantic embeddings. This compensates for the incompleteness of automatically acquired information and ensures that newly added categories also obtain high-quality semantic representations. Specifically, the semantic completion and embedding of target category i using a visual language model includes:
[0050] In the latest remote sensing knowledge graph Extract the core text attributes of the entity corresponding to target category i, and input the extracted core text attributes into the text encoder of the visual language model to obtain the basic semantic embedding vector. This vector encodes the literal core information of the class.
[0051] In the latest remote sensing knowledge graph In this process, a hierarchical relationship chain is established for target category i through entity linking or relational reasoning. The names of all entities in the hierarchical relationship chain (such as roads and oil refineries) are concatenated to obtain the hierarchical descriptive text of target category i. The hierarchical descriptive text of target category i The text encoder of the input visual language model is used to obtain hierarchical semantic embedding vectors. This vector injects stable domain classification framework semantics into the category;
[0052] In the latest remote sensing knowledge graph Find related entities in the target category i that have spatial or semantic relationships (such as proximity, co-occurrence, etc.) with the target category i, and extract the names and relationship types of the related entities. Generate contextual description text for the target category i based on the names and relationship types of the related entities of the target category i. (e.g., proximity to roads, frequent co-occurrence with oil refineries, etc.) The contextual description text of target category i. Input the text encoder of the visual language model to obtain context-related embedding vectors. This vector implies the typical application scenario or environment of this category.
[0053] The basic semantic embedding vector, hierarchical semantic embedding vector, and context-related embedding vector are fused to obtain the category concept semantic embedding vector of target category i. The formula is illustrated below:
[0054]
[0055] Among them, weight , , It can be adjusted based on experience or validation sets, and .
[0056] Combining the core text attributes and hierarchical descriptive text of target category i and contextual description text The comprehensive description text of target category i is obtained, and the core text attributes of the entity corresponding to target category i in the latest remote sensing knowledge graph are replaced with the comprehensive description text of target category i.
[0057] The dynamic detection vocabulary only contains all target categories involved in the current user query and their corresponding final semantic embedding vectors, rather than loading the full graph category embeddings. This ensures that no matter how large the backend knowledge graph grows, the number of vectors actually participating in the subsequent S5 step (cross-modal similarity calculation) is only related to this task, thus guaranteeing the efficiency of each detection task.
[0058] Step S4 focuses on abstracting category concepts and generating stable, universal "concept ID card" semantic embedding vectors as static reference anchors for cross-modal alignment.
[0059] S5. Based on the visual language model, dynamic detection vocabulary, and knowledge-enhanced visual-text input pair, perform visual-semantic cross-modal alignment and open vocabulary detection on each target category i to obtain the detection box set for each target category i.
[0060] like Figure 4 As shown, visual-semantic cross-modal alignment and open vocabulary detection for target category i includes:
[0061] S51. Enhance the task instruction text in the visual-text input pair with the knowledge of target category i, input it into the text encoder of the visual language model, and obtain the task conditional semantic vector of target category i. ;Conditional adaptation is performed on the task conditional semantic vector of target category i to obtain the adaptation parameters of target category i;
[0062] The task condition semantic vector encapsulates the target category, specific image conditions, and relevant domain knowledge hints for this detection.
[0063] The conditional adaptation of the task conditional semantic vector of target category i includes: obtaining the LoRA adapter, inputting the task conditional semantic vector of target category i into the LoRA adapter, and generating the LoRA adaptation parameters (A, B) of target category i.
[0064] S52. Adjust the parameters of the visual encoder of the visual language model according to the adaptation parameters of the target category i to obtain the visual encoder of the target category i; input each normalized remote sensing image tensor in the knowledge-enhanced visual-text input pair of the target category i into the visual encoder of the target category i to obtain the local visual features of each normalized remote sensing image tensor.
[0065] Adjusting the parameters of the visual language model based on the adaptation parameters of target category i includes: dynamically adjusting the weight distribution of the feedforward network or attention layer of the visual encoder in a lightweight manner (such as LoRA low-rank adaptation) (i.e., W=W+B×A), so as to guide the feature extraction process to focus on the target attributes and context emphasized in the task instructions without retraining the backbone network; where W is the weight of the feedforward network or attention layer of the visual encoder.
[0066] S53. Rearrange the local visual features of all standardized remote sensing image tensors according to their spatial positions in the remote sensing images to obtain a two-dimensional spatial feature map.
[0067] The feature encoding of each spatial location on the two-dimensional spatial feature map encodes the visual information of the corresponding local area in the remote sensing image, which is used for subsequent dense similarity calculation and target localization.
[0068] S54. For each spatial location p in the two-dimensional spatial feature map, calculate the similarity between its local visual features and the category concept semantic embedding vectors of each target category i in the dynamic detection vocabulary. ,exist The highest similarity and its corresponding target category are selected as its (i.e., spatial location p) confidence score and predicted label; where N is the number of target categories;
[0069] Preferably, before calculating the similarity, the local visual features and category concept semantic embedding vectors are normalized, and the similarity is cosine similarity.
[0070] S55. Set a confidence threshold and select spatial locations in the two-dimensional spatial feature map where the confidence level is higher than the confidence threshold. Select each spatial location By mapping these coordinates back to the coordinate system of the remote sensing image to be detected, the selected spatial locations can be obtained. The coordinates; where, The index for the selected spatial location;
[0071] The confidence threshold can be adjusted within the range of 0.2 to 0.5 based on the output characteristics of the visual language model and the actual needs of remote sensing target detection. For example, 0.4 can be used for scenarios requiring high confidence, and 0.25 can be used for scenarios requiring high recall. The threshold can be optimized according to the specific task.
[0072] S56. Construct detection boxes based on the coordinates, predicted labels, and confidence scores of each selected spatial location q. Integrate all detection boxes to obtain a set of detection boxes for target category i.
[0073] Integrating all detection boxes for target category i includes: applying non-maximum suppression to remove redundant boxes, forming a set of detection boxes for target category i. , as input for S6 knowledge verification; where, As coordinates, For predicting labels, Let l represent the confidence level, and l be the index of the detection box in the detection box set for target category i.
[0074] S6. Based on the latest remote sensing knowledge graph, perform spatial and semantic consistency checks on the detection box sets of each target category i to obtain the detection results of each target category i.
[0075] like Figure 5As shown, the spatial and semantic consistency verification of the detection box set for target category i includes:
[0076] S61. The set of detection boxes for target category i Each detection box in In the latest remote sensing knowledge graph Search for its predicted tags The corresponding entity is used to obtain the detection box. entity In the latest remote sensing knowledge graph Obtain each detection box entity Related entities and common-sense attributes;
[0077] entity The associated entity is the entity Entities that have spatial or semantic relationships; entities Its common-sense attributes are the latest remote sensing knowledge graph. The attribute has a defined value range.
[0078] S62, Based on each detection frame entity The common-sense attributes and their value ranges respectively correspond to the detection boxes. Perform hard constraint validation to obtain multiple valid detection boxes. Where m is the index of the valid detection box;
[0079] Hard constraint rules: Common-sense attributes define the objective and essential characteristics of this category; violations represent fundamental errors and result in direct rejection. Then, for the detection box... Hard constraint verification includes: within the detection bounding box of the remote sensing image to be detected. Calculate entities within the region respectively The actual attribute value of each common-sense attribute; if the actual attribute value of each common-sense attribute does not exceed its value range (the reasonable value range or enumeration list defined by the knowledge graph), then the check box is determined. It does not violate fundamental common sense in the field, namely the detection frame. If the verification passes, the detection box will be removed and an explanation of the knowledge entry will be generated. For example: "Detection box #23 (Category A) has been hard removed because the average NDVI value of the region is 0.15, which violates the 'typical NDVI range' [0.2, 0.6] defined for entity 'A' in the knowledge graph."
[0080] S63. Based on each verified detection frame entity The associated entities respectively target the corresponding detection boxes Perform soft constraint validation to obtain each valid checkbox. Final confidence level;
[0081] Soft constraint rules: Entity The associated entities describe the detection box. The typical context of the predicted label is the rationale supporting its existence.
[0082] For the detection boxes that pass the verification Performing soft constraint verification includes:
[0083] With detection box Using the center as a reference, expand outwards by a preset frame size to obtain the detection frame. The neighboring region;
[0084] In the detection box set Search for predicted labels for entities The category corresponding to any associated entity, and whose coordinates are located within the detection box. The detection boxes within the neighborhood region are used to obtain the associated detection box set;
[0085] For example, when the entity When the associated entities include roads and refineries, the predicted label of the detection box is either road or refinery, and the coordinates of the detection box are located within the detection box. Within the neighborhood of the bounding box, the bounding box is considered an associated detection box.
[0086] If the associated detection box set is not empty, then calculate the number of categories of all detection boxes in the associated detection box set (i.e., the number of categories of predicted labels for all detection boxes) to obtain the detection box. The number of categories of associated entities within the neighborhood area;
[0087] If the set of associated detection boxes is empty, then the entity is calculated using the visual language model. The semantic embedding vectors of the category concepts of each associated entity (similar to step S4) are used to select detection boxes in the two-dimensional spatial feature map. Within the neighborhood of the entity, calculate the local visual features corresponding to each selected spatial location u and the entity. Each related entity Similarity of category concept semantic embedding vectors ,exist Select similarities greater than the confidence threshold, calculate the number of associated entity categories corresponding to all selected similarities, and obtain the detection box. The number of related entity categories within the neighborhood region; where u is the index of the selected spatial location, U is the number of selected spatial locations, and d is the number of entity categories. The index of the associated entity, where D is the entity. The number of associated entities;
[0088] According to the detection frame The number of categories of associated entities within the neighborhood of the detection box confidence level Adjustments were made to obtain the final confidence level. It also generates explanations for knowledge entries, such as: "No 'refinery' was found around box #45 (oil tank) that is 'frequently co-occurring' in its knowledge definition. Based on the rule confidence / default coefficient, the confidence level drops from 0.88 to 0.62."
[0089] Preferably, for the detection frame confidence level The adjustments include: ;in, The attenuation factor is 0 < <1, n is the detection box The number of related entity categories within the neighborhood region, where R is the number of entities. The number of categories of associated entities, For detection box The confidence level, preferably, =0.7.
[0090] In one embodiment, the detection frame Soft constraint validation also includes calculating environmental attributes. For example, to verify whether it is located on a plain, the regional slope can be calculated; to verify whether it is adjacent to a water body, the water body index within the neighborhood can be calculated and its distribution evaluated. If the calculated attributes support the associations defined in the knowledge graph, the initial confidence level is maintained; if they clearly do not support them, the initial confidence level is reduced according to preset rules.
[0091] S64. For each valid checkbox... The detection results for target category i are obtained by sorting the results according to the final confidence scores. .
[0092] Output Simultaneously, the aforementioned generated structured explanatory text is attached to each result. For the rejected results, their explanation logs are also output. All explanations explicitly point to specific entities, attributes, and relationships in the knowledge graph, achieving complete traceability from the detection results to structured knowledge.
[0093] This invention deeply integrates incremental knowledge graphs with visual language models for open-vocabulary target detection in remote sensing imagery, effectively overcoming the shortcomings of existing closed-set detection methods, such as poor adaptability, insufficient utilization of professional knowledge, and uninterpretable results. Using a constructed dynamic and scalable knowledge graph as the core knowledge hub, it drives the generation of task instructions tailored to specific image conditions to guide visual perception, and constructs robust semantic concept prototypes for open vocabulary, namely category concept semantic embedding vectors. Furthermore, through lightweight adaptation and efficient alignment mechanisms, it achieves knowledge-enhanced cross-modal detection. Finally, it utilizes the hard constraints and soft associations inherent in the knowledge graph to automatically and interpretably verify and correct the results, forming a complete and reliable closed loop of "perception-knowledge-decision." The entire solution enables "out-of-the-box" open-vocabulary detection capabilities and, through knowledge-driven result verification and interpretation, endows the model with crucial logical consistency and decision transparency, providing a practical new paradigm for highly reliable intelligent remote sensing interpretation.
[0094] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for target detection in open vocabulary of remote sensing images, characterized in that, include: S1. Obtain publicly available remote sensing datasets and construct an initial remote sensing knowledge graph with remotely sensed ground objects as entities based on the publicly available remote sensing datasets; wherein, the initial remote sensing knowledge graph includes the attributes of entities and the relationships between entities; entities include multiple attributes, one of which is a core text attribute; S2. When the user inputs the remote sensing image to be detected and multiple target categories i, the current remote sensing knowledge graph is obtained, and the current remote sensing knowledge graph is queried to determine whether each target category i is a known category or a newly added category; the key knowledge of each known category is extracted from the current remote sensing knowledge graph, and the current remote sensing knowledge graph is incremented according to each newly added category to obtain the latest remote sensing knowledge graph, and the key knowledge of each newly added category is extracted from the latest remote sensing knowledge graph. S3. Combine the key knowledge of each target category i with the remote sensing image to be detected to generate knowledge-enhanced visual-text input pairs for each target category i. S4. Obtain the visual language model, and generate category concept semantic embedding vectors for each target category i based on the visual language model and the latest remote sensing knowledge graph; combine the category concept semantic embedding vectors for each target category i to obtain a dynamic detection vocabulary. The generated category concept semantic embedding vector for target category i includes: If target category i is a known class, then in the latest remote sensing knowledge graph Extract the core text attributes of the entity corresponding to the target category i, and input the extracted core text attributes into the text encoder of the visual language model to obtain the category concept semantic embedding vector of the target category i; If target category i is a newly added category, then based on the latest remote sensing knowledge graph... A visual language model is used to perform semantic completion and embedding on target category i, resulting in a category concept semantic embedding vector for target category i. Semantic completion and embedding for target category i includes: In the latest remote sensing knowledge graph Extract the core text attributes of the entity corresponding to the target category i, and input the extracted core text attributes into the text encoder of the visual language model to obtain the basic semantic embedding vector; In the latest remote sensing knowledge graph In this process, a hierarchical relationship chain of target category i is established through relational reasoning. The names of all entities in the hierarchical relationship chain are concatenated to obtain the hierarchical descriptive text of target category i. The hierarchical descriptive text of target category i is then input into the text encoder of the visual language model to obtain the hierarchical semantic embedding vector. In the latest remote sensing knowledge graph Find the associated entities that are related to the target category i and extract the names and relationship types of the associated entities. Construct the contextual description text of the target category i based on the names and relationship types of the associated entities of the target category i. Input the contextual description text of the target category i into the text encoder of the visual language model to obtain the contextual association embedding vector. The basic semantic embedding vector, hierarchical semantic embedding vector, and context-related embedding vector are fused to obtain the category concept semantic embedding vector of target category i. By combining the core text attributes, hierarchical descriptive text, and contextual descriptive text of target category i, a comprehensive descriptive text for target category i is obtained, incorporating the latest remote sensing knowledge graph. Replace the core text attributes of the entity corresponding to target category i with the comprehensive description text of target category i; S5. Based on the visual language model, dynamic detection vocabulary, and knowledge-enhanced visual-text input pair, perform visual-semantic cross-modal alignment and open vocabulary detection on each target category i to obtain the detection box set for each target category i. S6. Based on the latest remote sensing knowledge graph, perform spatial and semantic consistency verification on the detection box set of each target category i to obtain the detection results of each target category i.
2. The method for detecting open vocabulary targets in remote sensing images according to claim 1, characterized in that, The core text attributes are either comprehensive descriptive text or basic definition text; the comprehensive descriptive text is a natural language text that integrates the attributes and relationships of entities; based on each newly added class, the current remote sensing knowledge graph... Incremental processing includes: The system calls a multi-source information acquisition interface to retrieve information for each newly added class. Knowledge extraction and entity creation are then performed on the information for each newly added class to obtain its entities, basic definition text, and relationships. Finally, the entities, basic definition text, and relationships of each newly added class are added to the current remote sensing knowledge graph. To obtain the latest remote sensing knowledge graph Among them, the latest remote sensing knowledge graph The core text attribute of the entity corresponding to the known class is the comprehensive description text, and the core text attribute of the entity corresponding to the newly added class is the basic definition text.
3. The method for detecting open vocabulary targets in remote sensing images according to claim 1, characterized in that, Constructing knowledge-enhanced visual-text input pairs for target category i includes: S31. Standardize the remote sensing image to be detected to obtain multiple standardized remote sensing image tensors. S32. Combine the key knowledge of target category i with the metadata of the remote sensing image to be detected to generate the task instruction text for target category i. S33. Combine multiple standardized remote sensing image tensors with the task instruction text of target category i to obtain the knowledge-enhanced visual-text input pair of target category i.
4. The method for detecting open vocabulary targets in remote sensing images according to claim 3, characterized in that, Visual-semantic cross-modal alignment and open vocabulary detection for target category i include: S51. Input the task instruction text in the knowledge-enhanced visual-text input pair of target category i into the text encoder of the visual language model to obtain the task conditional semantic vector of target category i; perform conditional adaptation on the task conditional semantic vector of target category i to obtain the adaptation parameters of target category i. S52. Adjust the parameters of the visual encoder of the visual language model according to the adaptation parameters of the target category i to obtain the visual encoder of the target category i; input each normalized remote sensing image tensor in the knowledge-enhanced visual-text input pair of the target category i into the visual encoder of the target category i to obtain the local visual features of each normalized remote sensing image tensor. S53. Rearrange the local visual features of all standardized remote sensing image tensors according to their spatial positions in the remote sensing images to obtain a two-dimensional spatial feature map. S54. For each spatial location in the two-dimensional spatial feature map, calculate the similarity between its local visual features and the category concept semantic embedding vectors of each target category in the dynamic detection vocabulary, and select the highest similarity and its corresponding target category as its confidence and predicted label. S55. Set a confidence threshold, select spatial locations with confidence scores higher than the confidence threshold in the two-dimensional spatial feature map, and map each selected spatial location back to the coordinate system of the remote sensing image to be detected to obtain the coordinates of each selected spatial location. S56. Construct detection boxes based on the coordinates, predicted labels, and confidence scores of each selected spatial location. Integrate all detection boxes to obtain a set of detection boxes for target category i.
5. The method for detecting open vocabulary targets in remote sensing images according to claim 4, characterized in that, Each bounding box in the bounding box set includes: coordinates, predicted label, and confidence score; spatial and semantic consistency verification of the bounding box set for target category i includes: S61. The set of detection boxes for target category i Each detection box in In the latest remote sensing knowledge graph Search for its predicted tags The corresponding entity is used to obtain the detection box. entity In the latest remote sensing knowledge graph Obtain each detection box entity Related entities and common-sense attributes; among them, The index of the detection box in the detection box set for target category i, and the commonsense attribute is the latest remote sensing knowledge graph. The attribute has a range; S62, Based on each detection frame entity The common-sense attributes and their value ranges respectively correspond to the detection boxes. Perform hard constraint validation to obtain multiple valid detection boxes. Where m is the index of the valid detection box; S63. Based on each verified detection frame entity The associated entities respectively target the corresponding detection boxes Perform soft constraint validation to obtain each valid detection box. Final confidence level; S64. For each valid checkbox... The results are obtained by sorting the results according to their final confidence levels.
6. The method for detecting open vocabulary targets in remote sensing images according to claim 5, characterized in that, For the detection frame Hard constraint verification includes: within the detection bounding box of the remote sensing image to be detected. Calculate entities within the region respectively The actual attribute values of each common-sense attribute; if the actual attribute values of each common-sense attribute do not exceed their value range, then the detection box... The verification passed; otherwise, the verification failed.
7. The method for detecting open vocabulary targets in remote sensing images according to claim 5, characterized in that, For the detection boxes that pass the verification Performing soft constraint verification includes: With detection box Using the center as a reference, expand outwards by a preset frame size to obtain the detection frame. The neighboring region; In the collection of detection boxes Search for predicted labels for entities The category corresponding to any associated entity, and whose coordinates are located within the detection box. The detection boxes within the neighborhood region are used to obtain the associated detection box set; If the associated detection box set is not empty, then calculate the number of categories of all detection boxes in the associated detection box set to obtain the detection box count. The number of categories of associated entities within the neighborhood area; If the set of associated detection boxes is empty, then the entity is calculated using the visual language model. The semantic embedding vectors of the category concepts of each associated entity are used to select detection boxes in the two-dimensional feature map. Within the neighborhood of the entity, calculate the local visual features corresponding to each selected spatial location and their corresponding entities. The similarity of the semantic embedding vectors of the category concepts of each associated entity is calculated. Similarities greater than a confidence threshold are selected, and the number of associated entity categories corresponding to all selected similarities is calculated to obtain the detection box. The number of categories of associated entities within the neighborhood area; According to the detection frame The number of categories of associated entities within the neighborhood of the detection box confidence level Adjustments are made to obtain the final confidence level.
8. The method for detecting open vocabulary targets in remote sensing images according to claim 7, characterized in that, Detection box Final confidence level ; middle, The attenuation factor is 0 < <1, n is the detection box The number of related entity categories within the neighborhood region, where R is the number of entities. The number of categories of associated entities, For detection box The confidence level.