Cross-view target matching method based on semantic evidence graph and multi-modal large model

Abstract

The application discloses a cross-view target matching method based on a semantic evidence graph and a multi-modal large model, comprising the following steps: acquiring a UAV query image and a corresponding Top-k satellite candidate image set, generating an initial retrieval score for each satellite candidate image; converting the UAV query image and the Top-k satellite candidate image set into a structured object set; constructing an object-level semantic evidence graph; constructing a candidate difference table and a counter-evidence list; performing candidate evidence consistency reasoning and list-based reordering based on a multi-modal large model; calculating a final reordering score and outputting a final ordering result; and taking the satellite candidate image ranked first according to the final ordering result as the matching result of the UAV query image. The application has obvious advantages in precision, robustness, interpretability and system landing by cooperating the detection / segmentation model with the multi-modal large model, improving the cross-view target matching from the traditional feature similarity sorting to the object-level semantic evidence reasoning sorting.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and image matching technology, and in particular relates to a cross-view target matching method based on semantic evidence graphs and multimodal large models. Background Technology

[0002] Cross-view geolocalization aims to retrieve a corresponding area from an image taken from one perspective within an image or map database taken from another perspective. For example, it can retrieve corresponding satellite tiles from a large-scale satellite image database based on low-altitude oblique images taken by drones; or it can identify the same location from ground images within satellite images, drone images, or map tiles. This task has significant application value in fields such as drone navigation, disaster emergency response, urban management, target search, intelligent inspection, and robot localization.

[0003] Existing cross-view geolocation methods typically employ a two-stage process. The first stage extracts global or local features from UAV and satellite images using convolutional neural networks, visual Transformers, or cross-view matching networks, and then recalls Top-k candidate images from a satellite image database using cosine similarity, Euclidean distance, or learned metrics. The second stage outputs Top-1 or Top-k matching results based on the initial similarity list. Some methods also incorporate local attention, orientation alignment, polar coordinate transformation, multi-scale feature fusion, or re-ranking strategies to improve matching accuracy in complex scenes.

[0004] However, in real-world cross-view matching scenarios using drones, existing technologies still have the following problems.

[0005] First, the initial retrieval model can usually recall the correct satellite images from the Top-k candidates, but it is difficult to consistently select the most accurate Top-1 result. Due to differences in viewpoint, scale, occlusion, lighting variations, and shooting angles between drone images and satellite images, multiple candidate satellite images are often highly similar in overall texture or scene category. Relying solely on global feature similarity can easily lead to incorrectly ranking similar building clusters, similar road structures, or similar parking lots at the top.

[0006] Second, traditional reordering methods often rely on feature spatial distance or neighbor relationships, lacking an explicit understanding of road topology, building layout, landmark combinations, and spatial relationships. The correctness of cross-view matching often depends on structured semantic evidence such as "which side of the building is the road on," "whether the parking lot is adjacent to the long building," and "whether the relative relationships between the playground, water body, and road are consistent." Existing feature distance-based methods struggle to directly express these object-level relationships.

[0007] Third, directly inputting UAV images and multiple candidate satellite images into a multimodal large model for comparison is easily affected by factors such as image resolution, visual token compression, visibility of small targets, color differences, and shadow variations. Although the multimodal large model has strong visual semantic understanding capabilities, without structured evidence constraints such as detection, segmentation, and spatial relationships, it is prone to subjective judgments, overlooking small-scale features, or being misled by color and texture, leading to unstable re-ranking.

[0008] Fourth, existing cross-view matching systems typically only focus on "which candidate is more similar," lacking a systematic analysis of the differences between candidate satellite images. In reality, multiple candidates with similar roads or buildings often appear in the Top-k candidate pool, but they may differ significantly in key landmarks, road intersection patterns, building orientation, parking lot locations, or water boundaries. Analyzing the differences between candidates first, and then combining this with stable evidence from UAV images for a process of elimination, would help select more accurate matches from similar candidates.

[0009] In summary, existing cross-view target matching methods urgently need a technical solution that can organically combine detection and segmentation results, object-level spatial relationships, multimodal large-scale model semantic reasoning, and initial retrieval scores, so that the system can be upgraded from "overall image similarity ranking" to "object-level evidence consistency ranking", and can simultaneously utilize positive matching evidence and exclusionary counter-evidence to achieve more reliable Top-k re-ranking. Summary of the Invention

[0010] To address the aforementioned shortcomings in existing technologies, the cross-perspective target matching method based on semantic evidence graphs and multimodal large models provided by this invention solves the problems of insufficient matching accuracy, lack of explicit object-level understanding, numerous interference factors, and poor reordering stability in existing technologies.

[0011] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a cross-perspective target matching method based on semantic evidence graphs and multimodal large models, comprising: Obtain the drone query image and its corresponding Top-k satellite candidate image set, and generate an initial retrieval score for each satellite candidate image; Convert the UAV query images and their corresponding Top-k satellite candidate image sets into a set of structured objects; Constructing an object-level semantic evidence graph based on a structured object set; Construct a candidate difference table and a counter-evidence list based on the Top-k satellite candidate image set; The model takes the UAV query images, the Top-k satellite candidate image set, the initial retrieval score, the structured object set, the object-level semantic evidence graph, the candidate difference table, and the counter-evidence list as inputs to perform candidate evidence consistency reasoning and list-based reordering. The final re-ranking score is calculated based on the consistency reasoning of candidate evidence and the list-based re-ranking results, and the final ranking result is output based on the final re-ranking score. Based on the final ranking results, the top-ranked satellite candidate image is used as the matching result for the drone query image.

[0012] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a cross-view target matching method based on semantic evidence graphs and multimodal large models.

[0013] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform steps of a cross-view target matching method based on semantic evidence graphs and multimodal large models.

[0014] The beneficial effects of this invention are as follows: 1. Improve the re-ranking accuracy from Top-k to Top-1. This invention utilizes an initial cross-perspective retrieval model to ensure broad recall, and then employs object-level semantic evidence graphs and multimodal large-scale model inference to perform fine-grained judgment on Top-k candidates. This effectively improves the Top-1 matching accuracy even when the correct result has entered the Top-k but is not ranked first.

[0015] 2. Enhance the interpretability of cross-perspective matching. Traditional retrieval models typically only provide similarity scores, making it difficult to explain why a particular candidate is more reasonable. This invention can output positive evidence such as consistent road topology, consistent building layout, and matching parking lot locations, as well as negative evidence such as missing key roads, opposite landmark locations, and conflicting building numbers, facilitating manual review and system debugging.

[0016] 3. Reduce the instability of direct comparison between multimodal large models. By detecting, segmenting, and constructing object relationship graphs, this invention transforms the input of large models from unconstrained image observation into a joint input of the original image, overlay image, evidence card, and difference table. This subjectes the reasoning of large models to structured evidence constraints, thereby reducing misjudgments caused by overlooking small targets, color differences, shadow changes, or image scaling.

[0017] 4. Fully utilize the differences within the candidate set. This invention not only compares UAV images with each satellite candidate image, but also compares the key differences between the Top-k candidate satellite images. It can discover which candidates are easily confused, what the reasons for the confusion are, and which differences are most discriminative for UAV images, thereby achieving set-level list-based reordering.

[0018] 5. It has good versatility and deployability. As the second-stage re-ranking module of the cross-view matching system, this invention does not rely on a specific front-end retrieval model and can be integrated with existing UAV-satellite cross-view retrieval models, target geolocation models, or visual positioning systems; the detection and segmentation model and multimodal large model can also be replaced with local models or cloud models according to actual applications.

[0019] 6. Suitable for practical application scenarios of UAVs. This invention can be used for UAV self-localization, target geolocation, park inspection, emergency search and rescue, map updating, and visual-assisted positioning in GNSS-constrained environments, and can output more reliable satellite matching results in complex and similar scenarios. Attached Figure Description

[0020] Figure 1 A flowchart of a cross-view target matching method based on semantic evidence graphs and multimodal large models provided for an embodiment; Figure 2 A schematic diagram of the object-level semantic evidence graph construction module provided in this embodiment; Figure 3 This is a schematic diagram of the structure of the multimodal large model evidence consistency reasoning module provided in the embodiment. Detailed Implementation

[0021] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0022] like Figure 1 As shown, in one embodiment of the present invention, the cross-view target matching method based on semantic evidence graphs and multimodal large models includes the following steps: S1. Obtain the UAV query image and its corresponding Top-k satellite candidate image set, and generate an initial retrieval score for each satellite candidate image.

[0023] Step 1.1: Initial feature extraction.

[0024] An existing cross-view image retrieval model is used to perform initial recall of drone query images. This cross-view retrieval model can be FSRA, Sample4Geo, LPN, TransGeo, Game4Loc, or other models capable of drone-satellite image retrieval. This module is not part of this invention that requires retraining and can directly use an already trained model.

[0025] set up For drone image encoders, As a satellite image encoder, the query features are obtained after L2 normalization of the input image. and satellite image features :

[0026] In the formula, Query images for drones; Candidate images for satellites; This represents the total number of satellite candidate images.

[0027] Step 1.2: Establish the initial Top-k candidate set.

[0028] Cosine similarity was used to calculate the initial similarity between the UAV image and each satellite candidate image. :

[0029] Sort by similarity from high to low, and select the top... From candidate satellite images, a Top-k candidate set is obtained. : .

[0030] Each satellite candidate image retains its initial retrieval score. .

[0031] By constructing an initial candidate set, the problem of large-scale satellite image library retrieval is narrowed down to a small-scale candidate set, providing an input range for subsequent multimodal large model reordering.

[0032] In the specific implementation process, step S1 is executed by the cross-perspective candidate recall module.

[0033] S2. Convert the UAV query images and their corresponding Top-k satellite candidate image sets into a set of structured objects.

[0034] The UAV query image is obtained in step S1. and its corresponding Top-k satellite candidates gather Subsequently, target detection, semantic segmentation, or instance segmentation are performed on the UAV query images and candidate satellite images to extract stable ground objects that can be used for cross-view matching. The purpose of this step is to transform the original images into a structured set of objects containing object categories, geometric shapes, spatial locations, and confidence levels, providing basic input for subsequent object-level semantic evidence graph construction and multimodal large-scale model inference.

[0035] Step 2.1: Define a set of stable ground feature categories across different perspectives.

[0036] Define a stable set of land cover categories for cross-view matching, denoted as:

[0037] In the formula, This indicates the number of preset stable land cover categories. Indicates the first Stable land features include, but are not limited to, roads, buildings, building roofs, parking lots, playgrounds, sports fields, water bodies, bridges, squares, open spaces, green spaces, forests, vehicles, driveways, and parking spaces.

[0038] To differentiate the reliability of different land cover types in cross-view matching, a category weight mapping function is used to assign category weights to each type of stable land cover. :

[0039] In the formula, This represents the category weight mapping function. Roads, buildings, water bodies, playgrounds, parking lots, and other stable features across different viewpoints are typically assigned higher weights; vehicles, rows of trees, shadows, temporary obstructions, and other objects susceptible to changes over time, occlusion, or lighting conditions are typically assigned lower weights or used as supplementary evidence.

[0040] Step 2.2: Detect and segment the UAV query image.

[0041] This embodiment utilizes an open vocabulary object detection model, a semantic segmentation model, an instance segmentation model, or a combination thereof, to process drone query images. Belongs to the category set The system detects and segments stable ground objects. Specifically, the open vocabulary object detection model can use Grounding DINO, DINO-X, or YOLO-World, and the segmentation model can use SAM 2, SAM 2.1, or a combination of Grounded-SAM models. These models can directly use publicly available pre-trained weights and do not need to be retrained for the task of this invention.

[0042] Let the detection and segmentation operator be: ; The drone query image is detected and segmented to obtain a set of drone objects, which is expressed as:

[0043] In the formula, Represents a collection of drone objects; This indicates a drone query image; For detection and segmentation operators; A stable set of land cover categories for cross-view matching; Indicates the first in the drone query image One object; This indicates the number of objects detected in the image queried by the drone; Among them, the m Each drone detection target is represented as:

[0044] In the formula, For drones querying images of the first m One drone detection target; Indicates the object category label; Represents the bounding box of an object; Represents an object segmentation mask; Indicates the confidence level of detection or segmentation; Indicates the center point of the object; Indicates the proportion of the object's area; Indicates the main direction of the object; This indicates the relative position or region of an object within a drone query image; Among them, the center point of the object It can be calculated from the segmentation mask:

[0045]

[0046] In the formula, The x-coordinate of the object's center point; The ordinate of the object's center point; Indicates the number of pixels in the mask area; Represents the mask area The coordinates of any pixel in the array; Represents the mask area The x-coordinate of any pixel in the array; Represents the mask area The ordinate of any pixel in the array; object area ratio Specifically:

[0047] In the formula, To query the altitude of an image for a drone; The width of the image queried by the drone; object main direction The result is obtained through the minimum bounding rectangle of the mask region, principal component analysis, or second-order moment estimation. Taking principal component analysis as an example, let the set of pixels within the mask be: ; Perform eigenvalue decomposition on the pixel coordinate covariance matrix and extract the eigenvector corresponding to the largest eigenvalue. Then the main direction of the object is: .

[0048] The relative position or region of an object in a UAV query image is described by the normalized coordinates of the center point: .

[0049] Step 2.3: Detect and segment the Top-k satellite candidate images.

[0050] A method for detecting and segmenting UAV query images is used to detect and segment each satellite candidate image in the Top-k satellite candidate image set, thereby obtaining a set of satellite candidate objects.

[0051] Specifically, for satellite imagery, remote sensing building extraction models, road extraction models, water body segmentation models, remote sensing semantic segmentation models, open vocabulary remote sensing visual positioning models, or combinations thereof, can be prioritized to improve the accuracy of object recognition from a satellite perspective.

[0052] Let the detection and segmentation operators on the satellite image side be:

[0053] Then for the first Zhang Weixing candidate image Perform detection and segmentation to obtain a set of objects:

[0054] in, Indicates the first The number of objects detected in Zhang Weixing's candidate images Indicates the first The first of the candidate images of Zhang Weixing One object.

[0055] Arbitrary satellite object It can be represented as:

[0056] in, Indicates the object category; Represents the bounding box; Indicates the segmentation mask; Indicates the confidence level of detection or segmentation; Indicates the center point of the object; Indicates the proportion of the object's area; Indicates the main direction of the object; Indicates the relative position of an object in a satellite image.

[0057] The center point of the satellite object is defined as follows:

[0058] ; Area ratio is defined as:

[0059] in, and The first The height and width of the candidate satellite image.

[0060] The relative position of an object is defined as: .

[0061] object main direction Consistent with the calculation method for the principal orientation of UAV objects, it can be obtained through minimum bounding rectangle rotation, principal component analysis, or second-order moment estimation. For objects with obvious orientation, such as buildings, roads, bridges, runways, and parking lots, the principal orientation can serve as important geometric evidence for subsequent cross-view structural consistency assessments.

[0062] After this step, each satellite candidate image is converted into a set of objects. This forms a collection of drone objects. Comparable structured representations.

[0063] Step 2.4: Generate the detection segmentation coverage map.

[0064] To help the multimodal large model focus on key ground feature regions in subsequent steps, the system overlays the detection bounding box, segmentation mask, and object number onto the original image to generate a UAV detection and segmentation coverage map and a satellite candidate coverage map.

[0065] Let the covering graph generation function be: ; The overlay map of the drone query image is represented as follows: ; No. The overlay map of Zhang Weixing's candidate images is represented as follows:

[0066] in, This represents a drone detection and segmentation overlay map containing object detection bounding boxes, segmentation masks, and object numbers. Indicates the first Zhang is a satellite candidate coverage map with detection and segmentation results.

[0067] The purpose of overlay maps is twofold: firstly, to enable multimodal large models to quickly identify key areas such as roads, buildings, parking lots, water bodies, and green spaces; and secondly, to reduce misjudgments caused by image scaling, small targets, cluttered backgrounds, and differences in appearance across different viewpoints when directly observing the original image.

[0068] Step 2.5: Generate partial cropping images of key objects.

[0069] To further preserve local details, the system generates local clipping images of key objects based on object bounding boxes or segmentation masks. Key objects include, but are not limited to, road intersections, long buildings, building complexes, parking lots, playgrounds, water boundaries, bridges, and areas near the target.

[0070] Define the key object filtering function as follows:

[0071] In the formula, Filter functions for objects; Representation Object Category; This indicates the stability weight corresponding to this category; Indicates the confidence level of object detection or segmentation; Indicates the key object screening threshold; This indicates an indicator function; its value is 1 when the condition within the parentheses is met, and 0 otherwise. Applying a key object filtering function to the UAV object set and the satellite candidate object set yields the key object sets in the UAV query image and the satellite candidate image, respectively. The expression is as follows:

[0072]

[0073] In the formula, For drones querying images, a set of key objects is needed; A collection of drone objects; For the first Objects in Zhang Weixing's candidate images; For the first The set of satellite candidate objects corresponding to Zhang's satellite candidate images; For the first A set of key objects in Zhang Weixing's candidate images; The local clipping function is defined as follows:

[0074] In the formula, Indicates the input image; Indicates the object to be clipped; This indicates the cropping region expansion ratio; the local cropping function expands outwards from the bounding box of the object to be cropped in the input image. This multiplies the image to obtain a magnified view of the area containing the object to be cropped. Local cropping functions are applied to the key object set in the UAV query image and the key object set in the satellite candidate image to obtain the UAV key local cropping set and the satellite candidate key local cropping set, respectively.

[0075] The key local clipping set of the drone is represented as: ; The satellite candidate key local clipping set is represented as follows:

[0076] in, This represents the set of key local cropped images from the drone query image. Indicates the first A collection of key local cropped images from Zhang Weixing's candidate images.

[0077] Ultimately, the set of UAV objects, the set of satellite candidate objects, the UAV detection and segmentation overlay map and the satellite candidate overlay map, the set of key UAV local clippings, and the set of key satellite candidate local clippings constitute a structured object set.

[0078] In the specific implementation process, step S2 is executed by the detection and segmentation evidence extraction module.

[0079] S3. Construct an object-level semantic evidence graph based on a set of structured objects.

[0080] The drone object set is obtained in step S2. and satellite candidate set Next, this step constructs an object-level semantic evidence map of the objects and their spatial relationships in each image, which is used for subsequent candidate difference analysis and multimodal large model inference.

[0081] Step 3.1: Define the object-level semantic evidence graph.

[0082] For drone query images Construct an object-level semantic evidence graph:

[0083] in, This represents the set of object nodes in a drone image. Represents a set of object relation edges. This represents the global evidence attribute.

[0084] For the Zhang Weixing candidate image Construct an object-level semantic evidence graph:

[0085] in, , , They represent the first The set of object nodes, the set of object relationship edges, and global evidence attributes in Zhang Weixing's candidate images.

[0086] Step 3.2: Object node construction.

[0087] For a collection of drone objects: Each object in the drone object collection is converted into a node, resulting in a drone node collection: ; Based on the object representation in step 2: ; Define the node attributes for each node as follows:

[0088] In the formula, For the first m The attributes of the node corresponding to the drone detection object; Indicates the aspect ratio of the object; Represents the weight of evidence for the object.

[0089] If the bounding box is ,but: .

[0090] The weight of object evidence is defined as:

[0091] In the formula, Indicates stable features Category weights; Similarly, a method for constructing a set of drone nodes is used to construct a set of satellite candidate nodes based on a set of satellite candidate objects.

[0092] Construct a set of satellite candidate nodes: ; Its node attributes are: ; in: .

[0093] Step 3.3: Constructing spatial relationship edges.

[0094] For any two object nodes and The system constructs spatial relationship edges based on the object's center point, mask, bounding box, and main direction. :

[0095] in, It indicates the spatial relationship attributes between two things, including distance, intersection-to-union ratio, adjacency, directional angle, and relative orientation.

[0096] Define the spatial relationship attribute between any two object nodes as follows:

[0097]

[0098]

[0099]

[0100] In the formula, Represents any two object nodes and Spatial relationship attribute edge between them; Represents object nodes and Normalized center distance between them; Represents object nodes and The mask intersection and union; Represents object nodes and The difference in direction; Indicates relative directional relationships, including left, right, above, below, or east, west, south, north, etc. Indicates the type of relationship, including adjacent, containment, intersection, parallel, perpendicular, near, far, etc. and Each is an object node and The center point; and Each is an object node and The object mask; and Each is an object node and The main direction.

[0101] Based on spatial relationship attributes, sets of relationship edges are constructed for the UAV object set and the satellite candidate object set, respectively, and their expressions are as follows:

[0102]

[0103]

[0104] In the formula, This represents the set of edges corresponding to the set of drone nodes; Represents any two nodes in the drone node set. and Relationship edge; Represents a set of drone nodes; Indicates the first The set of edges corresponding to the set of satellite candidate nodes corresponding to the candidate images of Zhang satellite; Represents any two nodes in the set of satellite candidate nodes. and Relationship edge; Indicates the first The set of satellite candidate nodes corresponding to Zhang's satellite candidate images; Represents any two nodes and The relationship between them.

[0105] Step 3.4: Calculation of relational evidence weights.

[0106] Different spatial relationships have varying reliability in cross-perspective matching. For any relation edge... The weight of the relational evidence is defined as follows:

[0107] in, and This represents the evidence weights of two object nodes. This represents the reliability weighting function for relation types. Relationships such as the adjacency of roads and buildings, road topology, proximity of buildings and parking lots, water boundary relationships, and playground outline relationships are typically assigned higher weights; while vehicle quantity, shadow direction, color difference, and temporary occlusion relationships are typically assigned lower weights.

[0108] Step 3.5: Construct global evidence attributes.

[0109] We construct global evidence attributes for UAV query images and global evidence attributes for candidate images, respectively, with the following expressions:

[0110]

[0111] In the formula, This indicates the global evidence attributes of the drone query image; Indicates road topology attributes; Indicates building layout attributes; Indicates significant landmark attributes; Represents global spatial relationship attributes; Indicates the first Global evidence attributes of candidate images for Zhang Weixing's candidate images; , , , They represent the first The road topology attributes, building layout attributes, salient landmark attributes, and global spatial relationship attributes of the candidate images of Zhang Satellite.

[0112] Step 3.6: Constructing the evidence diagram.

[0113] Construct an object-level semantic evidence graph for drone query images based on the drone node set, the edge set corresponding to the drone node set, and the global evidence attributes of the drone query images. : ; Construct a satellite candidate image object-level semantic evidence graph based on the satellite candidate node set, the edge set corresponding to the satellite candidate node set, and the global evidence attributes of the candidate image. :

[0114] In the formula, This indicates the total number of satellite candidate images.

[0115] Step 3.7: Generate the object evidence card.

[0116] The object-level semantic evidence graphs of UAV query images and satellite candidate images are organized into UAV object evidence cards and candidate object evidence cards, respectively.

[0117] The object evidence card for drone images is represented as follows: ; No. The object evidence card for Zhang Weixing's candidate image is represented as follows: .

[0118] The object evidence card includes object statistics, a list of key objects, road topology, building layout, salient landmarks, spatial relationships between objects, reliable evidence, and unreliable evidence. Through the object evidence card, the complex visual information in the original image is transformed into structured semantic evidence that is easy to understand and compare using a multimodal, large-scale model.

[0119] After the above processing, step S3 finally outputs the drone object-level semantic evidence graph. Satellite object-level semantic evidence graph set drone object evidence card and satellite object evidence card set The above results serve as input for subsequent candidate difference analysis, multimodal large model semantic reasoning, and final cross-perspective reordering.

[0120] In the specific implementation process, step S3 is executed by the object-level semantic evidence graph construction module, and its structure is as follows: Figure 2 As shown.

[0121] S4. Construct a candidate difference table and a counter-evidence list based on the Top-k satellite candidate image set.

[0122] This step further compares the object composition, road topology, building layout, salient landmarks, and spatial relationships among the Top-k satellite candidates, generating a candidate difference table and a counter-evidence list to provide discriminative information for subsequent multimodal large model inference.

[0123] Step 4.1: Statistical comparison of candidate objects.

[0124] For the Zhang Weixing candidate image According to its collection of objects Count the number of various stable ground features to obtain object statistical vectors:

[0125] in, Indicates the first Zhang Weixing's candidate images are categorized as follows: The number of objects, This represents the total number of stable land cover categories.

[0126] For any two candidates and The statistical differences among its subjects can be expressed as :

[0127] in, This represents the norm. This difference is used to describe the differences in the quantity of objects such as roads, buildings, parking lots, water bodies, playgrounds, and green spaces among different candidates.

[0128] Step 4.2: Comparison of candidate road topologies.

[0129] The road topology attributes of the satellite candidate images are extracted from the candidate evidence cards and denoted as:

[0130] in, Indicates the direction of the main road. Indicates the type of road intersection. Indicates the degree of curvature of the road. It indicates the relative positional relationship between roads and building complexes or other landmarks.

[0131] The road topology difference between any two satellite candidate images is defined as:

[0132] In the formula, For any two satellite candidate images and Road topology differences; , , and These represent differences in road direction, intersection type, degree of curvature, and relative positional relationship, respectively. , , , All of these are weight parameters.

[0133] Step 4.3: Comparison of candidate building layouts.

[0134] Extract building layout attributes from satellite candidate images, denoted as:

[0135] in, Indicates the number of buildings. Indicates the main direction of the building. Indicates the shape or aspect ratio of a building. It indicates the spatial relationship between buildings and other land features such as roads, parking lots, and green spaces.

[0136] The difference in building layout between any two satellite candidate images is defined as:

[0137] In the formula, For any two satellite candidate images and Differences in architectural layout; Candidate images for satellites The number of buildings; Candidate images for satellites The number of buildings; Candidate images for satellites and Differences in the main orientation of the buildings; Represents satellite candidate images and Differences in the distribution of building shape or aspect ratio; Represents satellite candidate images and Differences in the spatial relationship between buildings and land features; , , , All of these are weight parameters.

[0138] Step 4.4: Comparison of candidate salient landmarks.

[0139] Extract landmark attributes from satellite candidate images, denoted as... :

[0140] In the formula, , , , , , These respectively represent the existence, quantity, or location attributes of parking lots, water bodies, playgrounds or sports fields, green spaces, bridges, and plazas. Define the landmark difference between any two satellite candidate images as :

[0141] In the formula, Represents a set of landmark categories; Indicates the first Importance weighting of landmark-like features; Represents satellite candidate images and In the Differences in landmarks.

[0142] Step 4.5: Generating easily confused candidate groups.

[0143] For any two satellite candidate images, define the candidate confusion level:

[0144]

[0145] In the formula, Represents satellite candidate images and Candidate confusion level; Indicates the candidate composite difference; exp represents the natural exponential function; and These are satellite candidate images. and The initial search score.

[0146] When the candidate confusion level exceeds a preset threshold, satellite candidate images will be... and These are marked as easily confused candidates. The system further extracts the most distinguishing objects, landmarks, or spatial relationships between the two, forming a candidate difference table. It is used to guide the comparison of key regions in multimodal large models.

[0147] A candidate difference table is formed based on the statistical differences of objects, road topology differences, building layout differences, landmark differences, and candidate confusion for each satellite candidate image.

[0148] Step 4.5: Counter-evidence mining.

[0149] Define the preset weighted evidence set in the drone query image as:

[0150] In the formula, This represents a pre-defined set of weighted evidence. Indicating evidence of drones Reliability score; This indicates a preset weighted evidence threshold; For the Zhang Weixing candidate images, searching for drone query image object-level semantic evidence graphs Semantic evidence maps of satellite candidate images at the object level Does the evidence correspond to the drone? If no corresponding evidence exists, or if there are contradictions in semantic, geometric, or spatial relationships, then the drone evidence will be considered valid. The record is used as counter-evidence. A set of counter-evidence is obtained for each satellite candidate image, generating a counter-evidence list. The specific counter-evidence set is as follows:

[0151] In the formula, Indicates the first A set of counter-evidence against Zhang Weixing's candidate images; Indicates the first The first in the set of counter-evidence against Zhang Weixing's candidate images One piece of counter-evidence; This represents evidence in the object-level semantic evidence graph of the drone query image; This represents the evidence matching function.

[0152] In the specific implementation process, step S4 is executed by the candidate difference diagnosis module.

[0153] S5. Take the UAV query image, Top-k satellite candidate image set, initial retrieval score, structured object set, object-level semantic evidence graph, candidate difference table and counter-evidence list as input to the multimodal large model, and perform candidate evidence consistency reasoning and list-based reordering.

[0154] Step 5.1: Construct a multimodal large model input.

[0155] For drone query images and its corresponding Top-k satellite candidate image set Constructing a multimodal large model input :

[0156] In the formula, This represents the coverage map segmented by the drone detection system. This represents a set of key local clippings from a drone. Indicates the first Satellite candidate overlay map corresponding to Zhang's satellite candidate image; Indicates the first The set of satellite candidate key local croppings corresponding to Zhang's satellite candidate images; Evidence card indicating the drone object; Evidence cards representing candidate entities; Indicates the initial search score. Represents the candidate difference table; Indicates the first A set of counter-evidence against Zhang Weixing's candidate images; Step 5.2: Set inference constraint prompts.

[0157] The inference constraint prompt This includes constraints on the reasoning basis and constraints on the output format, specifically: Weight priority constraint: Road topology, building layout, salient landmarks, and object spatial relationships have higher weights than color, shadows, temporary vehicles, and lighting textures; Output format constraints: For each candidate inference result, output at least the following fields: positive matching evidence field, negative exclusion evidence field, candidate key difference field, candidate score field, and final ranking field.

[0158] The reasoning process of a multimodal large model can be abstractly represented as:

[0159] in, This represents the output of a multimodal large model.

[0160] Step 5.3: Object evidence verification.

[0161] The multimodal large model first verifies the detection and segmentation results based on the original image, the detection and segmentation overlay map, and the object evidence card. For any candidate object evidence, the model can adjust its evidence weight based on visual consistency and semantic reasonableness, expressed as:

[0162] In the formula, This indicates the revised weight of evidence; Indicates the weight of the original evidence; This represents the validation coefficients of a large model; when the detection results are relatively reliable, Approaching 1; when there are false positives, false negatives, or semantic uncertainty, Less than 1. In subsequent object relationship graph matching and counter-evidence conflict calculation, if a piece of evidence has undergone multimodal large model verification, the corrected evidence weight will be used preferentially. .

[0163] Step 5.4: Evidence consistency scoring.

[0164] The multimodal large model scores each candidate satellite image from multiple dimensions. For the first... Zhang Candidate, Define the First The dimensional score vector of Zhang Weixing's candidate image is :

[0165] In the formula, Indicates the road topology consistency score; Indicates the building layout consistency score; Indicates a significant landmark consistency score; Indicates the consistency score of spatial relationships; Indicate the interpretability score of the perspective; The score indicates the degree of conflict of counter-evidence. Define the overall evidence consistency score as:

[0166] In the formula, Indicates the first The overall evidence consistency score of Zhang Weixing's candidate images; These are the weight parameters for each rating dimension.

[0167] Step 5.5: List-based reordering reasoning.

[0168] Input the multimodal large model The inference constraint prompts are input into a multimodal large model, and the overall candidate ranking for each satellite candidate image is output:

[0169] in, This represents the candidate ranking suggested by the multimodal large model. List-based reasoning avoids the scale inconsistency problem when scoring individual candidates independently and can make more stable judgments by utilizing the differences within the candidate set.

[0170] Step 5.6: Output the large model inference results.

[0171] The output of the multimodal large model includes a score for each candidate dimension. Consistency score of comprehensive evidence Suggested sorting The results include: optimal candidate number, confidence level, positive evidence supporting the optimal candidate, counter-evidence excluding other candidates, key differences between candidates, and uncertainties. These results serve as input for step S6, score fusion, and final re-ranking.

[0172] In the specific implementation process, step S5 is executed by the multimodal large model evidence consistency reasoning module, and its structure is as follows: Figure 3 As shown.

[0173] S6. Calculate the final re-ranking score based on the consistency reasoning of candidate evidence and the list-style re-ranking results, and output the final ranking result based on the final re-ranking score.

[0174] Based on the results of the multimodal large model inference, this step further combines the initial cross-perspective retrieval score, object relationship graph matching score, and counter-evidence conflict score to calculate the final re-ranking score.

[0175] Step 6.1: Calculate the object relationship graph matching score.

[0176] Calculate object relationship graph matching scores based on object-level semantic evidence graphs:

[0177]

[0178]

[0179] In the formula, For the first Zhang Weixing's candidate image's object relationship graph matching score; Indicates the matching score of the object node; The score represents the matching score of the object relationship edges; and These represent the categories of drone objects and satellite objects, respectively. and Indicates the weight of the evidence against the object; This indicates the similarity between the attributes of two object nodes; and Indicates the weight of the relation edge; and Indicates the attributes of the relation edges; Indicates spatial similarity; Step 6.2: Calculate the counter-evidence conflict score.

[0180] Calculate the first based on the set of counter-evidence. Counter-evidence conflict score of Zhang Weixing's candidate images:

[0181] In the formula, Indicates the weight of the counter-evidence type; Indicates the strength of counter-evidence.

[0182] Step 6.3: Merge the final reordered scores.

[0183] The final re-ranking score is calculated based on the initial retrieval score, overall evidence consistency score, object relationship graph matching score, and counter-evidence conflict score.

[0184] In the formula, For the first The final reorder score of Zhang Weixing's candidate images; All of these are weight parameters.

[0185] Step 6.4: Output the reordering results.

[0186] The images in the Top-k satellite candidate image set are rearranged from highest to lowest based on the final re-ranking score of each satellite candidate image, resulting in the final ranking: .

[0187] The candidate ranked first is the final matching result: .

[0188] When the difference between the highest and second-highest scores in the final ranking is less than the preset reliability threshold: , The current matching result will be marked as low confidence, and a prompt of "manual confirmation required" or "add image required" will be output.

[0189] In the specific implementation process, step S6 is executed by the score fusion and reordering module.

[0190] S7. Based on the final ranking results, the satellite candidate image ranked first is used as the matching result for the UAV query image.

[0191] Step 7.1: Output the final matching result.

[0192] Output the final ranked satellite image number, image location, corresponding satellite tile information, and final score after fusion. If the satellite imagery contains georeferenced information, the system can further output the geographic coordinates of the center of the candidate image or the target area.

[0193] Step 7.2: Output positive evidence explanations.

[0194] The system generates key evidence supporting the optimal candidate based on positive evidence output from the multimodal large model and object relationship graph matching results. This positive evidence includes, but is not limited to, consistency in road topology, building layout, salient landmarks, and spatial relationships between the target and surrounding objects. For example, the diagonal main road in the candidate image may have the same direction as the main road in the drone image; the adjacency relationship between the long, narrow buildings and the parking lot in the candidate image may be consistent with that in the drone image; and the number and arrangement of building clusters in the candidate image may be closer to those in the drone image.

[0195] Step 7.3: Output counter-evidence explanation.

[0196] The system uses the counter-evidence set Output the main evidence for excluding other candidates. The counter-evidence includes, but is not limited to, the absence of key road intersections, the absence of parking lots, bodies of water or playgrounds visible in the drone images, the opposite relative relationship between buildings and roads, and inconsistent road intersection types.

[0197] Step 7.4: Output candidate difference explanations.

[0198] The system outputs the key differences among the Top-k candidates, explaining which candidates are easily confused, the reasons for the confusion, and the discriminative criteria for selecting the optimal candidate. The explanation of candidate differences originates from the candidate difference table generated in step S4. And the list-style reordering reasoning results output in step S5.

[0199] Step 7.5: Output uncertainty prompt.

[0200] In low-confidence scenarios, the system outputs an uncertainty alert. If the score difference between the highest-scoring candidate and the second-ranked candidate is small, or if there are many uncertainties in the output of the multimodal large model, the system prompts the user or the upper-level system to supplement with higher-resolution UAV images, select key local areas, increase the UAV's observation perspective, expand the initial Top-k recall range, or conduct manual confirmation.

[0201] Thus, this invention completes the entire process from UAV query image acquisition, Top-k satellite candidate recall, detection and segmentation, object-level semantic evidence graph construction, candidate difference diagnosis, multimodal large model evidence consistency reasoning, to final reordering and interpretable output.

[0202] In the specific implementation process, step S7 is executed by the result interpretation and output module.

[0203] In summary, this invention, through the synergy of detection / segmentation models and multimodal large models, elevates cross-perspective target matching from traditional feature similarity ranking to object-level semantic evidence reasoning ranking, demonstrating significant advantages in accuracy, robustness, interpretability, and system feasibility.

Claims

1. A cross-perspective target matching method based on semantic evidence graphs and multimodal large models, characterized in that, include: Obtain the drone query image and its corresponding Top-k satellite candidate image set, and generate an initial retrieval score for each satellite candidate image; Convert the UAV query images and their corresponding Top-k satellite candidate image sets into a set of structured objects; Constructing an object-level semantic evidence graph based on a structured object set; Construct a candidate difference table and a counter-evidence list based on the Top-k satellite candidate image set; The model takes the UAV query images, the Top-k satellite candidate image set, the initial retrieval score, the structured object set, the object-level semantic evidence graph, the candidate difference table, and the counter-evidence list as inputs to perform candidate evidence consistency reasoning and list-based reordering. The final re-ranking score is calculated based on the consistency reasoning of candidate evidence and the list-based re-ranking results, and the final ranking result is output based on the final re-ranking score. Based on the final ranking results, the top-ranked satellite candidate image is used as the matching result for the drone query image.

2. The method according to claim 1, characterized in that, Obtain the drone query image and its corresponding Top-k satellite candidate image set, and generate an initial retrieval score for each satellite candidate image, including: Obtain drone query images and satellite images; The encoder is used to obtain query features and satellite image features based on UAV query images and satellite images; Calculate the initial retrieval score between the drone images and each satellite image; Based on the initial search scores, the top K satellite candidate images are selected from high to low to obtain the Top-k satellite candidate image set.

3. The method according to claim 1, characterized in that, Convert the UAV query images and their corresponding Top-k satellite candidate image sets into a set of structured objects, including: Define a set of stable feature categories for cross-view matching; the stable feature categories include roads, buildings, building roofs, parking lots, playgrounds, sports fields, water bodies, bridges, squares, open spaces, green spaces, forests, vehicles, driveways, and parking spaces; A category weight is assigned to each type of stable land cover using a category weight mapping function; The drone query image is detected and segmented to obtain a set of drone objects, which is expressed as: In the formula, Represents a collection of drone objects; This indicates a drone query image; For detection and segmentation operators; A stable set of land cover categories for cross-view matching; Indicates the first in the drone query image One object; This indicates the number of objects detected in the image queried by the drone; Among them, the m Each drone detection target is represented as: In the formula, For drones querying images of the first m One drone detection target; Indicates the object category label; Represents the bounding box of an object; Represents an object segmentation mask; Indicates the confidence level of detection or segmentation; Indicates the center point of the object; Indicates the proportion of the object's area; Indicates the main direction of the object; This indicates the relative position or region of an object within a drone query image; Among them, the center point of the object Specifically: In the formula, The x-coordinate of the object's center point; The ordinate of the object's center point; Indicates the number of pixels in the mask area; Represents the mask area The coordinates of any pixel in the array; Represents the mask area The x-coordinate of any pixel in the array; Represents the mask area The ordinate of any pixel in the array; object area ratio Specifically: In the formula, To query the altitude of an image for a drone; The width of the image queried by the drone; object main direction It is obtained by the minimum bounding rectangle of the mask region, principal component analysis, or second moment estimation. The relative position or region of an object in a UAV query image is described by the normalized coordinates of the center point: ； The method of detecting and segmenting UAV query images is used to detect and segment each satellite candidate image in the Top-k satellite candidate image set to obtain the satellite candidate object set; Based on the UAV object set and the satellite candidate object set, the detection box, segmentation mask and object number are superimposed on the corresponding UAV query image and satellite candidate image to generate UAV detection segmentation coverage map and satellite candidate coverage map; Generate partial clipping images of key objects: Define the key object filtering function as follows: In the formula, Filter functions for objects; Representation Object Category; This indicates the stability weight corresponding to this category; Indicates the confidence level of object detection or segmentation; Indicates the key object screening threshold; This indicates an indicator function; its value is 1 when the condition within the parentheses is met, and 0 otherwise. Applying a key object filtering function to the UAV object set and the satellite candidate object set yields the key object sets in the UAV query image and the satellite candidate image, respectively. The expression is as follows: In the formula, For drones querying images, a set of key objects is needed; A collection of drone objects; For the first Objects in Zhang Weixing's candidate images; For the first The set of satellite candidate objects corresponding to Zhang's satellite candidate images; For the first A set of key objects in Zhang Weixing's candidate images; The local clipping function is defined as follows: In the formula, Indicates the input image; Indicates the object to be clipped; This indicates the cropping region expansion ratio; the local cropping function expands outwards from the bounding box of the object to be cropped in the input image. This multiplies the image to obtain a magnified view of the area containing the object to be cropped. Apply local cropping functions to the key object set in the UAV query image and the key object set in the satellite candidate image to obtain the UAV key local cropping set and the satellite candidate key local cropping set, respectively. Among them, the set of UAV objects, the set of satellite candidate objects, the UAV detection and segmentation coverage map and the satellite candidate coverage map, the set of UAV key local clipping and the set of satellite candidate key local clipping constitute the set of structured objects.

4. The method according to claim 3, characterized in that, Constructing an object-level semantic evidence graph based on the aforementioned set of structured objects includes: Object node construction: Each object in the drone object collection is constructed as a node, resulting in a drone node collection; where each node has the following node attributes: In the formula, For the first m The attributes of the node corresponding to the drone detection object; Indicates the aspect ratio of the object; Indicates the weight of the evidence against the object; The weight of the object evidence is as follows: In the formula, Indicates stable features Category weights; A method for constructing a set of unmanned aerial vehicle (UAV) nodes is adopted to construct a set of satellite candidate nodes based on a set of satellite candidate objects; Spatial relationship construction: Define the spatial relationship attribute between any two object nodes as follows: In the formula, Represents any two object nodes and Spatial relationship attribute edge between them; Represents object nodes and Normalized center distance between them; Represents object nodes and The mask intersection and union; Represents object nodes and The difference in direction; Indicates relative orientation; Indicates the relation type; and Each is an object node and The center point; and Each is an object node and The object mask; and Each is an object node and The main direction; Based on spatial relationship attributes, sets of relationship edges are constructed for the UAV object set and the satellite candidate object set, respectively, and their expressions are as follows: In the formula, This represents the set of edges corresponding to the set of drone nodes; Represents any two nodes in the drone node set. and Relationship edge; Represents a set of drone nodes; Indicates the first The set of edges corresponding to the set of satellite candidate nodes corresponding to the candidate images of Zhang satellite; Represents any two nodes in the set of satellite candidate nodes. and Relationship edge; Indicates the first The set of satellite candidate nodes corresponding to Zhang's satellite candidate images; Represents any two nodes and Relationships between edges; Calculate arbitrary relation edges Relationship weights: In the formula, and This represents the evidence weights of two object nodes. Represents the reliability weighting function for relation types; We construct global evidence attributes for UAV query images and global evidence attributes for candidate images, respectively, with the following expressions: In the formula, This indicates the global evidence attributes of the drone query image; Indicates road topology attributes; Indicates building layout attributes; Indicates significant landmark attributes; Represents global spatial relationship attributes; Indicates the first Global evidence attributes of candidate images for Zhang Weixing's candidate images; , , , They represent the first The road topology attributes, building layout attributes, salient landmark attributes, and global spatial relationship attributes of Zhang Weixing's candidate images; Construct an object-level semantic evidence graph for drone query images based on the drone node set, the edge set corresponding to the drone node set, and the global evidence attributes of the drone query images. : ； Construct a satellite candidate image object-level semantic evidence graph based on the satellite candidate node set, the edge set corresponding to the satellite candidate node set, and the global evidence attributes of the candidate image. : In the formula, This indicates the total number of satellite candidate images; The object-level semantic evidence graphs of UAV query images and satellite candidate images are organized into UAV object evidence cards and candidate object evidence cards, respectively.

5. The method according to claim 4, characterized in that, A candidate difference table and a list of counter-evidence are constructed based on the Top-k satellite candidate image set, including: The number of various stable ground features is counted based on the satellite candidate object set corresponding to each satellite candidate image, and the object statistical vector is obtained. The absolute value of the difference between the object statistical vectors of any two satellite candidate images is defined as the object statistical difference. ; Extract road topology attributes from satellite candidate images from candidate evidence cards, including main road direction, road intersection type, road curvature, and relative positional relationship between roads and building complexes or other landmarks; The road topology difference between any two satellite candidate images is defined as: In the formula, For any two satellite candidate images and Road topology differences; , , and These represent differences in road direction, intersection type, degree of curvature, and relative positional relationship, respectively. , , , All are weighted parameters; Extract building layout attributes from satellite candidate images, including the number of buildings, main orientation of buildings, distribution of building shape or aspect ratio, and spatial relationship between buildings and ground features; The difference in building layout between any two satellite candidate images is defined as: In the formula, For any two satellite candidate images and Differences in architectural layout; Candidate images for satellites The number of buildings; Candidate images for satellites The number of buildings; Candidate images for satellites and Differences in the main orientation of the buildings; Represents satellite candidate images and Differences in building shape or aspect ratio; Represents satellite candidate images and Differences in the spatial relationship between buildings and land features; , , , All are weighted parameters; Extract landmark attributes from satellite candidate images, denoted as... : In the formula, , , , , , These respectively represent the existence, quantity, or location attributes of parking lots, water bodies, playgrounds or sports fields, green spaces, bridges, and plazas. Define the landmark difference between any two satellite candidate images as : In the formula, Represents a set of landmark categories; Indicates the first Importance weighting of landmark-like features; Represents satellite candidate images and In the Differences in landmarks; For any two satellite candidate images, define the candidate confusion level: In the formula, Represents satellite candidate images and Candidate confusion level; Indicates the candidate composite difference; exp represents the natural exponential function; and These are satellite candidate images. and The initial search score; When the candidate confusion level exceeds a preset threshold, satellite candidate images will be... and Mark as a potentially confusing candidate; A candidate difference table is formed based on the statistical differences of objects, road topology differences, building layout differences, landmark differences, and candidate confusion for each satellite candidate image; Define the preset weighted evidence set in the drone query image as: In the formula, This represents a pre-defined set of weighted evidence. Indicating evidence of drones Reliability score; This indicates a preset weighted evidence threshold; For the Zhang Weixing candidate images, searching for drone query image object-level semantic evidence graphs Semantic evidence map of satellite candidate images at the object level Does the evidence correspond to the drone? If no corresponding evidence exists, or if there are contradictions in semantic, geometric, or spatial relationships, then the drone evidence will be considered valid. The record is used as counter-evidence, thus obtaining the first... A set of counter-evidence for each satellite candidate image; a list of counter-evidence is generated based on the set of counter-evidence for each satellite candidate image; where, the first... The set of counter-evidence for Zhang Weixing's candidate images is as follows: In the formula, Indicates the first A set of counter-evidence against Zhang Weixing's candidate images; Indicates the first The first in the set of counter-evidence against Zhang Weixing's candidate images One piece of counter-evidence; This represents evidence in the object-level semantic evidence graph of the drone query image; This represents the evidence matching function.

6. The method according to claim 5, characterized in that, The model takes UAV query images, Top-k satellite candidate image sets, initial retrieval scores, structured object sets, object-level semantic evidence graphs, candidate difference tables, and counter-evidence lists as inputs to perform candidate evidence consistency reasoning and list-based reordering, including: For drone query images and its corresponding Top-k satellite candidate image set Constructing a multimodal large model input : In the formula, This represents the coverage map of drone detection segments; This represents a set of key local clippings from a drone. Indicates the first Satellite candidate overlay map corresponding to Zhang's satellite candidate image; Indicates the first The set of satellite candidate key local croppings corresponding to Zhang's satellite candidate images; Evidence card indicating the drone object; Evidence cards representing candidate entities; Indicates the initial search score. Represents the candidate difference table; Indicates the first A set of counter-evidence against Zhang Weixing's candidate images; Set inference constraint prompts, which include inference basis constraints and output format constraints, specifically: Weight priority constraint: Road topology, building layout, salient landmarks, and object spatial relationships have higher weights than color, shadows, temporary vehicles, and lighting textures; Output format constraints: For each candidate inference result, output at least the following fields: positive matching evidence field, negative exclusion evidence field, candidate key difference field, candidate score field, and final ranking field; The expression for the corrected evidence weights in the object-level semantic evidence graph is as follows: In the formula, This indicates the revised weight of evidence; Indicates the weight of the original evidence; Represents the validation coefficients of the large model; Definition of the first The dimensional score vector of Zhang Weixing's candidate image is : In the formula, Indicates the road topology consistency score; Indicates the building layout consistency score; Indicates a significant landmark consistency score; Indicates the consistency score of spatial relationships; The interpretability score is indicated by the perspective. The score indicates the degree of conflict of counter-evidence. Define the overall evidence consistency score as: In the formula, Indicates the first The overall evidence consistency score of Zhang Weixing's candidate images; These are the weighting parameters for each scoring dimension; Input the multimodal large model Inputting inference constraint prompts into a multimodal large model, the output is the overall candidate ranking for each satellite candidate image.

7. The method according to claim 6, characterized in that, The final re-ranking score is calculated based on the consistency reasoning of candidate evidence and the list-based re-ranking results. The final ranking results are then output based on the final re-ranking score, including: Calculate object relationship graph matching scores based on object-level semantic evidence graphs: In the formula, For the first Zhang Weixing's candidate image's object relationship graph matching score; Indicates the matching score of the object node; The score represents the matching score of the object relationship edges; and These represent the categories of drone objects and satellite objects, respectively. and Indicates the weight of the evidence against the object; This indicates the similarity between the attributes of two object nodes; and Indicates the weight of the relation edge; and Indicates the attributes of the relation edges; Indicates spatial similarity; Calculate the first based on the set of counter-evidence. Counter-evidence conflict score of Zhang Weixing's candidate images: In the formula, Indicates the weight of the counter-evidence type; Indicates the strength of counter-evidence; The final re-ranking score is calculated based on the initial retrieval score, overall evidence consistency score, object relationship graph matching score, and counter-evidence conflict score. In the formula, For the first The final reorder score of Zhang Weixing's candidate images; All are weighted parameters; The images in the Top-k satellite candidate image set are rearranged from high to low based on the final re-sorting score of each satellite candidate image, resulting in the final ranking.

8. The method according to claim 7, characterized in that, If the difference between the highest and second-highest scores in the final ranking is less than the preset confidence threshold, the current matching result is marked as low confidence, and a prompt of "manual confirmation required" or "supplement image" is output.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the cross-view target matching method based on semantic evidence graphs and multimodal large models as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The device stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the cross-view target matching method based on semantic evidence graphs and multimodal large models as described in any one of claims 1 to 8.

Images