A cross-view instance representation learning and retrieval positioning system and method

Abstract

This application discloses a cross-view instance representation learning and retrieval localization system and method, relating to the fields of computer vision, multi-view perception, and multimodal intelligent understanding. The cross-view instance representation learning and retrieval localization system includes: an input module, a target representation module, a candidate construction module, a reference building module, a relation encoding module, a semantic encoding module, a semantic fusion module, and a matching discrimination module. The cross-view instance representation learning and retrieval localization method includes the following steps: S100, data acquisition and preprocessing; S200, obtaining the target instance representation; S300, constructing a candidate instance set; S400, constructing a structural relation representation; S500, obtaining the fused semantic representation; S600, outputting the target localization result. This application outputs target localization results more accurately under complex task conditions, significantly improving the accuracy and stability of cross-view instance retrieval localization.

Description

Technical Field

[0001] This application relates to the fields of computer vision, multi-view perception and multimodal intelligent understanding, and in particular to a cross-view instance representation learning and retrieval localization system and method. Background Technology

[0002] With the development of intelligent robots, human-machine collaboration, augmented reality, intelligent monitoring, and first-person perspective (FPPS) technologies, cross-perspective scene understanding has gradually become an important research direction in the field of visual perception. First-person perspective refers to the viewpoint where a camera is mounted on a robot, wearable device, or the head / torso of a person, with the participant's own field of vision as the imaging center. This simulates the way the human eye directly observes a scene, and the perspective changes in real time with the movement of the device. Third-person perspective refers to the viewpoint where a camera is positioned at a fixed location or remote location within the scene environment, capturing a global view of the work scene and target objects from the perspective of an external observer. This viewpoint can completely cover the overall layout of the scene and the global state of the target. In practical applications, it is often necessary to retrieve, locate, and match the same physical instance in a third-person perspective image based on the target instance in the first-person perspective image.

[0003] For example, in human-robot collaboration scenarios, the operator can specify a target object through a first-person perspective device, and the system needs to accurately locate the target in the external perspective image to assist the robot in performing tasks such as grasping, tracking, interaction, or environmental understanding.

[0004] Existing cross-viewpoint instance matching technologies mostly rely on single appearance similarity modeling. By extracting visual features from target regions under different viewpoints and calculating similarity, cross-viewpoint target retrieval is achieved. However, in real-world scenarios, due to significant differences in viewpoints, scale variations, local occlusion, background interference, and dense distribution of similar instances between first- and third-view perspectives, relying solely on appearance features is insufficient to reliably distinguish instances with similar semantics but different identities, leading to a decrease in retrieval and localization accuracy.

[0005] In cross-view instance retrieval and localization scenarios, existing technologies have the following main shortcomings: 1. Significant differences in viewpoints lead to target representation drift, affecting matching stability. First-view and third-view images typically have significantly different imaging angles, visible areas, and spatial layouts. The appearance of the same instance differs considerably between the two viewpoints, causing a significant shift in traditional visual features during cross-viewpoint alignment, thus affecting the accuracy of target retrieval.

[0006] 2. Severe occlusion and background interference lead to a large number of redundant candidate regions. In third-person view images, target instances are often occluded by other objects or located in complex background environments. If retrieval and matching are performed directly across the entire image, it is easy to introduce a large number of irrelevant candidate regions, increasing the computational burden and reducing the reliability of subsequent matching decisions.

[0007] 3. Difficulty in distinguishing identities when similar instances are densely distributed. In scenarios such as desktop collaboration, indoor operations, and warehouse sorting, multiple instances of the same category often appear simultaneously, with similar colors, textures, and shapes. Traditional retrieval methods based on appearance similarity are prone to mismatching the target as adjacent similar instances, making it difficult to meet the requirement of precise instance-level positioning.

[0008] 4. Lack of effective modeling of cross-view spatial relationships. Most existing methods only focus on the visual features of the target itself, while ignoring the relative positional relationship, topological relationship and contextual structure information between the target and its surrounding reference objects. This results in the system lacking effective auxiliary judgment criteria when appearance is unreliable.

[0009] 5. Multimodal information is not fully utilized when there are language instructions or semantic constraints on the task. In practical applications, users often supplement the description of the target using natural language, but traditional methods struggle to model language information in a unified manner with visual and structural information, resulting in target retrieval results that fail to fully reflect user intent.

[0010] Therefore, those skilled in the art are dedicated to developing a cross-perspective instance representation learning and retrieval localization system and method. Summary of the Invention

[0011] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by this application is how to improve the accuracy and stability of cross-view target retrieval and localization.

[0012] The applicant argues that existing cross-viewpoint instance retrieval and localization methods typically determine target locations by performing global feature comparison, candidate region search, or directly relying on appearance similarity matching on the entire third-view image. However, these methods are prone to matching biases when the target's appearance changes significantly, local areas are occluded, there are numerous interfering targets in the background, or multiple similar instances appear simultaneously. Furthermore, existing technologies often lack stable cross-viewpoint reference relationships, making it difficult to effectively model the spatial organization between the target instance and surrounding reference instances. Therefore, they are not well-suited for instance-level retrieval and localization tasks in complex cross-viewpoint collaborative scenarios.

[0013] The applicant collects first-person perspective images to provide direct observation information of target instances, third-person perspective images to provide candidate retrieval space, and linguistic instruction information to provide semantic constraints for the task. This allows subsequent visual representation, structural relationship modeling, and matching discrimination to be simultaneously constrained by the target's appearance, spatial relationships, and semantic intent. A combination of target-guided candidate construction, shared reference instance establishment, local structural relationship encoding, and linguistic conditional semantic fusion is employed. Target representation generation and candidate region filtering reduce irrelevant search areas; shared reference instances establish cross-perspective local references; structural relationship encoding enhances the ability to distinguish between similar instances; and linguistic instruction information jointly modulates visual and structural information, thereby improving the accuracy and robustness of cross-perspective instance retrieval and localization.

[0014] In one embodiment of this application, a cross-perspective instance representation learning and retrieval localization system is provided, comprising: The input module acquires first-view images and obtains target region information from the first-view images; acquires third-view images; obtains language instructions, text prompts, or task semantic information related to the target instance and generates language instruction information; and performs preprocessing on the first-view images, third-view images, and language instruction information. The target representation module receives the first-view image, extracts features from the target region information, encodes them using a visual feature extraction network to obtain the initial feature representation of the target instance, and performs constrained extraction of the internal features of the target region to obtain the target instance representation. The candidate construction module performs similarity calculations based on the target instance representation and the third-view image to generate a response heatmap. It then performs image feature filtering and morphological adjustment on the response heatmap to generate candidate regions of interest. Within the candidate regions of interest, it performs instance segmentation or candidate region extraction to construct a set of candidate instances in the third-view image. The reference establishment module receives first-view images and third-view images, detects shared reference instances that co-occur with the target instance in the first-view images and third-view images respectively, and establishes cross-view reference correspondence. The relation encoding module uses the target instance and candidate instances as the central objects and the shared reference instance as the reference objects to calculate the edge features between the central objects and the reference objects, and constructs a structural relation representation. The semantic encoding module receives language instruction information, encodes it using an encoder, and generates a semantic representation. The semantic fusion module modulates and fuses the visual and structural information corresponding to the target instance and the candidate instance, based on the semantic representation, to obtain the fused semantic representation of the target instance and the fused semantic representation of the candidate instance, respectively. The matching and discrimination module sorts candidate instances based on the similarity between the fused semantic representation of the target instance and the fused semantic representation of the candidate instances, compares the sorting with a preset matching threshold, and outputs the target localization result in the third-view image. The input module is communicatively connected to the semantic encoding module, the target representation module, and the reference building module. The target representation module is communicatively connected to the input module, the candidate construction module, and the semantic fusion module. The candidate construction module is communicatively connected to the target representation module, the reference building module, the relation encoding module, and the semantic fusion module. The reference building module is communicatively connected to the input module, the candidate construction module, and the relation encoding module. The relation encoding module is communicatively connected to the reference building module, the candidate construction module, and the semantic fusion module. The semantic encoding module is communicatively connected to the input module and the semantic fusion module. The semantic fusion module is communicatively connected to the relation encoding module and the matching discrimination module.

[0015] Optionally, in the cross-view instance representation learning and retrieval localization system of the above embodiments, the input module includes: First-person view input unit, acquiring first-person view images. And target region information in the first-person view image; Third-view input unit, acquiring third-view images ; The semantic input unit acquires natural language instructions, text prompts, or task semantic information related to the target instance and generates language instruction information.

[0016] Optionally, in the cross-view instance representation learning and retrieval localization system of any of the above embodiments, the target region information includes a target bounding box, a target instance mask, or an interactively specified region.

[0017] Furthermore, in the cross-view instance representation learning and retrieval localization system described in the above embodiments, preprocessing includes image preprocessing and semantic information preprocessing.

[0018] Optionally, in the cross-view instance representation learning and retrieval localization system in the above embodiments, image preprocessing includes size unification, color space conversion, input format standardization, image normalization, and data inspection.

[0019] Optionally, in the cross-perspective instance representation learning and retrieval localization system in the above embodiments, semantic information preprocessing includes text cleaning, invalid character filtering, sentence truncation or completion, and text encoding interface adaptation.

[0020] Optionally, in the cross-view instance representation learning and retrieval localization system in the above embodiments, the visual feature extraction network uses a deep vision model with regional feature modeling capabilities to extract semantic information, texture information, contour information and local appearance information of the target region.

[0021] Optionally, in the cross-view instance representation learning and retrieval localization system of any of the above embodiments, the formula for similarity calculation is as follows: ; in, Position in a third-person perspective image The response value at that location, This is a first-view image feature map. This is a feature map of a third-view image. This represents the feature location inside the target instance mask of the first-person view image. Indicates the location of features in a third-view image. This represents the inner product operation. Represents the L2 norm, This represents the mask of the target instance in the first-person view image.

[0022] Furthermore, in the cross-view instance representation learning and retrieval localization system of the above embodiments, image feature filtering includes threshold filtering and connected region filtering.

[0023] Furthermore, in the cross-view instance representation learning and retrieval localization system in the above embodiments, morphological adjustment includes region expansion, local peak preservation, and image morphological processing.

[0024] Furthermore, in the cross-view instance representation learning and retrieval localization system of the above embodiments, the candidate instance set is denoted as... ,in, For a single candidate instance, N Indicates the number of candidate instances.

[0025] Furthermore, in the cross-view instance representation learning and retrieval localization system in the above embodiments, the shared reference instance is a target instance that is visible in both the first-view image and the third-view image, has stable boundaries, and is less occluded, providing a unified local reference relationship for the target instance and the candidate instance.

[0026] Furthermore, in the cross-view instance representation learning and retrieval localization system in the above embodiments, the less occluded target instance is defined as one whose occlusion rate does not exceed 40%.

[0027] Furthermore, in the cross-view instance representation learning and retrieval localization system of the above embodiments, multiple shared reference instances form a shared reference instance set, denoted as […]. ,in, Indicates the number of shared reference instances. For the first A shared reference instance.

[0028] Optionally, in the cross-view instance representation learning and retrieval localization system of any of the above embodiments, the edge feature calculation formula is as follows: ; in, As edge features, Represents candidate instances The scale parameter, Indicates a stable term. and Represents local basis vectors. For the first The candidate instance and the first The relative distance between shared reference instances Indicates the first The candidate instance and the first The relative distance between shared reference instances is taken as the L2 norm, i.e., the relative Euclidean distance. and These represent the projection components in the two local base directions relative to each other.

[0029] Optionally, in the cross-view instance representation learning and retrieval localization system of any of the above embodiments, the relative distance is represented as follows: ; in, For the first Candidate instance centers, For the first A shared reference instance center.

[0030] Optionally, in the cross-perspective instance representation learning and retrieval localization system of any of the above embodiments, the encoder includes a text encoder and a language model encoder.

[0031] Optionally, in the cross-view instance representation learning and retrieval localization system of any of the above embodiments, the calculation formula for the fused semantic representation of the target instance is as follows: ; in, For the fusion semantic representation of the target instance, For the visual modal weights of the target instance, For the topological modal weights of the target instance, and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of the target instance, reflecting visual information. This is a topological representation of the target instance after normalization and modulation, reflecting structural information.

[0032] Optionally, in the cross-view instance representation learning and retrieval localization system of any of the above embodiments, the calculation formula for the fused semantic representation of candidate instances is as follows: ; in, The fused semantic representation of candidate instances. For the visual modal weights of candidate instances, The topological modal weights of candidate instances. and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of candidate instances, reflecting visual information. This is a normalized modulated topological representation of candidate instances, reflecting structural information.

[0033] Optionally, in the cross-view instance representation learning and retrieval localization system of any of the above embodiments, the matching discrimination module sets up an empty matching discrimination mechanism, calculates the matching score corresponding to each candidate instance by using the similarity index between the fused semantic representation of the target instance and the fused semantic representation of the candidate instance; sorts all candidate instances with matching scores higher than a preset matching threshold, and takes the candidate instance with the highest matching score as the target localization result; when the matching scores of all candidate instances are lower than the preset matching threshold, the target localization result is target missing; and outputs the target localization result.

[0034] Furthermore, in the cross-perspective instance representation learning and retrieval localization system in the above embodiments, the preset matching threshold is 55%-75% of the highest matching score.

[0035] Preferably, in the cross-view instance representation learning and retrieval localization system of the above embodiments, the preset matching threshold is 60% of the highest matching score.

[0036] In another embodiment of this application, a cross-perspective instance representation learning and retrieval localization method is provided, including the following steps: S100. Data acquisition and preprocessing: Acquire first-view images and obtain target region information from the first-view images; acquire third-view images; obtain language instructions, text prompts, or task semantic information related to the target instance and generate language instruction information; perform preprocessing on the first-view images, third-view images, and language instruction information. S200. Obtain the target instance representation, extract features from the target region information, encode using a visual feature extraction network to obtain the initial feature representation of the target instance, and perform constrained extraction of the internal features of the target region to obtain the target instance representation. S300. Construct a candidate instance set. Based on the target instance representation, generate a response heatmap in the third-view image. Perform image feature filtering and morphological adjustment on the response heatmap to generate candidate regions of interest. Perform instance segmentation or candidate region extraction within the candidate regions of interest to construct a candidate instance set in the third-view image. S400, Construct a structural relationship representation, receive a first-view image and a third-view image, detect shared reference instances that co-occur with the target instance in the first-view image and the third-view image respectively, establish cross-view reference correspondence, and construct a structural relationship representation; S500: Obtain the fused semantic representation, receive language instruction information, encode it using an encoder, generate a semantic representation, and modulate and fuse it with the visual information and structural information corresponding to the target instance and candidate instance to obtain the fused semantic representation of the target instance and the fused semantic representation of the candidate instance respectively. S600. Output the target localization result. Sort the candidate instances according to the similarity between the fused semantic representation of the target instance and the fused semantic representation of the candidate instances, compare it with the preset matching threshold, and output the target localization result in the third-view image.

[0037] Optionally, in the cross-view instance representation learning and retrieval localization method in the above embodiments, step S100 includes: S110. Acquire first-view images. And target region information in the first-person view image; S120. Acquire third-view images. ; S130. Generate language instruction information, obtain natural language instructions, text prompts or task semantic information related to the target instance, and generate language instruction information; S140. Preprocessing: Perform preprocessing on the first-view image, third-view image, and language instruction information to ensure that the first-view image, third-view image, and language instruction information have a unified data organization method.

[0038] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, the preprocessing includes image preprocessing and semantic information preprocessing.

[0039] Optionally, in the cross-view instance representation learning and retrieval localization method in the above embodiments, image preprocessing includes size unification, color space conversion, input format standardization, image normalization, and data inspection.

[0040] Optionally, in the cross-perspective instance representation learning and retrieval localization method in the above embodiments, semantic information preprocessing includes text cleaning, invalid character filtering, sentence truncation or completion, and text encoding interface adaptation.

[0041] Optionally, in the cross-view instance representation learning and retrieval localization method of any of the above embodiments, step S200 includes: S210. Determine the target region, extract features from the target region information in the first-view image to obtain the target region, and separate the region range corresponding to the target to be matched from the first-view image. S220, Visual feature extraction: Visual feature extraction network is used to extract visual features from the target region to obtain the initial feature representation of the target instance, and the target region is converted into a feature representation suitable for cross-view matching. S230, Constraint Extraction: Constraint extraction is performed on the internal features of the target region to generate a target instance representation, retaining the effective features inside the target region and reducing the influence of the background region outside the target on the representation result.

[0042] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, S210 includes determining the target region based on the target box when the target region information is a target bounding box; accurately extracting the target region based on the target instance mask when the target region information is a target instance mask; and determining the target region based on the user interaction result when the target region information is an interactively specified region.

[0043] Optionally, in the cross-view instance representation learning and retrieval localization method in any of the above embodiments, the visual feature extraction network uses a deep vision model with regional feature modeling capabilities to extract semantic information, texture information, contour information and local appearance information of the target region.

[0044] Optionally, in the cross-view instance representation learning and retrieval localization method of any of the above embodiments, step S300 includes: S310. Obtain dense feature map: Receive a third-view image, obtain a dense feature map of the third-view image through a visual feature extraction network, and uniformly map the features of each position in the third-view image to a feature space with computable similarity. S320. Generate a response heatmap. Based on the correlation between the target instance representation and the dense feature map of the third-view image, perform similarity calculation to obtain the response value of each position in the third-view image to the target instance in the first-view image, and generate a response heatmap. S330. Filter candidate regions of interest, perform image feature filtering and morphological adjustment on the response heatmap, filter candidate regions of interest, and retain regions with high response values ​​and spatial continuity. S340. Generate candidate instances. Input the candidate regions of interest into the target perception network and perform instance segmentation or candidate region extraction to generate multiple candidate instances in the third-view image. S350. Construct a candidate instance set, extract the attribute information of each candidate instance, and form a candidate instance set in the third-view image from multiple candidate instances.

[0045] Optionally, in the cross-view instance representation learning and retrieval localization method of any of the above embodiments, the formula for similarity calculation is as follows: ; in, Position in a third-person perspective image The response value at that location, This is a first-view image feature map. This is a feature map of a third-view image. This indicates the feature location inside the mask of a target instance in a first-person view image. Indicates the location of features in a third-view image. This represents the inner product operation. Represents the L2 norm, This represents the mask of the target instance in the first-person view image.

[0046] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, image feature filtering includes threshold filtering and connected component filtering.

[0047] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, morphological adjustment includes region expansion, local peak preservation, and image morphological processing.

[0048] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, the target perception network includes an instance segmentation network or a candidate extraction network.

[0049] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, the attribute information includes region-level visual features, geometric center location, region area, and scale parameters.

[0050] Furthermore, in the cross-perspective instance representation learning and retrieval localization method in the above embodiments, the candidate instance set is denoted as... ,in, For a single candidate instance, N Indicates the number of candidate instances.

[0051] Optionally, in the cross-view instance representation learning and retrieval localization method of any of the above embodiments, step S400 includes: S410. Establish cross-view reference correspondence. Detect shared reference instances that co-occur with the target instance in the first-view image and the third-view image respectively. Establish cross-view reference correspondence based on visual consistency, region correspondence or local response consistency. S420. Generate a structured relation representation, using the target instance and candidate instance as the central object and the shared reference instance as the reference object, and encode the relative distance, directional relationship and scale relationship between the central object and the reference object to generate a structured relation representation. S430. Calculate edge features. Calculate the edge features between the central object and the reference object, and represent the relative Euclidean distance between the central object and the reference object and the projection relationship on the local basis direction as structured features.

[0052] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, the shared reference instance is a target instance that is visible in both the first-view image and the third-view image, has stable boundaries, and is less occluded, providing a unified local reference relationship for the target instance and the candidate instance.

[0053] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, the target instance with less occlusion is defined as one whose occlusion rate does not exceed 40% of the target instance.

[0054] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, multiple shared reference instances form a shared reference instance set, denoted as . ,in, Indicates the number of shared reference instances. For the first A shared reference instance.

[0055] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, the edge feature calculation formula is as follows: ; in, As edge features, Represents candidate instances The scale parameter, Indicates a stable term. and Represents local basis vectors. For the first The candidate instance and the first The relative distance between shared reference instances Indicates the first The candidate instance and the first The relative distance between shared reference instances is taken as the L2 norm, i.e., the relative Euclidean distance. and These represent the projection components in the two local base directions relative to each other.

[0056] Furthermore, in the cross-view instance representation learning and retrieval localization method in the above embodiments, the relative distance is represented as follows: ; in, For the first Candidate instance centers, For the first A shared reference instance center.

[0057] Optionally, in the cross-view instance representation learning and retrieval localization method of any of the above embodiments, step S500 includes: S510. Generate semantic representation: Receive language instruction information, encode it using an encoder, and generate semantic representation. S520. Generate a fused semantic representation of the target instance. Using the semantic representation as a condition, modulate and fuse the visual information and structural information corresponding to the target instance, and generate a fused semantic representation of the target instance through the fused semantic representation calculation formula. S530. Generate a fusion semantic representation of candidate instances. Using the semantic representation as a condition, modulate and fuse the visual and structural information corresponding to the candidate instances, and generate a fusion semantic representation of the candidate instances through the fusion semantic representation calculation formula.

[0058] Optionally, in the cross-perspective instance representation learning and retrieval localization method in any of the above embodiments, the encoder includes a text encoder and a language model encoder.

[0059] Furthermore, in the cross-perspective instance representation learning and retrieval localization method in the above embodiments, the calculation formula for the fused semantic representation of the target instance is as follows: ; in, For the fusion semantic representation of the target instance, For the visual modal weights of the target instance, For the topological modal weights of the target instance, and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of the target instance, reflecting visual information. This is a topological representation of the target instance after normalization and modulation, reflecting structural information.

[0060] Furthermore, in the cross-perspective instance representation learning and retrieval localization method in the above embodiments, the calculation formula for the fused semantic representation of candidate instances is as follows: ; in, The fused semantic representation of candidate instances. For the visual modal weights of candidate instances, The topological modal weights of candidate instances. and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of candidate instances, reflecting visual information. This is a normalized modulated topological representation of candidate instances, reflecting structural information.

[0061] Optionally, in the cross-view instance representation learning and retrieval localization method of any of the above embodiments, step S600 includes: S610. Calculate the matching score by using the similarity index between the fused semantic representation of the target instance and the fused semantic representation of the candidate instance to obtain the matching score corresponding to each candidate instance. S620. Output the target localization result. Sort all candidate instances with matching scores higher than the preset matching threshold, and take the candidate instance with the highest matching score as the target localization result. When the matching scores of all candidate instances are lower than the preset matching threshold, the target localization result is "target missing". Output the target localization result.

[0062] Furthermore, in the cross-perspective instance representation learning and retrieval localization method in the above embodiments, the preset matching threshold is 55%-75% of the highest matching score.

[0063] Preferably, in the cross-perspective instance representation learning and retrieval localization method in the above embodiments, the preset matching threshold is 60% of the highest matching score.

[0064] This application achieves synchronous access and unified processing of cross-view images, prioritizing the selection of candidate regions related to the target in third-view images, thus reducing redundant search processes; it enhances the ability to distinguish identities under complex backgrounds and conditions with dense similar instances; and it jointly modulates linguistic semantics with visual and structural information to output target localization results more accurately under complex task conditions, significantly improving the accuracy and stability of cross-view instance retrieval and localization.

[0065] The following will further explain the concept, specific structure and technical effects of this application in conjunction with the accompanying drawings, so as to fully understand the purpose, features and effects of this application. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the structure of a cross-perspective instance representation learning and retrieval localization system, as an exemplary embodiment. Figure 2 This is a flowchart of a cross-perspective instance representation learning and retrieval localization method, which is an exemplary embodiment. Detailed Implementation

[0067] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of this application to make its technical content clearer and easier to understand. This application can be embodied in many different forms, and the scope of protection of this application is not limited to the embodiments mentioned herein.

[0068] The applicant designed a cross-perspective instance representation learning and retrieval localization system, such as Figure 1 As shown, it includes: The input module acquires first-person perspective images and obtains target region information from these images, including bounding boxes, target instance masks, or interactively specified regions. It also acquires third-person perspective images, obtains language instructions, text prompts, or task semantic information related to the target instance, and generates language instruction information. The module performs preprocessing on the first-person and third-person perspective images and the language instruction information, including image preprocessing and semantic information preprocessing. Image preprocessing includes size unification, color space conversion, input format standardization, image normalization, and data checking. Semantic information preprocessing includes text cleaning, invalid character filtering, sentence truncation or completion, and text encoding interface adaptation. First-person view input unit, acquiring first-person view images. And target region information in the first-person view image; Third-view input unit, acquiring third-view images ; The semantic input unit acquires natural language instructions, text prompts, or task semantic information related to the target instance and generates language instruction information.

[0069] The target representation module receives the first-view image, extracts features from the target region information, encodes them using a visual feature extraction network to obtain the initial feature representation of the target instance, and performs constrained extraction of the internal features of the target region to obtain the target instance representation. The visual feature extraction network uses a deep vision model with region feature modeling capabilities to extract semantic information, texture information, contour information, and local appearance information of the target region.

[0070] The candidate construction module performs similarity calculations based on the target instance representation and the third-view image to generate a response heatmap. It then performs image feature filtering and morphological adjustment on the response heatmap to generate candidate regions of interest (ROIs). Within these ROIs, it performs instance segmentation or region extraction to construct a set of candidate instances in the third-view image. This set of candidate instances is denoted as […]. ,in, For a single candidate instance, N This represents the number of candidate instances; the formula for similarity calculation is as follows: ; in, Position in a third-person perspective image The response value at that location, This is a first-view image feature map. This is a feature map of a third-view image. This represents the feature location inside the target instance mask of the first-person view image. Indicates the location of features in a third-view image. This represents the inner product operation. Represents the L2 norm, This represents a target instance mask in a first-person view image; image feature filtering includes threshold filtering and connected region filtering, and morphological adjustment includes region expansion, local peak preservation, and image morphological processing.

[0071] The reference establishment module receives first-view and third-view images, and detects shared reference instances that co-occur with the target instance in both the first-view and third-view images, establishing cross-view reference correspondences. Shared reference instances are target instances that are visible in both the first-view and third-view images, have stable boundaries, and are minimally occluded, providing a unified local reference relationship for the target instance and candidate instances. Minimal occlusion in a target instance is defined as an occlusion rate not exceeding 40% of the target instance. Multiple shared reference instances form a shared reference instance set, denoted as […]. ,in, Indicates the number of shared reference instances. For the first A shared reference instance.

[0072] The relation encoding module uses the target instance and candidate instances as central objects and the shared reference instance as reference objects to calculate the edge features between the central objects and the reference objects, thus constructing a structural relation representation. The edge feature calculation formula is as follows: ; in, As edge features, Represents candidate instances The scale parameter, Indicates a stable term. and Represents local basis vectors. For the first The candidate instance and the first The relative distance between shared reference instances Indicates the first The candidate instance and the first The relative distance between shared reference instances is taken as the L2 norm, i.e., the relative Euclidean distance. and These represent the projection components of the relative distance along the two local base directions; relative distance It is expressed as follows: ; in, For the first Candidate instance centers, For the first A shared reference instance center.

[0073] The semantic encoding module receives language instruction information, encodes it using an encoder, and generates a semantic representation. The encoder includes a text encoder and a language model encoder. The semantic fusion module modulates and fuses the visual and structural information corresponding to the target instance and candidate instances, based on semantic representation, to obtain the fused semantic representation of the target instance and the fused semantic representation of the candidate instances, respectively. The calculation formula for the fused semantic representation of the target instance is as follows: ; in, For the fusion semantic representation of the target instance, For the visual modal weights of the target instance, For the topological modal weights of the target instance, and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of the target instance, reflecting visual information. The normalized modulated topological representation of the target instance reflects structural information; the fused semantic representation of the candidate instance is calculated using the following formula: ; in, The fused semantic representation of candidate instances. For the visual modal weights of candidate instances, The topological modal weights of candidate instances. and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of candidate instances, reflecting visual information. This is a normalized modulated topological representation of candidate instances, reflecting structural information.

[0074] The matching and discrimination module sorts candidate instances based on the similarity between the fused semantic representation of the target instance and the fused semantic representation of the candidate instances, compares the sorted instances with a preset matching threshold, and outputs the target localization result in the third-view image. The matching and discrimination module also includes an empty matching discrimination mechanism, which calculates a matching score for each candidate instance using a similarity index between the fused semantic representation of the target instance and the fused semantic representation of the candidate instances. All candidate instances with matching scores higher than the preset matching threshold are sorted, and the candidate instance with the highest matching score is taken as the target localization result. When the matching scores of all candidate instances are lower than the preset matching threshold, the target localization result is "target missing." The target localization result is then output, with the preset matching threshold being 60% of the highest matching score.

[0075] The input module is communicatively connected to the semantic encoding module, the target representation module, and the reference building module. The target representation module is communicatively connected to the input module, the candidate construction module, and the semantic fusion module. The candidate construction module is communicatively connected to the target representation module, the reference building module, the relation encoding module, and the semantic fusion module. The reference building module is communicatively connected to the input module, the candidate construction module, and the relation encoding module. The relation encoding module is communicatively connected to the reference building module, the candidate construction module, and the semantic fusion module. The semantic encoding module is communicatively connected to the input module and the semantic fusion module. The semantic fusion module is communicatively connected to the relation encoding module and the matching discrimination module.

[0076] Based on the above embodiments, the applicant provides a cross-perspective instance representation learning and retrieval localization method, such as... Figure 2 As shown, it includes the following steps: S100. Data Acquisition and Preprocessing: Acquire first-view images and obtain target region information from the first-view images; acquire third-view images; obtain language instructions, text prompts, or task semantic information related to the target instance, and generate language instruction information; perform preprocessing on the first-view images, third-view images, and language instruction information; specifically including: S110. Acquire first-view images. And target region information in the first-person view image; S120. Acquire third-view images. ; S130. Generate language instruction information, obtain natural language instructions, text prompts or task semantic information related to the target instance, and generate language instruction information; S140. Preprocessing: Perform preprocessing on the first-view image, third-view image, and language instruction information to ensure that the first-view image, third-view image, and language instruction information have a unified data organization method. Preprocessing includes image preprocessing and semantic information preprocessing. Image preprocessing includes size unification, color space conversion, input format standardization, image normalization, and data checking. Semantic information preprocessing includes text cleaning, invalid character filtering, sentence truncation or completion, and text encoding interface adaptation.

[0077] S200. Obtain the target instance representation, extract features from the target region information, encode them using a visual feature extraction network to obtain the initial feature representation of the target instance, and perform constrained extraction of the internal features of the target region to obtain the target instance representation; specifically including: S210. Determine the target region by extracting features from the target region information in the first-view image to obtain the target region, and separating the region range corresponding to the target to be matched from the first-view image; including determining the target region based on the target bounding box when the target region information is a target bounding box; accurately extracting the target region based on the target instance mask when the target region information is a target instance mask; and determining the target region based on the user interaction result when the target region information is an interactively specified region. S220, Visual Feature Extraction: The visual feature extraction network is used to extract visual features from the target region to obtain the initial feature representation of the target instance, and the target region is converted into a feature representation suitable for cross-view matching. The visual feature extraction network uses a deep vision model with the ability to model regional features to extract semantic information, texture information, contour information and local appearance information of the target region. S230, Constraint Extraction: Constraint extraction is performed on the internal features of the target region to generate a target instance representation, retaining the effective features inside the target region and reducing the influence of the background region outside the target on the representation result.

[0078] S300. Construct a candidate instance set. Based on the target instance representation, generate a response heatmap in the third-view image. Perform image feature filtering and morphological adjustment on the response heatmap to generate candidate regions of interest. Perform instance segmentation or candidate region extraction within the candidate regions of interest to construct a candidate instance set in the third-view image. Specifically, this includes: S310. Obtain dense feature map: Receive a third-view image, obtain a dense feature map of the third-view image through a visual feature extraction network, and uniformly map the features of each position in the third-view image to a feature space with computable similarity. S320. Generate a response heatmap. Based on the correlation between the target instance representation and the dense feature map of the third-view image, perform similarity calculation to obtain the response value of each position in the third-view image to the target instance in the first-view image, and generate a response heatmap. The formula for similarity calculation is as follows: ; in, Position in a third-person perspective image The response value at that location, This is a first-view image feature map. This is a feature map of a third-view image. This indicates the feature location inside the mask of a target instance in a first-person view image. Indicates the location of features in a third-view image. This represents the inner product operation. Represents the L2 norm, Represents a mask for target instances in a first-person view image; S330. Filter candidate regions of interest, perform image feature filtering and morphological adjustment on the response heatmap, filter candidate regions of interest, and retain regions with high response values ​​and spatial continuity; image feature filtering includes threshold filtering and connected region filtering, and morphological adjustment includes region expansion, local peak preservation and image morphological processing. S340. Generate candidate instances by inputting the candidate regions of interest into the target perception network and performing instance segmentation or candidate region extraction to generate multiple candidate instances in the third-view image. The target perception network includes an instance segmentation network or a candidate extraction network. S350. Construct a candidate instance set, extract the attribute information of each candidate instance, including region-level visual features, geometric center location, region area, and scale parameters. Multiple candidate instances form a candidate instance set in the third-view image, denoted as [the candidate instance set is missing here]. ,in, For a single candidate instance, N Indicates the number of candidate instances.

[0079] S400, Constructing a structural relation representation: Receive a first-view image and a third-view image; detect shared reference instances that co-occur with the target instance in both the first-view image and the third-view image; establish cross-view reference correspondence; and construct a structural relation representation; specifically including: S410. Establish cross-view reference correspondence. Detect shared reference instances that co-occur with the target instance in both the first-view and third-view images. Establish cross-view reference correspondence based on visual consistency, region correspondence, or local response consistency. Shared reference instances are target instances that are visible in both the first-view and third-view images, have stable boundaries, and are minimally occluded. They provide a unified local reference relationship for the target instance and candidate instances. Minimal occlusion in a target instance is defined as an occlusion rate not exceeding 40% of the target instance. Multiple shared reference instances form a shared reference instance set, denoted as […]. ,in, Indicates the number of shared reference instances. For the first A shared reference instance; S420. Generate a structured relation representation, using the target instance and candidate instance as the central object and the shared reference instance as the reference object, and encode the relative distance, directional relationship and scale relationship between the central object and the reference object to generate a structured relation representation. S430. Calculate edge features: Calculate the edge features between the central object and the reference object, and uniformly represent the relative Euclidean distance between the central object and the reference object and their projection relationship on the local basis directions as structured features; the edge feature calculation formula is as follows: ; in, As edge features, Represents candidate instances The scale parameter, Indicates a stable term. and Represents local basis vectors. For the first The candidate instance and the first The relative distance between shared reference instances Indicates the first The candidate instance and the first The relative distance between shared reference instances is taken as the L2 norm, i.e., the relative Euclidean distance. and These represent the projection components of the relative distance along the two local base directions; relative distance It is expressed as follows: ; in, For the first Candidate instance centers, For the first A shared reference instance center.

[0080] S500: Obtain the fused semantic representation. Receive language instruction information, encode it using an encoder, generate a semantic representation, and modulate and fuse it with the visual and structural information corresponding to the target instance and candidate instances to obtain the fused semantic representation of the target instance and the fused semantic representation of the candidate instances, respectively. Specifically, this includes: S510, Generate semantic representation: Receive language instruction information, encode it using an encoder, including a text encoder and a language model encoder, and generate semantic representation; S520. Generate a fused semantic representation of the target instance. Using the semantic representation as a condition, modulate and fuse the visual and structural information corresponding to the target instance. Generate the fused semantic representation of the target instance using the fused semantic representation calculation formula. The calculation formula for the fused semantic representation of the target instance is as follows: ; in, For the fusion semantic representation of the target instance, For the visual modal weights of the target instance, For the topological modal weights of the target instance, and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of the target instance, reflecting visual information. This is a normalized modulated topological representation of the target instance, reflecting structural information; S530. Generate a fused semantic representation of candidate instances. Using the semantic representation as a condition, modulate and fuse the visual and structural information corresponding to the candidate instances. Generate a fused semantic representation of the candidate instances using the fused semantic representation calculation formula. The fused semantic representation calculation formula for candidate instances is as follows: ; in, The fused semantic representation of candidate instances. For the visual modal weights of candidate instances, The topological modal weights of candidate instances. and These are learnable parameters generated from the semantic representation output by the semantic encoding module, reflecting the importance of different modalities in the current task context. This is a normalized modulated visual representation of candidate instances, reflecting visual information. This is a normalized modulated topological representation of candidate instances, reflecting structural information.

[0081] S600. Output the target localization result. Based on the similarity between the fused semantic representation of the target instance and the fused semantic representation of the candidate instances, rank the candidate instances and compare them with a preset matching threshold. Output the target localization result in the third-view image. Specifically, this includes: S610. Calculate the matching score by using the similarity index between the fused semantic representation of the target instance and the fused semantic representation of the candidate instance to obtain the matching score corresponding to each candidate instance. S620. Output the target localization result. Sort all candidate instances with matching scores higher than the preset matching threshold, and take the candidate instance with the highest matching score as the target localization result. When the matching scores of all candidate instances are lower than the preset matching threshold, the target localization result is "target missing". Output the target localization result. The preset matching threshold is 60% of the highest matching score.

[0082] To verify the technical effectiveness of the above embodiments in end-to-end application scenarios, the applicant conducted experiments based on publicly available cross-view datasets. The experiments used first-view images, third-view images, and language instructions related to the target instance as input. Based on the target region information and language instruction information in the first-view image, the system automatically completed candidate region generation, candidate instance extraction, shared reference instance establishment, structural relationship encoding, semantic fusion, and matching discrimination in the third-view image, and finally output the target localization result in the third-view image.

[0083] Unlike experiments that only validate candidate matching modules, end-to-end experiments more closely resemble real-world applications. In real-world human-machine collaboration, robot operation, or augmented reality-assisted localization scenarios, it is typically necessary to automatically construct and match candidates based on target instances and task semantics in first-person view images. Therefore, the above embodiments employ end-to-end experimental settings to verify the overall operational capability of the embodiments from input acquisition to result output.

[0084] The experimental data uses a subset of matchable collaborative scenes from the validation set of the publicly available dataset Ego-Exo4D. In these scenes, both the first-view and third-view images contain the same target instance, and the third-view image typically contains multiple candidate instances, background interference, or similar targets. This setup effectively verifies the cross-view retrieval and localization capabilities of the above embodiments under conditions of large viewpoint changes, target occlusion, interference from similar instances, and complex backgrounds.

[0085] During the experiment, the input data included a first-view image, a target region or target instance mask in the first-view image, a third-view image, and language instructions related to the target instance. First, the input module received the first-view image, the third-view image, and the language instructions. Then, the target representation module generated a target instance representation. Subsequently, the candidate construction module generated a response heatmap in the third-view image and extracted candidate regions of interest. After instance segmentation or candidate extraction within the candidate regions of interest, the reference establishment module and the relation encoding module further established shared reference relations and generated structural relation representations. Finally, the semantic fusion module and the matching discrimination module output the target localization results in the third-view image.

[0086] To verify the end-to-end effectiveness of the above embodiments, the following representative prior art and the above embodiments were selected for comparison: 1. The GroundingDINO+SAM method combines text-based localization with a general segmentation model, employing open-vocabulary object detection and a general segmentation model for object localization and segmentation. Its basic process involves first locating the target region in a third-view image using language or textual cues, and then generating a target instance mask using a segmentation model. While this method possesses strong open-vocabulary object detection capabilities, it primarily relies on language-driven single-view object localization and segmentation. It lacks identity-level correspondence modeling between target instances in the first-view image and candidate instances in the third-view image, and it also lacks structural relationship discrimination based on shared reference instances.

[0087] 2. The PSALM zero-shot method, which employs a pixel-level multimodal segmentation model (PSALM) with zero-shot processing, belongs to the category of zero-shot methods in visual-language segmentation. It can segment image regions based on language input. This type of method does not require retraining for the target scene and has a certain generalization ability. However, in cross-viewpoint instance retrieval and localization tasks, language cues typically only provide category or attribute constraints, making it difficult to guarantee that the segmented target and the specified instance in the first-viewpoint image are the same physical instance. Therefore, in situations with densely packed similar instances, significant viewpoint differences, or partial target occlusion, this method is prone to identity confusion.

[0088] 3. In the above embodiments, the candidate construction module automatically generates candidate regions of interest and candidate instance sets in the third-view image; simultaneously, the reference establishment module and relation encoding module introduce shared reference instances and spatial organization relationships, and the semantic fusion module jointly modulates linguistic, visual, and structural information. These embodiments not only consider the visual appearance of the target itself but also utilize the relative relationships between the target and surrounding reference instances for identity determination, thus making them more suitable for instance-level retrieval and localization in complex cross-view scenarios.

[0089] The experiment used Top-1, Recall@5, and Loc.E as the main evaluation indicators.

[0090] Top-1 indicates the candidate instance with the highest matching score among the candidate instances. It is used to evaluate whether the highest-scoring candidate instance output by the system is the correct target instance, reflecting the accuracy of the system's direct output results.

[0091] Recall@5 indicates whether the correct target appears in the top five candidate instances with the highest scores, reflecting the stability and recall capability of the system's candidate ranking.

[0092] Loc.E represents the distance error between the predicted target location and the actual target location; a lower value indicates a more accurate localization result. This metric reflects the system's accuracy in locating the target in a third-person view image after identity matching.

[0093] Table 1. Comparison of technical effects between prior art and the above embodiments: As shown in Table 1, the experimental results demonstrate that, under end-to-end settings, the localization error outperforms the two existing methods in all three metrics: Top-1, Recall@5, and Loc.E (localization error). Specifically, compared to the GroundingDINO+SAM method, Loc.E not only generates candidate regions using linguistic and visual information but also further utilizes the structural relationship between the target instance in the first-view image and the candidate instance in the third-view image for identity discrimination, thus reducing false matches caused by similar instances and background interference. Compared to the PSALM zero-shot method, Loc.E does not simply search for the target region in a single third-view image based on linguistic cues but uses the target representation in the first-view image, candidate instances in the third-view image, shared reference instances, and linguistic semantic information together for matching and discrimination, thus exhibiting stronger stability in cross-view identity preservation.

[0094] The preferred embodiments of this application have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of this application without inventive effort. Therefore, any technical solutions that can be obtained by those skilled in the art based on the concept of this application through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A cross-perspective instance representation learning and retrieval localization system, characterized in that, include: The input module acquires first-person view images and obtains target area information; Acquire third-person perspective images; Obtain language instructions, text prompts, or task semantic information related to the target instance, and generate language instruction information; Perform preprocessing; The target representation module receives the first viewpoint image, extracts and encodes features from the target region information to obtain an initial feature representation of the target instance, and performs constrained extraction of the internal features of the target region to obtain a target instance representation. The candidate construction module performs similarity calculation, image feature filtering, and morphological adjustment based on the target instance representation and the third-view image to generate candidate regions of interest. Instance segmentation or candidate region extraction is then performed within the candidate regions of interest to construct a candidate instance set. The reference establishment module receives the first view image and the third view image, respectively detects shared reference instances that co-occur with the target instance, and establishes cross-view reference correspondence. The relation encoding module uses the target instance and candidate instances as the center object and the shared reference instance as the reference object to calculate the edge features between the center object and the reference object. The semantic encoding module receives the language instruction information and generates a semantic representation; The semantic fusion module modulates and fuses the visual and structural information corresponding to the target instance and the candidate instance, based on the semantic representation, to obtain the fused semantic representation of the target instance and the fused semantic representation of the candidate instance, respectively. The matching and discrimination module sorts the candidate instances according to the similarity between the fused semantic representation of the target instance and the fused semantic representation of the candidate instances, and outputs the target localization result in the third-view image; The input module is communicatively connected to the semantic encoding module, the target representation module, and the reference establishment module. The target representation module is communicatively connected to the candidate construction module and the semantic fusion module. The candidate construction module is communicatively connected to the reference establishment module, the relation encoding module, and the semantic fusion module. The reference establishment module is communicatively connected to the relation encoding module. The relation encoding module is communicatively connected to the semantic fusion module. The semantic encoding module is communicatively connected to the semantic fusion module. The semantic fusion module is communicatively connected to the matching discrimination module.

2. The cross-perspective instance representation learning and retrieval localization system as described in claim 1, characterized in that, The input module includes: The first perspective input unit acquires the first perspective image and target region information in the first perspective image; Third-view input unit, acquires the third-view image; The semantic input unit acquires natural language instructions, text prompts, or task semantic information related to the target instance and generates the language instruction information.

3. The cross-perspective instance representation learning and retrieval localization system as described in claim 1, characterized in that, The formula for calculating similarity is as follows: ； in, Position in the third-view image The response value at that location, The feature map of the first-view image. This is a feature map of a third-view image. This indicates the feature location inside the target instance mask of the first viewpoint image. This indicates the feature location in the third-view image. This represents the inner product operation. Represents the L2 norm, This represents the target instance mask in the first viewpoint image.

4. The cross-perspective instance representation learning and retrieval localization system as described in claim 1, characterized in that, The formula for calculating the edge features is as follows: ； in, As edge features, Represents candidate instances The scale parameter, Indicates a stable term. and Represents local basis vectors. For the first The candidate instance and the first The relative distance between shared reference instances Indicates the first The candidate instance and the first The relative distance between shared reference instances is taken as the L2 norm, i.e., the relative Euclidean distance. and These represent the projection components in the two local base directions relative to each other.

5. A cross-perspective instance representation learning and retrieval localization method, using the cross-perspective instance representation learning and retrieval localization system as described in any one of claims 1-4, characterized in that, Includes the following steps: S100. Data acquisition and preprocessing: Acquire a first-view image and obtain target region information from the first-view image; acquire a third-view image; obtain language instructions, text prompts, or task semantic information related to the target instance and generate language instruction information; perform preprocessing on the first-view image, the third-view image, and the language instruction information. S200. Obtain the target instance representation, extract features from the target region information, encode the feature using a visual feature extraction network to obtain the initial feature representation of the target instance, and perform constrained extraction of the internal features of the target region to obtain the target instance representation. S300. Construct a candidate instance set, generate a response heatmap in the third-view image based on the target instance representation, perform image feature filtering and morphological adjustment on the response heatmap to generate candidate regions of interest, perform instance segmentation or candidate region extraction in the candidate regions of interest, and construct a candidate instance set in the third-view image. S400: Construct a structural relationship representation. Receive the first viewpoint image and the third viewpoint image, detect shared reference instances that co-occur with the target instance in the first viewpoint image and the third viewpoint image respectively, establish cross-viewpoint reference correspondence, and construct a structural relationship representation. S500: Obtain the fused semantic representation. Receive the language instruction information, encode it using an encoder, generate a semantic representation, and modulate and fuse it with the visual information and structural information corresponding to the target instance and candidate instance to obtain the fused semantic representation of the target instance and the fused semantic representation of the candidate instance, respectively. S600. Output the target localization result. Based on the similarity between the fused semantic representation of the target instance and the fused semantic representation of the candidate instance, sort the candidate instances and compare them with a preset matching threshold. Output the target localization result in the third-view image.

6. The cross-perspective instance representation learning and retrieval localization method as described in claim 5, characterized in that, S100 includes: S110. Acquire a first-view image and obtain the first-view image and target region information in the first-view image; S120. Acquire a third-view image; S130. Generate language instruction information by acquiring natural language instructions, text prompts, or task semantic information related to the target instance and generating the language instruction information. S140. Preprocessing: Perform preprocessing on the first perspective image, the third perspective image, and the language instruction information to make the first perspective image, the third perspective image, and the language instruction information have a unified data organization method.

7. The cross-perspective instance representation learning and retrieval localization method as described in claim 5 or 6, characterized in that, S200 includes: S210. Determine the target region, extract features from the target region information in the first viewpoint image to obtain the target region, and separate the region range corresponding to the target to be matched from the first viewpoint image. S220, Visual feature extraction: The visual feature extraction network is used to extract visual features from the target region to obtain an initial feature representation of the target instance, and the target region is converted into a feature representation suitable for cross-view matching. S230, Constraint Extraction: Perform constraint extraction on the internal features of the target region to generate a target instance representation.

8. The cross-perspective instance representation learning and retrieval localization method as described in claim 7, characterized in that, The S300 includes: S310. Obtain a dense feature map. Receive the third-view image and obtain a dense feature map of the third-view image through the visual feature extraction network. Map the features of each position in the third-view image to a feature space where similarity can be computed. S320. Generate a response heatmap. Based on the correlation between the target instance representation and the dense feature map, perform similarity calculation to obtain the response value of each position in the third-view image to the target instance in the first-view image, and generate a response heatmap. S330. Filter candidate regions of interest, perform image feature filtering and morphological adjustment on the response heatmap, filter candidate regions of interest, and retain regions with high response values ​​and spatial continuity. S340. Generate candidate instances, input the candidate regions of interest into the target perception network, perform instance segmentation or candidate region extraction, and generate multiple candidate instances in the third-view image. S350. Construct a candidate instance set, extract the attribute information of each candidate instance, and form a candidate instance set in the third-view image from multiple candidate instances.

9. The cross-perspective instance representation learning and retrieval localization method as described in claim 8, characterized in that, The S400 includes: S410. Establish cross-view reference correspondence, detect shared reference instances that co-occur with the target instance in the first view image and the third view image respectively, and establish cross-view reference correspondence based on visual consistency, region correspondence or local response consistency. S420. Generate a structured relation representation. Using the target instance and the candidate instance as the central object and the shared reference instance as the reference object, encode the relative distance, directional relationship and scale relationship between the central object and the reference object to generate a structured relation representation. S430. Calculate edge features: Calculate the edge features between the central object and the reference object, and uniformly represent the relative Euclidean distance between the central object and the reference object and the projection relationship in the local base direction as structured features.

10. The cross-perspective instance representation learning and retrieval localization method as described in claim 9, characterized in that, The S500 includes: S510. Generate semantic representation: Receive the language instruction information, encode it using an encoder, and generate semantic representation; S520. Generate a fused semantic representation of the target instance. Using the semantic representation as a condition, modulate and fuse the visual information and structural information corresponding to the target instance, and generate the fused semantic representation of the target instance through the fused semantic representation calculation formula. S530. Generate a fusion semantic representation of the candidate instance. Using the semantic representation as a condition, modulate and fuse the visual information and structural information corresponding to the candidate instance, and generate the fusion semantic representation of the candidate instance through the fusion semantic representation calculation formula.

Images