A language-guided target positioning method for embodied visual interaction and related device

Abstract

The application discloses a language-guided target positioning method for embodied visual interaction and related equipment, and the original language instruction is enhanced and rewritten through a first visual language model, and complex function description, reference point constraint and spatial relationship are converted into more direct visual expression; a double verification mechanism combining hard rule checking and visual decision model is introduced; the hard rule checking filters obvious invalid results such as no candidate point, format error and boundary out-of-bounds in a low-cost manner, and the visual decision model makes a structured judgment on the candidate point from multiple dimensions such as rewriting fidelity, point number correctness, target internal property and spatial relationship correctness; the reference point is brought into the enhanced instruction generation and visual decision stage, so that the reference point constraint plays a role in the whole positioning process, and the positioning accuracy of the space guidance type task is improved, and the application can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a language-guided target localization method and related equipment for embodied visual interaction. Background Technology

[0002] Embodied visual interaction requires the system to determine the target object or target location from the currently observed image after receiving natural language instructions, and to further pass the result to the click, grab, instruction, or action planning modules. Unlike traditional category detection, this type of task often includes information such as spatial relationships, reference point constraints, functional semantics, multi-object sets, and abstract reasoning. Therefore, the final output must not only "find the right object" but also "provide an executable pointing location".

[0003] For these types of tasks, relying solely on bounding boxes or region-level results is usually insufficient. Downstream interaction modules are truly concerned with whether the pointing point falls within the target object, whether it satisfies the reference relationship, whether it faithfully adheres to the original language instruction, and whether the point count is correct in multi-target scenarios. As the capabilities of visual language models improve, these language-guided pointing localization tasks are becoming increasingly common, and are beginning to shift from general visual understanding scenarios to embodied execution scenarios.

[0004] Existing research generally follows three routes. The first route involves open vocabulary detection, phrase localization, or pointer arrival localization, and then converting bounding boxes, region boxes, or segmentation masks into representative points. This type of method is more effective in scenarios where the target category is clear and the spatial relationships are relatively simple, but the core output is usually still a region-level result, rather than a pixel-level pointer that can be directly executed.

[0005] The second approach directly utilizes a large multimodal model to generate coordinates or pointing points. This type of method inputs the image and natural language instructions into a visual language model, which then performs semantic understanding, spatial relationship inference, and point generation in one step. Compared to region-level methods, this approach is closer to the output format required for embodied interaction, but it still faces significant challenges in complex language disambiguation, reference point handling, number constraints, and failure recovery.

[0006] The third category can be summarized as image localization and relocalization schemes for complex text descriptions. This includes both region-based localization solutions and re-search or relocalization schemes with failure recovery mechanisms. This type of technology indicates that existing systems have begun to focus on target localization capabilities under complex description conditions, as well as recovery mechanisms after initial localization failures. However, judging from publicly available solutions, most methods still revolve around target bounding boxes, target regions, or temporal tracking results, and are still some distance from the final point selection output truly needed in embodied interaction scenarios. Summary of the Invention

[0007] The main objective of this invention is to provide a method, device, electronic device, storage medium, and program product for language-guided target localization in embodied visual interaction, aiming to solve at least one problem of the prior art.

[0008] To achieve the above objectives, one aspect of this invention proposes a language-guided target localization method for embodied visual interaction, the method comprising: Acquire task data for the target task; the task data includes embodied scene observation images, raw language commands, and reference point information associated with the embodied scene observation images; Based on the original language instructions, embodied scene observation images, and reference point information, enhanced pointing instructions are generated through a first visual language model. Based on enhanced pointing instructions and embodied scene observation images, candidate pointing points are generated through a candidate point generation model; Hard rule checks are performed on candidate pointing points to obtain the check results; among them, hard rule checks include checks for no candidate points, format validity checks, and image boundary checks. If the check result is a failure, the candidate pointer is updated through relocation processing, and the process of performing hard rule checks on the candidate pointer is returned to obtain the check result, until the check result is a success. In response to a pass result, a visual decision is made based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points, using a visual decision model to obtain a structured decision result. The final pointer is generated based on the candidate pointers and the structured decision results.

[0009] In some optional approaches, the task data also includes task category information. Based on the original language instructions, embodied scene observation images, and reference point information, enhanced pointing instructions are generated through a first visual language model, including the following steps: Based on the original language instructions, combined with preset rewriting prompts and task category information, the first rewriting input content is constructed; Determine if the reference point information is empty; If the reference point information is empty, the first rewritten input content and the embodied scene observation image are input into the first visual language model so that the first visual language model can semantically rewrite the original language instructions and generate enhanced pointing instructions. If the reference point information is not empty, the reference point information is converted into pixel-level marker information as reference point markers in the embodied scene observation image; wherein, the reference point markers are visualized on the embodied scene observation image with a predetermined color and shape; Add preset supplementary content to the first rewritten input content to construct the second rewritten input content; The second rewritten input content and the embodied scene observation image with reference point markers are input into the first visual language model so that the first visual language model can semantically rewrite the original language instructions and generate enhanced directional instructions.

[0010] Among some alternative approaches, candidate pointers are generated using a candidate point generation model based on enhanced pointing instructions and embodied scene observation images, including the following steps: Obtain the task type of the target task; where the task type includes multi-point output task or single-point output task; the target number required by the target task is configured in the original language instructions. If the target number is unique, the task type is determined to be a single-point output task. If the target number is not unique, the task type is determined to be a multi-point output task. If the target task is a single-point output task, construct the first constraint condition based on the preset task output constraints; If the target task is a multi-point output task, the task output constraints are combined with the output configuration that allows multi-point output to construct the first condition constraint; The candidate point generation instruction is obtained by organizing the enhanced pointing instruction and the first constraint condition. Input the candidate point generation instruction and the embodied scene observation image into the candidate point generation model so that the candidate point generation model outputs the coordinates of at least one candidate point; The candidate point coordinates are converted into pixel-level coordinates corresponding to the embodied scene observation image to obtain the candidate pointing points.

[0011] Among some optional approaches, based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points, a visual decision-making model is used to perform visual decision-making to obtain a structured decision result, including the following steps: A first visual marker is marked on the embodied scene observation image based on candidate pointing points, and a second visual marker is marked on the embodied scene observation image based on reference point information to generate a decision visualization image; The first visual marker and the second visual marker use different colors and shapes; The original language instructions, enhanced pointing instructions, and adjudication visualization images are input into the visual adjudication model so that the visual adjudication model can perform visual adjudication and generate structured adjudication results. The structured adjudication results include rewrite fidelity markers, point correctness markers, target internality markers, spatial relationship correctness markers, confidence markers, final acceptance markers, and adjudication reasons. If the rewritten fidelity marker, point correctness marker, target internality marker, spatial relationship correctness marker, and final acceptance marker are all true and the confidence marker meets the preset conditions, the structured decision result is determined to be that the candidate pointing point passes the visual decision; otherwise, the structured decision result is determined to be that the candidate pointing point fails the visual decision.

[0012] In some alternative methods, the final pointer is generated based on the candidate pointers and the structured decision results, including the following steps: If the structured decision result indicates that the candidate pointing point passes the visual decision, the candidate pointing point is taken as the final pointing point; If the structured adjudication result indicates that the candidate pointing point fails the visual adjudication, a fallback pointing point is generated based on the original language instruction, the enhanced pointing instruction, and the embodied scene observation image, using a fallback visual language model. The catch-all point is re-adjudicated to obtain the re-adjudication result; If the re-decision is passed, the fallback point will be used as the final point; otherwise, the candidate point will be used as the final point.

[0013] Among some alternative approaches, a fallback pointing point is generated using a fallback visual language model based on the original language instructions, enhanced pointing instructions, and embodied scene observation images, including the following steps: Obtain the task type of the target task; where the task type includes multi-point output task or single-point output task; the target number required by the target task is configured in the original language instructions. If the target number is unique, the task type is determined to be a single-point output task. If the target number is not unique, the task type is determined to be a multi-point output task. If the target task is a single-point output task, construct a second constraint condition based on the preset task output constraints; If the target task is a multi-point output task, the task output constraints are combined with the output configuration that allows multi-point output to construct a second condition constraint; The original language instructions are used as the source of truth semantics, the enhanced pointing instructions are used as auxiliary context, and a fallback relocation prompt is constructed by combining the indication information of candidate pointing points that fail the visual decision and the second constraint. The fallback relocation cue and the embodied scene observation image are input into the fallback visual language model so that the fallback visual language model outputs structured coordinate results; wherein, the structured coordinate results include the coordinate data of at least one fallback pointing point; The structured coordinate results are converted into pixel-level coordinates corresponding to the embodied scene observation image to obtain the bottom-point pointing. Among them, the catch-all visual language model adopts a different model from the candidate point generation model.

[0014] In some alternative approaches, a re-adjudication of the catch-all point is performed to obtain a re-adjudication result, including the following steps: Perform hard rule checks on the catch-all pointers; Once the catch-all point passes the hard rule check, a third visual marker is marked on the embodied scene observation image based on the catch-all point, and a fourth visual marker is marked on the embodied scene observation image based on the reference point information, generating a catch-all decision visualization image; wherein, the third visual marker and the fourth visual marker use different colors and shapes; The original language instructions, enhanced pointing instructions, and fallback adjudication visualization images are input into the visual adjudication model so that the visual adjudication model can perform visual adjudication and generate re-adjudication results. The re-adjudication results include rewriting the fidelity marker, the point correctness marker, the target internality marker, the spatial relationship correctness marker, the confidence level marker, the final acceptance marker, and the reasons for the adjudication. If the rewritten fidelity marker, point accuracy marker, target internality marker, spatial relationship accuracy marker, and final acceptance marker are all true and the confidence marker meets the preset conditions, the re-adjudication result is determined to be passed; otherwise, the re-adjudication result is determined to be failed.

[0015] To achieve the above objectives, another aspect of the present invention provides a language-guided target localization device for embodied visual interaction, the device comprising: The first module is used to acquire task data for the target task; the task data includes embodied scene observation images, original language commands, and reference point information associated with the embodied scene observation images; The second module is used to generate enhanced pointing instructions based on the original language instructions, embodied scene observation images, and reference point information through the first visual language model. The third module is used to generate candidate pointing points based on enhanced pointing instructions and embodied scene observation images through a candidate point generation model. The fourth module is used to perform hard rule checks on candidate pointing points and obtain the check results; among which, hard rule checks include no candidate point checks, format validity checks, and image boundary checks. The fifth module is used to update the candidate pointer point through relocation processing if the check result is not passed, and return to execute the operation of the fourth module until the check result is passed. The sixth module is used to respond to a pass result by performing a visual decision based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points, and obtaining a structured decision result through a visual decision model. The seventh module is used to generate the final pointer based on the candidate pointers and the structured decision results.

[0016] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0017] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0018] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0019] The present invention provides at least the following beneficial effects: It offers a method, apparatus, electronic device, storage medium, and program product for language-guided target localization in embodied visual interaction. This solution acquires task data for a target task; the task data includes embodied scene observation images, original language instructions, and reference point information associated with the embodied scene observation images; based on the original language instructions, the embodied scene observation images, and the reference point information, an enhanced pointing instruction is generated using a first visual language model; based on the enhanced pointing instruction and the embodied scene observation images, candidate pointing points are generated using a candidate point generation model; hard rule checks are performed on the candidate pointing points to obtain check results; the hard rule checks include checks for no candidate points, format validity checks, and image boundary checks; if the check result is a failure, the candidate pointing points are updated through relocation processing, and the process returns to performing hard rule checks on the candidate pointing points to obtain check results, until the check result is a success; in response to a success result, a visual decision is made using a visual decision model based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points to obtain a structured decision result; and a final pointing point is generated based on the candidate pointing points and the structured decision result. This invention directly outputs pixel-level pointing points, avoiding the problem of the landing point not being inside the target. The output pointing points can be directly used by downstream interactive modules such as clicking and grabbing. It enhances and rewrites the original language instructions through a first visual language model, transforming complex functional descriptions, reference point constraints, and spatial relationships into more direct visual expressions. This solves the problem of semantic drift and reference point misjudgment that easily occur in the single inference of existing direct point output routes. A dual verification mechanism combining hard rule checking and a visual adjudication model is introduced: hard rule checking filters obviously invalid results such as no candidate points, format errors, and boundary violations in a low-cost manner; the visual adjudication model performs structured judgment on candidate points from multiple dimensions such as rewriting fidelity, point number correctness, target internality, and spatial relationship correctness. Reference point information is explicitly acquired and processed, and the reference points are brought into the enhanced instruction generation and visual adjudication stages, allowing reference point constraints to play a role in the entire positioning process, improving the positioning accuracy of spatial guidance tasks. This invention, through a closed-loop process of "enhancement—generation—verification—output," can effectively improve the semantic fidelity and positioning reliability of language-guided pointing positioning. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an implementation environment for a language-guided target localization method for embodied visual interaction provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the language-guided target localization method for embodied visual interaction provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the overall framework of the language-guided target localization method for embodied visual interaction provided in this embodiment of the invention. Figure 4 This is a schematic diagram of the overall technical process of the language-guided target localization method for embodied visual interaction provided in this embodiment of the invention; Figure 5 This is a schematic diagram of the module architecture for implementing a language-guided target localization method for embodied visual interaction provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the language-guided target localization device for embodied visual interaction provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0022] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0023] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0024] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0025] In order to truly serve embodied visual interaction, the related technologies need to be more closely linked together to form a stable closed loop, including language enhancement, candidate products, structured decision-making, and failure correction.

[0026] In view of this, this embodiment of the invention provides a language-guided target localization method and related device for embodied visual interaction. This method acquires task data for the target task; wherein the task data includes embodied scene observation images, original language instructions, and reference point information associated with the embodied scene observation images; based on the original language instructions, embodied scene observation images, and reference point information, an enhanced pointing instruction is generated using a first visual language model; based on the enhanced pointing instruction and embodied scene observation images, candidate pointing points are generated using a candidate point generation model; hard rule checks are performed on the candidate pointing points to obtain check results; wherein the hard rule checks include checks for no candidate points, format validity checks, and image boundary checks; if the check result is a failed result, the candidate pointing points are updated through relocation processing, and the process returns to performing hard rule checks on the candidate pointing points to obtain check results, until the check result is a passed result; in response to a passed check result, a visual adjudication is performed using a visual adjudication model based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points to obtain a structured adjudication result; and a final pointing point is generated based on the candidate pointing points and the structured adjudication result. This invention directly outputs pixel-level pointing points, avoiding the problem of the landing point not being inside the target. The output pointing points can be directly used by downstream interactive modules such as clicking and grabbing. It enhances and rewrites the original language instructions through a first visual language model, transforming complex functional descriptions, reference point constraints, and spatial relationships into more direct visual expressions. This solves the problem of semantic drift and reference point misjudgment that easily occur in the single inference of existing direct point output routes. A dual verification mechanism combining hard rule checking and a visual adjudication model is introduced: hard rule checking filters obviously invalid results such as no candidate points, format errors, and boundary violations in a low-cost manner; the visual adjudication model performs structured judgment on candidate points from multiple dimensions such as rewriting fidelity, point number correctness, target internality, and spatial relationship correctness. Reference point information is explicitly acquired and processed, and the reference points are brought into the enhanced instruction generation and visual adjudication stages, allowing reference point constraints to play a role in the entire positioning process, improving the positioning accuracy of spatial guidance tasks. This invention, through a closed-loop process of "enhancement—generation—verification—output," can effectively improve the semantic fidelity and positioning reliability of language-guided pointing positioning.

[0027] It is understood that the language-guided target localization method for embodied visual interaction provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0028] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0029] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0030] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0031] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0032] For example, based on Figure 1The implementation environment shown in this embodiment of the invention provides a language guidance target localization method for embodied visual interaction. The following description uses the application of this language guidance target localization method for embodied visual interaction in server 101 as an example. It can be understood that this language guidance target localization method for embodied visual interaction can also be applied in terminal 102.

[0033] Reference Figure 2 , Figure 2 This is an optional flowchart of a language-guided target localization method for embodied visual interaction provided in an embodiment of the present invention. The executing entity of the language-guided target localization method for embodied visual interaction can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S1 to S7.

[0034] Step S1: Obtain the task data for the target task; The task data includes embodied scene observation images, original language instructions, and reference point information associated with the embodied scene observation images; For example, in some specific implementations, the system first acquires an embodied scene observation image, the original language instructions, and optional reference point information associated with the image. The reference point information may originate from an upstream system, historical interaction context, or labeled metadata. The reference point information can be empty; when empty, the system processes the task as a non-reference point task.

[0035] Step S2: Based on the original language instructions, the embodied scene observation image, and the reference point information, an enhanced pointing instruction is generated through the first visual language model; It should be noted that the task data also includes task category information. In some embodiments, step S2 may include the following steps: constructing a first rewritten input content based on the original language instruction, combined with preset rewriting prompts and task category information; determining whether the reference point information is empty; if the reference point information is empty, inputting the first rewritten input content and the embodied scene observation image into a first visual language model, so that the first visual language model semantically rewrites the original language instruction and generates an enhanced pointing instruction; if the reference point information is not empty, converting the reference point information into pixel-level marker information as reference point markers in the embodied scene observation image; wherein, the reference point markers are visualized on the embodied scene observation image with predetermined colors and shapes; adding preset supplementary content to the first rewritten input content to construct a second rewritten input content; inputting the second rewritten input content and the embodied scene observation image with reference point markers into the first visual language model, so that the first visual language model semantically rewrites the original language instruction and generates an enhanced pointing instruction.

[0036] For example, in some specific implementations, generating enhanced pointing instructions can be specifically implemented as follows: This step mainly involves rewriting the original language instructions to be more direct, so that the subsequent candidate point generation model can be easier to understand.

[0037] Specifically, the system feeds the original language instructions, task category information, preset rewrite prompts, and scene images into the first visual language model. This model, combined with the image content, rewrites the original language instructions and outputs an enhanced directional instruction. The output here is not a complex structured result, but a text instruction that can be directly used for subsequent localization.

[0038] If the current task includes reference points, the system will first mark the reference points on the image, and then send the marked image to the first visual language model. The purpose of this is to make the model clearer about which position on the image the "current point" and "reference point" mentioned in the original instruction correspond to, thereby further determining what the real target is.

[0039] The rewritten enhanced pointing instructions are generally shorter, more direct, and closer to the actual target content visible in the image than the original instructions. For example, if the original instructions contain reference points, movement directions, function descriptions, or convoluted language, the rewritten instructions will try to directly state the final target object, target area, or target set.

[0040] However, this rewriting step should not change the original meaning. The actual target, quantity requirements, spatial relationships, and reference constraints in the original language instructions should, in principle, be preserved. They should not become a different target after rewriting, nor should multiple targets be changed into a single target.

[0041] Specifically, generating enhanced pointing instructions can be achieved through the following steps: 1. First, determine if reference point information exists in the current task; 2. If reference point information exists, the reference points are converted into marker information in the image, and an image with reference point markers is generated; 3. Construct rewritten input content based on preset rewriting prompts, and input the original language instructions, task category information, and scene image or image with reference point markers into the first visual language model; 4. The original language instructions are rewritten by combining the first visual language model with the image content; 5. Output enhanced pointing instructions for use in subsequent candidate point generation steps.

[0042] The first visual language model does not have any requirements regarding model architecture or training. This model is a pre-trained visual language model, such as a visual language model that is already open-source and in use.

[0043] Examples of specific application scenarios are as follows: Original command: Points to an object that can be used for mixing and stirring.

[0044] The first visual model receives the original instruction, the image, and the rewritten prompt, and then the model outputs the enhanced instruction.

[0045] The rewritten enhanced instruction: points to the spoon.

[0046] The default overwrite prompts are as follows: Example of a system prompt for the model: "You are an expert in rewriting instructions for vision-oriented localization and vision-oriented object detection and segmentation models (such as the GroundingDINO and SAM series models)."

[0047] You will receive an image and a raw user instruction.

[0048] Your task is to understand the target object, object component, group of objects, or image region that the original instruction refers to in the image, and then rewrite it into a shorter, more explicit, and more visually-based instruction.

[0049] Objectives: Preserve the original intent; utilize image parsing to understand the content indicated by the instructions; rewrite the instructions into visually concrete language; prioritize the use of explicit object names, attributes, locations, regions, and relationships; remove unnecessary conversational language; convert abstract functional descriptions into possible visible object names whenever possible; make the output more easily understood by detection, localization, and segmentation models.

[0050] Rules: Output only one rewritten instruction; use concise language; keep it to one sentence if possible; always begin with "pointing to..."; do not explain your reasoning process; do not mention hidden category labels; do not fabricate invisible facts; use images as primary evidence to determine the final goal; prioritize visible nouns over descriptions of human intent; retain plural forms when the original instruction refers to multiple objects; retain spatial relationships such as left, right, up, down, nearest, farthest, middle, front, and back; if the original instruction describes movement, rewrite it as pointing to the final target object or region if possible; the rewritten instruction must already contain the final, simple goal and should not require another round of reasoning or navigation; do not retain intermediate navigation language such as "current location," "reference point," "red dot," "move," "move towards." "Arrival" or "Until" is used, unless the target itself is that point or area; if the original instruction is anchored to a reference point, use image analysis to determine the final target and then directly name that final target; if the original instruction describes a function or availability, rewrite it as the most likely category of visible object; if the original instruction asks for evidence of identity, such as a brand, country, or name, rewrite it as visible evidence, such as signs, text, placards, flags, or license plates; if the original instruction is about the direction of movement, use visually indicating moving objects, arrows, lane markings, wheel direction, or frontal orientation as the target; if the original instruction refers to an open space or area, use nearby visible anchor points to describe the area; if the original instruction is vague but the image makes the target clear, rewrite it as a visually clear target; if there are multiple possible interpretations, choose the one that the image best supports.

[0051] Rewrite heuristic rules: Functional description → specific object name. For example, "a tool people use to drink soup" → "points to a spoon." "something used to open a door" → "points to a doorknob" or "points to a key." "a tool people use to write" → "points to a pen" or "points to a pencil." Move command → final target location. For example, "Move down until the search button" → "Point to the search button." "Button to the right of the current location" → If the image shows the final target is the submit button, then write "Point to the submit button." Counting instructions → explicit targets of multiple numbers. For example, "count all the monkeys in the image" → "point to all the monkeys". Identity / Evidence Instruction → Visible Evidence. For example, "The part of the train that indicates which country it may come from" → "Point to the flag or country markings on the train." Standardization preferences: Prioritize the use of standard visual labels, such as: cars, buses, bicycles, chairs, tables, windows, door handles, spoons, forks, knives, pens, pencils, buttons, search bars, signs, logos, license plates, arrows, wheels, seats, mirrors, and handles.

[0052] Choose either the singular or plural form to match the original meaning.

[0053] Use the final resolved object name instead of any reference point terminology.

[0054] Avoid mentioning reference points unless the target itself is the reference point. If reference point information is available, the following additional tips (i.e., supplementary content) can be added later: "If the task category is 'guided,' a large red reference point may have been drawn on the image."

[0055] Treat the large red dot as the existing reference point mentioned in the instruction.

[0056] The expected final target is inferred by combining the red reference point and the image content.

[0057] The rewritten instructions must name the actual target in the image, not the reference point itself.

[0058] Unless the original instruction explicitly requires pointing to the point itself, do not retain phrases such as "current location," "reference point," or "red dot" in the final rewritten instruction. The prompt sent by the user to the model is as follows: Rewrite the following original instruction into a visually-based instruction.

[0059] Original instructions: {Original command (e.g., pointing to the object used for mixing).} Only return the rewritten instructions. Step S3: Based on the enhanced pointing instruction and the embodied scene observation image, candidate pointing points are generated through the candidate point generation model; It should be noted that in some embodiments, step S3 may include the following steps: obtaining the task type of the target task; wherein, the task type includes a multi-point output task or a single-point output task; the original language instruction is configured with the number of targets required by the target task. If the number of targets is unique, the task type is determined to be a single-point output task; if the number of targets is not unique, the task type is determined to be a multi-point output task; if the target task is a single-point output task, a first constraint condition is constructed based on the preset task output constraint; if the target task is a multi-point output task, the task output constraint is combined with the output configuration that allows multi-point output to construct a first condition constraint; the candidate point generation instruction is obtained based on the enhanced pointing instruction and the first constraint condition; the candidate point generation instruction and the embodied scene observation image are input into the candidate point generation model so that the candidate point generation model outputs at least one candidate point coordinate; the candidate point coordinates are converted into pixel-level coordinates corresponding to the embodied scene observation image to obtain the candidate pointing point.

[0060] For example, in some specific implementations, generating candidate pointing points using a candidate point generation model can be achieved as follows: Candidate pointing points are generated using a candidate point generation model based on the enhanced pointing instructions and the embodied scene observation image.

[0061] The candidate point generation model can adopt a visual language model, and its deployment method can be local deployment, server deployment, or network call. This invention does not limit this method.

[0062] For single-point output tasks, a single-point constraint can be added so that the candidate point generation model outputs only one candidate point when the original language instruction does not explicitly require multiple targets.

[0063] In some specific application scenarios, step S3 can be implemented as follows: 1. Construct candidate point generation instructions; 2. Input the candidate point generation instructions and the embodied scene observation image into the candidate point generation model; 3. Obtain the model output results; 4. Obtain the coordinates of the candidate points from the model output; 5. Convert the analytical coordinates to pixel-level coordinates.

[0064] Specifically, "single-point / multi-point constraints" are controlled at the input end. Based on predefined rules (such as the presence of words like "all" in the original instruction), and combining the original and enhanced instructions, it is determined whether the instruction is a single-point or multi-point constraint. If it is a multi-point constraint, the instruction is classified as a multi-point output; otherwise, it is a single-point output. Different prompts are applied depending on whether it is a multi-point output. In this step, there is no system prompt from the model.

[0065] Multi-point output prompt (user input): Point to: {enhanced instructions (e.g., point to all spoons)}; Single-point output prompt (user input): Pointing to: {Enhanced instructions (e.g., pointing to a spoon)} Additional hard constraints for single-point output tasks: Unless the instruction explicitly requires a count or multiple target instances, only one point is returned.

[0066] If the instructions and images do not explicitly require multiple objects, only one point will be output.

[0067] When the choice is uncertain between one point and multiple points, select one point.

[0068] Multiple points are returned only if there is an explicit count or an explicit request for multiple instances and the image supports it.

[0069] Output examples: Multi-point output: {(344,566),(356,654),...}; Single-point output: {(244,236)}.

[0070] Specifically, for multi-point output tasks, it is not required that the number of points generated by the candidate point generation model be exactly the same as the actual number, since in reality, it is impossible to know how many there are.

[0071] The prompt will not display any quantifiers. The candidate point generation model is only responsible for generating point coordinates and does not make any decisions. The subsequent visual decision-making model will not know the actual number of points; it will only judge whether these points are complete and correct based on its own capabilities. If they are incorrect, a fallback mechanism will be triggered. No post-processing additions or deletions will be performed.

[0072] Step S4: Perform hard rule checks on the candidate pointing points to obtain the check results; Hard rule checks include no candidate point checks, format validity checks, and image boundary checks. For example, in some specific implementations, hard rule checks are performed on candidate pointers. Hard rule checks include: 1. Checking for no candidate points; 2. Checking the validity of candidate point formats; 3. Checking image boundaries.

[0073] When a candidate point is empty, has an invalid candidate point format, or falls outside the image boundary, it can be directly determined that the candidate point does not meet the preset decision conditions and enter the failure processing branch.

[0074] Step S5: If the check result is a failure, update the candidate pointers through relocation processing, return to the step of performing hard rule checks on the candidate pointers to obtain the check result, until the check result is a success. Step S6: In response to the inspection result being passed, a visual decision is made based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points, using a visual decision model to obtain a structured decision result. It should be noted that in some embodiments, the structured decision result obtained by visual decision-making through a visual decision-making model based on the original language instruction, enhanced pointing instruction, reference point information, and candidate pointing points may include the following steps: marking a first visual marker on the embodied scene observation image based on the candidate pointing points, marking a second visual marker on the embodied scene observation image based on the reference point information, and generating a decision visualization image; wherein the first visual marker and the second visual marker use different colors and shapes; inputting the original language instruction, enhanced pointing instruction, and decision visualization image into the visual decision-making model so that the visual decision-making model performs visual decision-making and generates a structured decision result; wherein the structured decision result includes a rewrite fidelity marker, a point count correctness marker, a target internality marker, a spatial relationship correctness marker, a confidence marker, a final acceptance marker, and a decision reason; when the rewrite fidelity marker, the point count correctness marker, the target internality marker, the spatial relationship correctness marker, and the final acceptance marker are all true and the confidence marker meets a preset condition, the structured decision result is determined to be that the candidate pointing point passes the visual decision; otherwise, the structured decision result is determined to be that the candidate pointing point fails the visual decision.

[0075] For example, in some specific implementations, performing visual decision-making on candidate points can be achieved as follows: When a candidate point passes the hard rule check, a decision visualization image is generated based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points, and the visual decision is executed through the visual decision model.

[0076] One approach is to: 1. Represent candidate pointing points with a first visual marker; 2. Represent reference points with a second visual marker; 3. Represent reference points and candidate points with distinguishable colors or shapes.

[0077] The visual adjudication model can output structured adjudication results, including at least: 1. rewrite fidelity; 2. point count correctness; 3. target internality; 4. spatial relationship correctness; 5. confidence score; 6. final acceptance label; 7. adjudication reason; 8. structured verification trajectory.

[0078] In some specific application scenarios, the adjudication process is roughly as follows: given a prompt and an image, it is handed over to a visual adjudication model (such as commercial visual language models like GPT, or other open-source visual language models) to complete the process, outputting structured data, i.e., JSON Schema. External tools such as bounding boxes are not used. Essentially, the model replaces human review. Constraints like "objective internality" are required in the prompt to check whether the current result meets these constraints. The effectiveness of this adjudication depends on the capabilities of the model itself.

[0079] Here is an example of a Prompt: "You are a strict and conservative verifier of the pointing point results."

[0080] Your goal is to maximize accuracy: accept only results that are clearly correct at the red dot. Reject any results with any uncertainty, ambiguity, missing targets, redundant targets, boundary issues, or rewrite mismatches. It is better to reject a correct result than to accept a wrong one.

[0081] You will receive a visualization: This is the original image without a segmentation mask overlay; red dots are candidate point results; blue dots (if present) are reference / context points; blue reference points are not prediction targets unless the text instructions explicitly require the same location.

[0082] Explicitly use text input in reasoning: treat the original user input as a source of truth. Use augmented text only as supplementary context, ignoring it if it conflicts with the original user input. Don't jump directly to the conclusion from the red dot. First infer the expected target from the text and the complete image, then compare the red dot with the expected target.

[0083] Execute and return the following precise "System 2" verification trajectory, consisting of 5 steps: Step 1. Global observation: Inspect the entire scene, identify the main objects, and locate all possible targets and anchor points.

[0084] Step 2. Instruction Deconstruction: First, parse the original user input, then parse the enhanced text to extract precise target semantics, quantitative constraints, and spatial relationships.

[0085] Step 3. Derivation of Expected Target: Before judging any red dots, determine which object(s) should be selected in this image and verify whether the blue dots are anchor points or targets in this query.

[0086] Step 4. Candidate point analysis: Examine each red point one by one, indicating on what exact object / subject area it falls on, whether it is clearly inside the target boundary, and whether it meets the required relationship with the anchor point or the referenced object.

[0087] Step 5. Final Match Verification: Compare the complete set of red dots with the expected target set obtained in Steps 1-3, and then determine whether the candidate set is strictly correct.

[0088] Strict acceptance criteria: 1. Candidate points must satisfy the original user input, not just a seemingly reasonable paraphrase. 2. Infer from the original user input whether the task requires a single or multiple targets. 3. If the instruction requires a specific target, there must be one and only one red point. 4. If the instruction requires "all / every / two / more" targets, all clearly visible target instances must be pointed to. 5. The center of each red marker must be clearly inside the visible target object / subject. 6. Reject if a red point is only near the target, on a boundary / background / shadow, or on an adjacent object. 7. For spatial or guided tasks, verify the exact relationship with the blue reference point or the referenced object. 8. Reject if any red point falls on a seemingly reasonable but not derived final target from the text and scene. 9. Reject if the confidence level is low.

[0089] Rules: Return only JSON format. Do not generate new points. Return the complete 5-step validation trace in JSON. Do not leave any validation steps blank. Keep each step concise, specific, and rooted in actual images and text. Only set "accept" to true if all rigorous checks pass.

[0090] Original user input: {Original command: points to the object used for stirring and mixing} Enhanced text input (enhanced_query): {enhanced instruction: point to spoon} The exact JSON structure returned is as follows: json { "verification_trace": { "step_1_global_observation": "Concise scene-level observation", "step_2_instruction_deconstruction": "A concise explanation of the text instructions", "step_3_expected_target_derivation": "Which target should be selected before inspecting the red dot?" "step_4_candidate_point_analysis": "Where each red dot falls and whether it is valid", "step_5_final_match_verification": "Final comparison between the expected target set and candidate points" }, "rewrite_faithful": true or false, "point_count_correct": true or false "all_points_clearly_inside_target": true or false, "spatial_relation_correct": true or false, "high_confidence": true or false, "accept": true or false "reason": a brief explanation }” The input "decision visualization image" visualizes the coordinates of these points on an image. The labeling methods (color and shape) for candidate and reference points have not been validated through ablation experiments, and the optimal visual cues are unknown. In the current method, candidate points are represented by a red outer ring with an inner cross, while reference points are represented by a large blue dot. Candidate points are red, and reference points are blue for differentiation.

[0091] High confidence level: there's only one true or false value here. In reality, whether a decision passes depends on whether the values ​​of multiple fields in the output are true. Specifically, it depends on whether the values ​​of the following fields in the JSON are true.

[0092] "accept", ultimately accepting the flag. "rewrite_faithful", rewrites faithfulness. "point_count_correct", point count correctness "all_points_clearly_inside_target", target internality "spatial_relation_correct", spatial relation correctness "high confidence" If some of the five conditions are True and some are False (e.g., "objective internality" is True but "spatial relationship" is False), the adjudication model will not accept this result and will enter the fallback stage.

[0093] Step S7: Generate the final pointer based on the candidate pointers and the structured decision results; It should be noted that in some embodiments, step S6 may include the following steps: if the structured decision result indicates that the candidate pointing point passes the visual decision, the candidate pointing point is used as the final pointing point; if the structured decision result indicates that the candidate pointing point fails the visual decision, a fallback pointing point is generated based on the original language instruction, the enhanced pointing instruction, and the embodied scene observation image through a fallback visual language model; the fallback pointing point is re-decided to obtain a re-deciding result; if the re-deciding result is passed, the fallback pointing point is used as the final pointing point; otherwise, the candidate pointing point is used as the final pointing point.

[0094] For example, in some specific implementations, the final pointing point is determined based on the visual decision result of the candidate pointing point, the re-decision result of the fallback pointing point, the candidate pointing point itself, and the fallback pointing point itself.

[0095] The final point can be determined according to the following rules: 1. When a candidate pointing point passes visual decision, the candidate pointing point is determined as the final pointing point; 2. When a candidate pointing point fails the visual decision and the fallback pointing point passes the re-decision, the fallback pointing point is determined as the final pointing point; 3. When a candidate pointing point fails the visual decision and the fallback pointing point also fails the re-decision, the candidate pointing point is retained as the final pointing point; 4. When both the candidate pointer and the fallback pointer are unavailable, output an empty result.

[0096] It should be noted that in some embodiments, generating a fallback pointing point through a fallback visual language model based on the original language instructions, enhanced pointing instructions, and embodied scene observation images may include the following steps: obtaining the task type of the target task; wherein, the task type includes a multi-point output task or a single-point output task; the original language instructions are configured with the number of targets required by the target task; if the number of targets is unique, the task type is determined to be a single-point output task; if the number of targets is not unique, the task type is determined to be a multi-point output task; if the target task is a single-point output task, construct a second constraint condition based on a preset task output constraint; if the target task is a multi-point output task, combine the task output constraint with the allowable multiple The output configuration of the point output constructs a second conditional constraint; the original language instruction is used as the source of truth semantics, the enhanced pointing instruction is used as auxiliary context, and the indication information of the candidate pointing point failing the visual decision and the second constraint condition are combined to construct a fallback relocation prompt; the fallback relocation prompt and the embodied scene observation image are input into the fallback visual language model so that the fallback visual language model outputs a structured coordinate result; wherein, the structured coordinate result includes the coordinate data of at least one fallback pointing point; the structured coordinate result is converted into pixel-level coordinates corresponding to the embodied scene observation image to obtain the fallback pointing point; wherein, the fallback visual language model adopts a different model than the candidate point generation model.

[0097] For example, in some specific implementations, when the visual decision fails, a fallback relocation prompt is constructed based on the original language instruction, the enhanced pointing instruction, and the embodied scene observation image, and the fallback pointing point is regenerated through the fallback visual language model.

[0098] One approach is to have the fallback relocation prompt meet the following requirements: 1. Primitive language instructions serve as the source of truth-value semantics; 2. Enhanced pointer instructions are only used as auxiliary contexts; 3. If the task requires a single point, then only output one point; 4. If the task explicitly requires multiple objectives, then output multiple points; 5. The output uses a structured coordinate format.

[0099] The fallback relocalization model and the initial candidate point generation model are not the same model; they are other visual language models with localization capabilities. For example, if the initial candidate model uses Qwen3-VL-4B, then Qwen3-VL-8B can be used here. However, these are all models from the same series, and other stronger models can be used instead, such as Google's Gemini 3.1pro model.

[0100] Furthermore, the fallback relocation prompt differs from the initial prompt. It informs the model that previous candidate points failed the visual decision. A specific example of the prompt is as follows: "You are a visual pointing positioning model."

[0101] You will receive the original image and must produce a corrected location answer.

[0102] The previous candidate points have been rejected by a strict validator.

[0103] Use raw user input as the source of truth.

[0104] The enhanced text is used only as supplementary context; if it conflicts with the original input, it is ignored.

[0105] Do not repeat the same mistakes.

[0106] Rules: Return only JSON format. Infer whether a single target or multiple targets are needed from the original user input. If a single target is needed, return an exact point. If multiple targets are explicitly needed, return a point for each clearly visible target instance. Use Gemini coordinate format: [y, x], normalized to 0-1000. Prefer this structure: [{"point":[y, x]}]. Do not provide explanatory text.

[0107] Original user input: {Original command: point to the object that can be mixed} Enhanced text input (enhanced_query): {enhanced instruction: point to spoon} If the second ruling still fails, the system will use the result generated by the previous candidate point generation model as the final result. This is because the ruling might be too strict, leading to a misjudgment of the candidate result. An empty result might mean that the current result is not executed, resulting in a lower success rate. The system does not consider the task's risk level and lacks a multiple fallback mechanism: there is only one fallback mechanism. Allowing multiple fallbacks would result in multiple invalid loops and wasted resources, without providing any real improvement in performance.

[0108] It should be noted that in some embodiments, re-adjudicating the catch-all pointing point to obtain a re-adjudication result may include the following steps: performing a hard rule check on the catch-all pointing point; when the catch-all pointing point passes the hard rule check, marking a third visual marker on the embodied scene observation image based on the catch-all pointing point, and marking a fourth visual marker on the embodied scene observation image based on the reference point information, generating a catch-all adjudication visualization image; wherein the third visual marker and the fourth visual marker use different colors and shapes; inputting the original language instruction, the enhanced pointing instruction, and the catch-all adjudication visualization image into the visual adjudication model, so that the visual adjudication model performs visual adjudication to generate a re-adjudication result; wherein the re-adjudication result includes a rewrite fidelity marker, a point count correctness marker, a target internality marker, a spatial relationship correctness marker, a confidence marker, a final acceptance marker, and an adjudication reason; when the rewrite fidelity marker, the point count correctness marker, the target internality marker, the spatial relationship correctness marker, and the final acceptance marker are all true and the confidence marker meets the preset conditions, the re-adjudication result is determined to be passed; otherwise, the re-adjudication result is determined to be failed.

[0109] For example, in some specific implementations, a catch-all ruling visualization image is generated based on the catch-all reference point, and this visualization image is input into the visual ruling model to obtain the re-ruling result. The re-ruling process can be consistent with the visual ruling process, including hard rule checks, structured rulings, and strict acceptance condition judgments.

[0110] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0111] First, it should be noted that most existing methods output bounding boxes, candidate regions, or masks directly. These results are suitable for detection or segmentation evaluation. However, if the center of the region is directly taken as the final click position, it is often impossible to guarantee that the point actually falls inside the target body, and it is also difficult to meet the requirements of embodied interaction for accurate landing point.

[0112] Secondly, while direct point output routes are closer to the target form, they typically require the model to simultaneously complete complex instruction understanding, spatial relationship parsing, and final point generation in a single inference. In tasks such as "the button closest to the reference object," "the component used to open a door," or "all targets that meet the conditions," the model is prone to semantic drift, misjudgment of reference points, or inconsistencies in the number of reference points.

[0113] Furthermore, existing technologies still lack a systematic approach to handling reference point roles, multi-point constraints, and result validity checks. Many methods focus more on finding relevant regions, while less on explicitly determining whether output points meet quantity requirements, are within the target entity, or adhere to spatial relationship constraints in the original language.

[0114] Finally, although some work has introduced the concepts of relocalization or research, a stable closed-loop mechanism is still lacking in the language-guided pointing task of a single image, which truly connects candidate point generation, low-cost pre-filtering, independent visual decision-making, fallback relocalization after failure, and final decision-making. Existing technologies have provided several valuable components, but a complete processing framework suitable for embodied visual interaction scenarios has not yet been formed.

[0115] Therefore, the core problem addressed by this invention is: how to construct a pointing and positioning method in embodied visual interaction scenarios that balances semantic fidelity, positioning accuracy, point correctness, result adjudicability, and error correction when faced with raw language instructions that include reference point constraints, spatial relationships, multi-objective sets, functional semantics, and abstract reasoning. Specifically, it aims to solve the following problems: 1. How to visually enhance the original language instructions before localization to make it easier for the candidate point generation model to generate correct candidate points.

[0116] 2. How to improve the reliability of the final result by combining low-cost hard rules with high-precision visual decision-making after candidate point generation.

[0117] 3. How to improve the ability to recover results by using a fallback relocation mechanism when a candidate point fails to pass the adjudication.

[0118] 4. How to adopt a unified and decomposable processing framework between reference point tasks and non-reference point tasks.

[0119] Specifically, this invention implements a language-guided pointing and positioning task in embodied visual interaction. The system receives scene observation images, raw language commands, and optional reference point information, and outputs one or more pixel-level pointing points for use by downstream click, grasp, instruction, action planning, or human-computer interaction modules.

[0120] The overall process adopts a phased closed-loop approach: "language enhancement—candidate product points—rigorous visual adjudication—backup repositioning—final decision." Specifically: 1. The language enhancement stage is responsible for rewriting the original language instructions into more direct and visually clear enhanced instructions.

[0121] 2. In the candidate point generation stage, a candidate point generation model is used to generate candidate pointing points on the original image.

[0122] 3. The rigorous visual adjudication stage performs hard rule checks and structured multidimensional adjudication on candidate points.

[0123] 4. In the fallback relocation phase, candidate points are regenerated when the adjudication fails, and a re-adjudication is performed on the regenerated results.

[0124] 5. In the final decision-making stage, the final direction is output based on the ruling and the fallback outcome.

[0125] like Figure 3 The diagram illustrates the overall framework of the technical solution of this invention. In some specific embodiments, this invention provides a language-guided target localization method for embodied visual interaction, such as... Figure 4 As shown, the specific steps may include the following: Step S100: Acquire the image, original language instructions, and optional reference points: Acquire the embodied scene observation image, the original language command, and optional reference point information associated with the image. Reference point information can originate from the upstream system, historical interaction context, or labeled metadata. Reference point information can be empty; when empty, the system processes the task as a non-reference point task.

[0126] Step S200, generate enhanced pointing instructions: 1. First, determine if reference point information exists in the current task; 2. If reference point information exists, the reference points are converted into marker information in the image, and an image with reference point markers is generated; 3. Construct rewritten input content based on preset rewriting prompts, and input the original language instructions, task category information, and scene image or image with reference point markers into the first visual language model; 4. The original language instructions are rewritten by combining the first visual language model with the image content; 5. Output enhanced pointing instructions for use in subsequent candidate point generation steps.

[0127] The first visual language model does not have any requirements regarding model architecture or training. This model is a pre-trained visual language model, such as a visual language model that is already open-source and in use.

[0128] An example of this step: Original command: Points to an object that can be used for mixing and stirring.

[0129] The first visual model receives the original instruction, the image, and the rewritten prompt, and then the model outputs the enhanced instruction.

[0130] The rewritten enhanced instruction: points to the spoon.

[0131] Step S300: Generate candidate pointing points using the candidate point generation model: Candidate pointing points are generated using a candidate point generation model based on the enhanced pointing instructions and the embodied scene observation image.

[0132] The candidate point generation model can adopt a visual language model, and its deployment method can be local deployment, server deployment, or network call. This invention does not limit this method.

[0133] For single-point output tasks, a single-point constraint can be added so that the candidate point generation model outputs only one candidate point when the original language instruction does not explicitly require multiple targets.

[0134] Step S300 may include the following process: 1. Construct candidate point generation instructions; 2. Input the candidate point generation instructions and the embodied scene observation image into the candidate point generation model; 3. Obtain the model output results; 4. Obtain the coordinates of the candidate points from the model output; 5. Convert the analytical coordinates to pixel-level coordinates.

[0135] Step S400: Perform hard rule checks on candidate points. Perform hard rule checks on candidate pointers. Hard rule checks include: 1. Checking for no candidate points; 2. Checking the validity of candidate point formats; 3. Checking image boundaries.

[0136] When a candidate point is empty, has an invalid candidate point format, or falls outside the image boundary, it can be directly determined that the candidate point does not meet the preset decision conditions and enter the failure processing branch.

[0137] Step S500: Perform visual decision-making on candidate points: When a candidate point passes the hard rule check, a decision visualization image is generated based on the original language instructions, enhanced pointing instructions, reference point information, and candidate pointing points, and the visual decision is executed through the visual decision model.

[0138] One approach is to: 1. Represent candidate pointing points with a first visual marker; 2. Represent reference points with a second visual marker; 3. Represent reference points and candidate points with distinguishable colors or shapes.

[0139] The visual adjudication model can output structured adjudication results, including at least: 1. rewrite fidelity; 2. point count correctness; 3. target internality; 4. spatial relationship correctness; 5. confidence score; 6. final acceptance label; 7. adjudication reason; 8. structured verification trajectory.

[0140] Step S600: Determine whether the visual decision is successful. Based on the visual decision, determine whether the candidate pointing point passes the strict decision.

[0141] A visual ruling is deemed successful only if all of the following conditions are met: 1. The acceptance flag in the visual decision is true; 2. Rewrite fidelity as true; 3. The correctness of the count is true; 4. The internality of the objective is true; 5. The spatial relationship is true; 6. The confidence level meets the preset conditions (e.g., it is greater than the confidence level threshold).

[0142] If step S700 fails, perform a fallback relocation: When the visual decision fails, a fallback relocation prompt is constructed based on the original language instructions, the enhanced pointing instructions, and the embodied scene observation image, and the fallback pointing point is regenerated through the fallback visual language model.

[0143] One approach is to have the fallback relocation prompt meet the following requirements: 1. Primitive language instructions serve as the source of truth-value semantics; 2. Enhanced pointing instructions are only used as auxiliary contexts; 3. If the task requires a single point, then only output one point; 4. If the task explicitly requires multiple objectives, then output multiple points; 5. The output uses a structured coordinate format.

[0144] Step S800: Perform a re-determination on the catch-all pointing point: A catch-all ruling visualization image is generated based on the catch-all reference point, and this visualization image is input into the visual ruling model to obtain the re-ruling result.

[0145] The re-adjudication process can be consistent with the visual adjudication process in steps S500 to S600, including hard rule checks, structured adjudication, and strict acceptance condition judgment.

[0146] Step S900: Determine the final pointing point based on the ruling result: The final pointing point is determined based on the visual decision of the candidate pointing points, the re-decision of the fallback pointing point, the candidate pointing points themselves, and the fallback pointing point itself.

[0147] The final point can be determined according to the following rules: 1. When a candidate pointing point passes visual decision, the candidate pointing point is determined as the final pointing point; 2. When a candidate pointing point fails the visual decision and the fallback pointing point passes the re-decision, the fallback pointing point is determined as the final pointing point; 3. When a candidate pointing point fails the visual decision and the fallback pointing point also fails the re-decision, the candidate pointing point is retained as the final pointing point; 4. When both the candidate pointer and the fallback pointer are unavailable, output an empty result.

[0148] Step S1000, output the final pointer result: The final pointer is output to the downstream interactive system, action execution system, or result recording module, and can be further executed as needed: 1. Finally points to the result record; 2. Save the results description information; 3. Results visualization and export.

[0149] In some specific implementations, such as Figure 5 As shown, the technical solution of the present invention can also be implemented through the following modular architecture: 1. Input Acquisition Module: Responsible for acquiring samples to be processed, including scene images, original language instructions, task categories, and optional reference point information, and performing pixel coordinate alignment or validity judgment on the reference points to provide a unified input for subsequent processing.

[0150] 2. Enhanced instruction generation module: Based on the original language instructions, this module combines image content and optional reference point context to generate more direct and easier-to-understand enhanced pointing instructions for the localization model, reducing the interference of complex language expressions on the subsequent candidate point generation.

[0151] 3. Candidate Pointer Generation Module: This module is responsible for calling the candidate point generation model, generating candidate pointers based on the enhanced pointing instructions and scene images, and completing coordinate analysis, pixel coordinate recovery, and single-point or multi-point constraint processing.

[0152] 4. Verification and Error Correction Module: Responsible for executing the complete reliability control process, including hard rule checks on candidate points, visual decision-making on candidate points, fallback relocation after decision failure, and re-decision of the fallback result, thus forming a closed-loop error correction mechanism.

[0153] 5. Result Output Module: This module is responsible for combining the visual decision results of the candidate pointers, the re-decision results of the fallback pointers, and the candidate and fallback pointers themselves to determine the final pointer and output it to the downstream embodied interaction interface or action execution interface.

[0154] In some specific application scenarios, the key control conditions applied by the technical solution of this invention are summarized in Table 1. Table 1 describes the control logic in the technical solution, and does not correspond to a specific software implementation name, nor does it limit the specific parameter naming method.

[0155] Table 1

[0156] In an example scenario, the system inputs an embodied scene image, a raw language instruction, and a reference point. The raw language instruction could be "point to the button located to the right of the reference point." The system first marks the reference point in the image, then calls a first visual language model to rewrite the raw language instruction into a more direct, enhanced pointing instruction. Subsequently, a candidate point generation model generates candidate pointing points on the original image. The system then generates a decision visualization image, where a visual decision model determines whether the candidate point truly falls within the target button body and whether it satisfies the spatial constraint of "located to the right of the reference point." If it fails, a fallback model is called to regenerate the pointing point on the original image and make another decision, finally outputting the final pointing point.

[0157] In summary, compared with the prior art, the present invention has at least the following beneficial effects: 1. Enhance the pointing instruction generation stage by rewriting complex original language instructions into expressions that are more suitable for the localization model to understand, which will make the candidate point generation more stable.

[0158] 2. The two-stage structure of "candidate point generation + strict visual decision-making" balances processing efficiency and result reliability.

[0159] 3. The combination of hard rule checking and structured visual adjudication can cover a variety of errors, such as no candidate points, incorrect point format, points out of bounds, incorrect point count, insufficient internality of the target, and inconsistent spatial relationships.

[0160] 4. The fallback relocation and re-decision constitute a recovery branch, which still provides an opportunity for error correction when the initial candidate point fails, and is more stable under complex tasks.

[0161] 5. Adhering to the principle of prioritizing the original language instructions and enhancing the auxiliary instructions during adjudication can reduce the risk of the rewritten result deviating from the original semantics.

[0162] 6. After adding explicit point constraints to multi-objective tasks, it is easier to put multi-point output tasks and single-point output tasks into the same framework.

[0163] 7. Explicitly marking reference points in images and incorporating their context into subsequent stages can improve the localization stability of reference point-guided tasks.

[0164] like Figure 6 As shown, this embodiment of the invention also provides a language-guided target localization device 900 for embodied visual interaction, which can implement the above-described method. This device may include: The first module 901 is used to acquire task data for the target task; wherein, the task data includes embodied scene observation images, original language instructions, and reference point information associated with the embodied scene observation images; The second module 902 is used to generate enhanced pointing instructions based on the original language instructions, the embodied scene observation image and reference point information, through the first visual language model. The third module 903 is used to generate candidate pointing points based on enhanced pointing instructions and embodied scene observation images through a candidate point generation model. The fourth module 904 is used to perform hard rule checks on candidate pointing points and obtain the check results; among which, hard rule checks include no candidate point checks, format validity checks, and image boundary checks. Module 5, 905, is used to update the candidate pointer point through relocation processing if the check result is unsuccessful, and return to execute the operation of Module 4 until the check result is successful. The sixth module 906 is used to respond to a pass result by performing a visual decision based on the original language instructions, enhanced pointing instructions, reference point information and candidate pointing points, and obtaining a structured decision result through a visual decision model. Module 7, 907, is used to generate the final pointer based on the candidate pointers and the structured decision results.

[0165] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0166] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0167] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0168] like Figure 7 As shown, Figure 7 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0169] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0171] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0172] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0173] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0175] The present invention provides a language-guided target localization method, device, electronic device, storage medium, and program product for embodied visual interaction. It acquires task data for a target task, including embodied scene observation images, original language commands, and reference point information associated with the embodied scene observation images. Based on the original language commands, embodied scene observation images, and reference point information, it generates enhanced pointing commands using a first visual language model. Based on the enhanced pointing commands and embodied scene observation images, it generates candidate pointing points using a candidate point generation model. It performs hard rule checks on the candidate pointing points to obtain check results, including checks for no candidate points, format validity, and image boundaries. If the check result is unsuccessful, it updates the candidate pointing points through relocation processing and returns to the step of performing hard rule checks on the candidate pointing points until the check result is successful. In response to a successful check result, it performs visual adjudication using a visual adjudication model based on the original language commands, enhanced pointing commands, reference point information, and candidate pointing points to obtain a structured adjudication result. Finally, it generates the final pointing point based on the candidate pointing points and the structured adjudication result. This invention directly outputs pixel-level pointing points, avoiding the problem of the landing point not being inside the target. The output pointing points can be directly used by downstream interactive modules such as clicking and grabbing. It enhances and rewrites the original language instructions through a first visual language model, transforming complex functional descriptions, reference point constraints, and spatial relationships into more direct visual expressions. This solves the problem of semantic drift and reference point misjudgment that easily occur in the single inference of existing direct point output routes. A dual verification mechanism combining hard rule checking and a visual adjudication model is introduced: hard rule checking filters obviously invalid results such as no candidate points, format errors, and boundary violations in a low-cost manner; the visual adjudication model performs structured judgment on candidate points from multiple dimensions such as rewriting fidelity, point number correctness, target internality, and spatial relationship correctness. Reference point information is explicitly acquired and processed, and the reference points are brought into the enhanced instruction generation and visual adjudication stages, allowing reference point constraints to play a role in the entire positioning process, improving the positioning accuracy of spatial guidance tasks. This invention, through a closed-loop process of "enhancement—generation—verification—output," can effectively improve the semantic fidelity and positioning reliability of language-guided pointing positioning.

[0176] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0177] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0180] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A language-guided target localization method for embodied visual interaction, characterized in that, The method includes the following steps: Acquire task data for the target task; wherein, the task data includes embodied scene observation images, original language instructions, and reference point information associated with the embodied scene observation images; Based on the original language instructions, the embodied scene observation image, and the reference point information, an enhanced pointing instruction is generated through the first visual language model; Based on the enhanced pointing instruction and the embodied scene observation image, candidate pointing points are generated through a candidate point generation model; Hard rule checks are performed on the candidate pointing points to obtain the check results; wherein, the hard rule checks include no candidate point checks, format validity checks, and image boundary checks; If the check result is a failure, the candidate pointer is updated through relocation processing, and the process returns to the step of performing hard rule checks on the candidate pointer to obtain the check result, until the check result is a success. In response to the inspection result being a pass, a visual decision is made based on the original language instruction, the enhanced pointing instruction, the reference point information, and the candidate pointing point, using a visual decision model to obtain a structured decision result; The final pointer is generated based on the candidate pointers and the structured decision result.

2. The method according to claim 1, characterized in that, The task data also includes task category information. The step of generating enhanced pointing instructions based on the original language instructions, the embodied scene observation image, and the reference point information using a first visual language model includes the following steps: Based on the original language instructions, the first rewritten input content is constructed by combining the preset rewriting prompts and the task category information; Determine whether the reference point information is empty; If the reference point information is empty, the first rewritten input content and the embodied scene observation image are input into the first visual language model so that the first visual language model can semantically rewrite the original language instruction and generate the enhanced pointing instruction. If the reference point information is not empty, the reference point information is converted into pixel-level marker information as reference point markers in the embodied scene observation image; wherein, the reference point markers are visualized on the embodied scene observation image with predetermined colors and shapes; Add preset supplementary content to the first rewritten input content to construct the second rewritten input content; The second rewritten input content and the embodied scene observation image with the reference point marker are input into the first visual language model so that the first visual language model semantically rewrites the original language instruction and generates the enhanced pointing instruction.

3. The method according to claim 1, characterized in that, The process of generating candidate pointing points based on the enhanced pointing command and the embodied scene observation image through a candidate point generation model includes the following steps: Obtain the task type of the target task; wherein, the task type includes a multi-point output task or a single-point output task; the original language instructions are configured with the target quantity required by the target task, if the target quantity is unique, the task type is determined to be a single-point output task, if the target quantity is not unique, the task type is determined to be a multi-point output task; If the target task is a single-point output task, construct the first constraint condition based on the preset task output constraints; If the target task is a multi-point output task, the task output constraints are combined with the output configuration that allows multi-point output to construct the first conditional constraint; Based on the enhanced pointing instruction and the first constraint condition, a candidate point generation instruction is obtained. The candidate point generation instruction and the embodied scene observation image are input into the candidate point generation model so that the candidate point generation model outputs candidate point coordinate results; wherein, the candidate point coordinate results include empty candidate point coordinates or at least one candidate point coordinate; The candidate point coordinates are converted into pixel-level coordinates corresponding to the embodied scene observation image to obtain the candidate pointing point.

4. The method according to claim 1, characterized in that, The step of performing visual adjudication based on the original language instruction, the enhanced pointing instruction, the reference point information, and the candidate pointing points, using a visual adjudication model to obtain a structured adjudication result, includes the following steps: Based on the candidate pointing points, a first visual marker is marked on the embodied scene observation image to generate a ruling visualization image; If the reference point information is not empty, the adjudication visualization image also includes a second visual marker based on the reference point information, wherein the first visual marker and the second visual marker use different colors and shapes; The original language instructions, the enhanced pointing instructions, and the adjudication visualization image are input into the visual adjudication model so that the visual adjudication model performs visual adjudication and generates the structured adjudication result; The structured decision result includes rewrite fidelity markers, point correctness markers, target internality markers, spatial relationship correctness markers, confidence markers, final acceptance markers, and decision reasons; If the rewrite fidelity flag, the point count correctness flag, the target internality flag, the spatial relationship correctness flag, and the final acceptance flag are all true and the confidence flag meets the preset conditions, the structured decision result is determined to be that the candidate pointing point passes the visual decision; otherwise, the structured decision result is determined to be that the candidate pointing point fails the visual decision.

5. The method according to claim 1, characterized in that, The process of generating the final pointer based on the candidate pointers and the structured decision result includes the following steps: If the structured decision result indicates that the candidate pointing point passes the visual decision, the candidate pointing point is taken as the final pointing point; If the structured decision result indicates that the candidate pointing point fails the visual decision, a fallback pointing point is generated based on the original language instruction, the enhanced pointing instruction, and the embodied scene observation image through a fallback visual language model. The catch-all point is re-determined to obtain the re-determined result. If the re-decision result is passed, the fallback point will be used as the final point; otherwise, the candidate point will be used as the final point.

6. The method according to claim 5, characterized in that, The process of generating a fallback pointing point based on the original language instruction, the enhanced pointing instruction, and the embodied scene observation image through a fallback visual language model includes the following steps: Obtain the task type of the target task; wherein, the task type includes a multi-point output task or a single-point output task; the original language instructions are configured with the target quantity required by the target task, if the target quantity is unique, the task type is determined to be a single-point output task, if the target quantity is not unique, the task type is determined to be a multi-point output task; If the target task is a single-point output task, a second constraint condition is constructed based on the preset task output constraint. If the target task is a multi-point output task, the task output constraints are combined with the output configuration that allows multi-point output to construct the second conditional constraint; The original language instructions are used as the source of truth semantics, the enhanced pointing instructions are used as auxiliary context, and a fallback relocation prompt is constructed by combining the indication information that the candidate pointing point has not passed the visual decision and the second constraint condition. The fallback relocation prompt and the embodied scene observation image are input into the fallback visual language model so that the fallback visual language model outputs structured coordinate results; wherein, the structured coordinate results include coordinate data of at least one fallback pointing point; The structured coordinate results are converted into pixel-level coordinates corresponding to the embodied scene observation image to obtain the bottom-point pointing point; The catch-all visual language model is different from the candidate point generation model.

7. The method according to claim 5, characterized in that, The process of re-determining the catch-all pointing point to obtain the re-determining result includes the following steps: Perform the hard rule check on the bottom-line pointing point; When the catch-all pointing point passes the hard rule check, a third visual marker is marked on the embodied scene observation image based on the catch-all pointing point, and a catch-all ruling visualization image is generated; wherein, if the reference point information is not empty, the catch-all ruling visualization image also includes a fourth visual marker marked based on the reference point information, and the third visual marker and the fourth visual marker adopt different colors and shapes; The original language instruction, the enhanced pointing instruction, and the fallback adjudication visualization image are input into the visual adjudication model so that the visual adjudication model performs visual adjudication and generates the re-adjudication result; The re-adjudication results include rewritten fidelity markers, point correctness markers, target internality markers, spatial relationship correctness markers, confidence markers, final acceptance markers, and adjudication reasons. If the rewrite fidelity flag, the point correctness flag, the target internality flag, the spatial relationship correctness flag, and the final acceptance flag are all true and the confidence flag meets the preset conditions, the re-adjudication result is determined to be passed; otherwise, the re-adjudication result is determined to be failed.

8. A language-guided target localization device for embodied visual interaction, characterized in that, The device includes: The first module is used to acquire task data for the target task; wherein, the task data includes embodied scene observation images, original language instructions, and reference point information associated with the embodied scene observation images; The second module is used to generate enhanced pointing instructions based on the original language instructions, the embodied scene observation image and the reference point information, through the first visual language model. The third module is used to generate candidate pointing points based on the enhanced pointing instructions and the embodied scene observation image through a candidate point generation model. The fourth module is used to perform hard rule checks on the candidate pointing points and obtain the check results; wherein, the hard rule checks include no candidate point checks, format validity checks, and image boundary checks; The fifth module is used to update the candidate pointer point through relocation processing if the inspection result is a failure, and return to execute the operation of the fourth module until the inspection result is a success. The sixth module is used to respond to the result that the inspection is passed by performing a visual decision based on the original language instruction, the enhanced pointing instruction, the reference point information and the candidate pointing point, through a visual decision model, to obtain a structured decision result. The seventh module is used to generate the final pointer based on the candidate pointers and the structured decision result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images