Long field of view vision measurement feedback closed loop interleaved inference method, device and storage medium

Abstract

The application discloses a long field of view visual measurement feedback closed loop interleaved reasoning method, equipment and storage medium, comprising: generating action text based on the current working image, the original text instruction and the historical callback text; inputting the structured state variable and the current working image into the rendering function, outputting the visual measurement image superimposed with the graphic primitive through the rendering function; generating callback text using the visual measurement image and the action text; determining whether the current task state satisfies the termination condition based on the visual measurement image and the callback text; if yes, ending the reasoning, if not, repeating the operation starting from generating the action text until the termination condition is satisfied. The application generates a structured state variable and obtains a visual measurement image through deterministic rendering, forces the model to verify the action based on feedback callback at each step, thereby timely cutting off the error propagation chain, and significantly improving the robustness and accuracy of long field of view reasoning.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a long-field-of-view visual measurement feedback closed-loop interleaved inference method, device and storage medium. Background Technology

[0002] Visual language models often employ text-based reasoning chains in multi-step reasoning tasks. This involves converting visual input into a text description and then processing it using the native text reasoning mechanism of the large language model. However, this pure text-based reasoning chain essentially adjusts the narrative generated by the model itself, rather than providing real-time feedback to the underlying visual task state. In long-field state-based decision-making, if a local deviation occurs in an intermediate reasoning step after each action updates the underlying environmental state, the entire reasoning chain will drift. Subsequent reasoning steps may remain consistent in textual logic, but they are severely misaligned with the actual visual state, ultimately leading to trajectory-level task failure.

[0003] To alleviate these problems, traditional techniques have introduced intermediate visual feedback. However, these solutions have failed to fundamentally address the core deficiency of "lack of state-aligned closed-loop feedback." Generative methods suffer from randomness and composite artifacts, external tools are prone to generating unreliable visual illusions, and latent tokens are highly abstract and difficult to optimize.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a long-field-of-view visual measurement feedback closed-loop interleaved inference method, device and storage medium, which aims to solve the technical problem that existing visual language models fail in multi-step inference due to the lack of closed-loop feedback that is strictly aligned with the real visual state, resulting in the accumulation of intermediate deviations and causing trajectory-level task failure.

[0006] To achieve the above objectives, this application proposes a long-field-of-view visual measurement feedback closed-loop interleaved inference method, applied to a long-field-of-view visual measurement feedback closed-loop interleaved inference device, the method comprising: Set the original input image as the current working image for the first reasoning round; Based on the current working image, the original text command, and the historical callback text, an action text is generated, wherein the historical callback text is initialized to empty before the action text is generated; The structured state variables contained in the action text and the current working image are used as inputs to the rendering function, which generates a visual measurement image superimposed with graphic primitives. The callback text is generated using the visual measurement image and the action text. Based on the visual measurement image and the callback text, determine whether the current task status meets the termination condition; if it does, end the reasoning.

[0007] In one embodiment, the step of using the structured state variables contained in the action text and the current working image as input to a rendering function, and generating a visual measurement image superimposed with graphic primitives through the rendering function, includes: Based on the spatial geometric information carried in the structured state variables, graphic primitives matching the type of spatial geometric information are superimposed on the corresponding positions of the current working image, wherein the graphic primitives are selected from one or more of point markers, line segments, polylines, directional arrows, bounding boxes, or trajectory curves; The graphic primitives are drawn on the current working image using a deterministic algorithm to generate the visual measurement image.

[0008] In one embodiment, the step of generating callback text using the visual measurement image and the action text includes: The visual measurement image and the action text are input together into a multimodal language model; The multimodal language model is used to detect whether the graphic primitives in the visual measurement image are consistent with the expected spatial relationship declared in the action text; If they match, output a callback text confirming the validity of the action text; If there is a discrepancy, output a callback text containing corrective information about the structured state variables.

[0009] In one embodiment, before the step of inputting the visual measurement image and the action text together into the multimodal language model, the method further includes: Action text, visual measurement images, and corresponding callback text are used as supervisory data to perform supervised fine-tuning of the initial multimodal language model; To reinforce learning, verifiable reward signals are set, including accuracy rewards and efficiency rewards; The multimodal language model is obtained by using a group-relative policy optimization algorithm and the verifiable reward signal to train the supervised fine-tuned multimodal language model through reinforcement learning.

[0010] In one embodiment, the verifiable reward signal includes accuracy rewards and efficiency rewards, and the step of setting verifiable reward signals for reinforcement learning includes: Based on whether the final state of the task at the end of the current reasoning round has reached the target state, a positive or negative reward is assigned according to the judgment result. A length penalty factor is calculated based on the total length of the action text and callback text generated in the current inference trajectory. When the accuracy reward is determined to be positive, the efficiency reward is set based on the length penalty factor. The length penalty factor is positively correlated with the efficiency reward.

[0011] In one embodiment, the step of using a group-relative policy optimization algorithm and the verifiable reward signal to train the supervised fine-tuned multimodal language model through reinforcement learning to obtain the multimodal language model includes: A low-rank adaptation module is inserted in parallel next to the query mapping matrix and value mapping matrix of each layer attention calculation module in the multimodal language model. The low-rank adaptation module contains a dimension reduction matrix and an dimension increase matrix. The dimension reduction matrix compresses the original hidden state to a low-dimensional space, and the dimension increase matrix restores the low-dimensional features to the original dimension. During the training process of the reinforcement learning, the original pre-training parameters of the multimodal language model are frozen, thereby performing gradient updates on the dimensionality reduction matrix parameters and dimensionality increase matrix parameters in the low-rank adaptation module.

[0012] In one embodiment, the step of determining whether the current task state meets the termination condition based on the visual measurement image and the callback text, and ending the inference process if the condition is met, includes: Read the spatial overlap state between the current location marker and the target area marker from the visual measurement image, or extract the task completion identifier from the callback text; If the spatial overlap state reaches a preset threshold or the task completion flag is true, then the current task state is determined to meet the termination condition.

[0013] In one embodiment, the long-field-of-view visual measurement feedback closed-loop interleaved inference method further includes: In the next inference round, the visual measurement image is used as the updated current working image; The updated current working image is used as the input base map for the rendering function, so that the newly generated visual measurement image inherits the graphic primitives superimposed in all previous rounds; The updated historical callback text and the original text instruction are used together as text conditions for generating the next round's action text, enabling the multimodal language model to adjust subsequent actions based on historical correction information.

[0014] Furthermore, to achieve the above objectives, this application also proposes a long-field-of-view visual measurement feedback closed-loop interleaved inference device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the long-field-of-view visual measurement feedback closed-loop interleaved inference method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the long field-of-view visual measurement feedback closed-loop interleaved inference method as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: Set the original input image as the current working image for the first inference round; generate action text based on the current working image, the original text instruction, and the historical callback text, wherein the historical callback text is initialized to empty before generating the action text; use the structured state variables contained in the action text and the current working image as input to the rendering function, and generate a visual measurement image superimposed with graphic primitives through the rendering function; generate callback text using the visual measurement image and the action text; determine whether the current task state meets the termination condition based on the visual measurement image and the callback text, and end the inference if it does.

[0017] The technical solution of this application uses the structured state variables in the action text to generate a visual measurement image of superimposed graphic primitives as deterministic closed-loop feedback, thereby achieving real-time state alignment and error blocking, and significantly improving the reliability and success rate of long trajectory reasoning. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the long-field-of-view visual measurement feedback closed-loop interleaved reasoning method of this application; Figure 2 This is a detailed schematic diagram of step S30 in the first embodiment; Figure 3 This is a detailed schematic diagram of step S40 in the first embodiment; Figure 4 This is a detailed process diagram based on step S50 in the first embodiment; Figure 5This is a flowchart illustrating the second embodiment of the long-field-of-view spatial long-field-of-view visual measurement feedback closed-loop interleaved reasoning closed-loop method of this application; Figure 6 This is a schematic diagram of the closed-loop reasoning steps; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the long field-of-view visual measurement feedback closed-loop interleaved inference method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] In the field of long-field-of-view spatial reasoning in visual language models, conventional technical approaches mainly follow the pure text thinking chain or generative visual feedback schemes, but they have inherent defects and are difficult to balance reasoning closure and state alignment accuracy.

[0024] Pure text-based reasoning chains essentially adjust the model's own narrative, lacking real-time feedback on the underlying task state. When a local deviation occurs in an intermediate inference step, the entire inference chain drifts. Subsequent steps, while logically consistent within the text, are severely misaligned with the actual visual state, ultimately leading to the failure of trajectory cascading. Some solutions attempt to introduce generative images or external tools to provide visual feedback, but the randomness and composite artifacts in multi-step generation inject noise into the inference loop, and external tools also produce unreliable artifacts when parameters are incorrect. While latent token schemes avoid pixel generation, they are highly abstract and difficult to optimize.

[0025] A comprehensive analysis reveals that the core dilemma faced by the aforementioned technical approaches lies in the fact that, although the reasoning process using open-loop textual narration or high-noise visual channels is intuitive, its lack of feedback and continuous accumulation of errors fundamentally contradict the inherent requirements of long-field-of-view spatial reasoning for real-time state anchoring and high robustness, and thus cannot actively block the step-by-step propagation of errors during the reasoning process.

[0026] To address the aforementioned shortcomings, this application proposes a visual measurement feedback and closed-loop interleaved reasoning method for long-field-of-view spatial reasoning. This method decomposes each reasoning step into a three-stage closed loop of "think-observation-callback": the model first generates action text containing structured state variables, then uses a deterministic rendering function to overlay these variables onto the original image to output a visual measurement image, and finally forces the model to generate confirmation or correction callback text based on this visual feedback. The visual measurement image and the callback text are then used as input for the next iteration, forming a state-aligned closed-loop correction mechanism.

[0027] Through the above-mentioned technical means, this application provides noise-free state anchors with lightweight deterministic rendering without relying on image generation backbone or external tools, realizing timely cut-off and active correction of errors, solving the technical problem of deviation accumulation and lack of state feedback correction capability under open-loop text reasoning, and completing the technical leap from unidirectional narration to closed-loop interleaved reasoning.

[0028] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0029] Based on this, embodiments of this application provide a long-field-of-view visual measurement feedback closed-loop interleaved reasoning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the long-field-of-view visual measurement feedback closed-loop interleaved reasoning method of this application. In this embodiment, the long-field-of-view visual measurement feedback closed-loop interleaved reasoning method includes steps S10 to S60: Step S10: Set the original input image as the current working image for the first reasoning round; In this embodiment, the system initialization operation is first performed to obtain three parts of input data: the original input image, the original text instructions, and the historical callback text. The original input image is usually a snapshot of the visual state of the task scene, such as a top-down view of a maze or a photo of the arrangement of objects on a table. This image serves as the absolute visual reference for the entire reasoning process. The original text instructions describe the final goal of the task in natural language, such as "start from the green dot and walk along the path to the red pentagram." The historical callback text is a container used to accumulate and store corrective feedback generated in previous reasoning rounds. Before the start of the first reasoning round, the historical callback text is explicitly set to null.

[0030] The original input image is designated as the current working image for the first inference round. The current working image is a dynamic variable that updates over time and is replaced by a new visual measurement image after each inference round. In contrast, the original input image remains unchanged throughout the inference lifecycle, serving as a reference for the original environment layout. While the history callback text is empty in the first round, it will contain correction records from all previous rounds starting from the second round.

[0031] The core function of this data processing logic step is to establish three initial state variables for closed-loop inference iteration. Specifically, the original input image is loaded through an image decoding library and converted into a pixel matrix in memory. The original text instructions are stored as strings. The historical callback text is initialized as an empty list, with each element being a callback text string. For easy access in subsequent steps, these three variables are encapsulated in a context object, which also contains an iteration counter initialized to 0.

[0032] Furthermore, a key technical approach in this process lies in clearly distinguishing between the "original input image" and the "current working image." Conventional methods often maintain only a single image, resulting in the loss of original information after each update. This embodiment retains the original image as an unchanging anchor point, allowing the current working image to accumulate visual measurement markers, thus enabling the model to consistently compare the current state with the initial layout. Additionally, the seemingly simple design of empty historical callback text actually lays the foundation for subsequent loop closure. When the model executes its first inference round, the historical callback text provides no additional information; the model performs its initial actions solely based on the original image and instructions. From the second iteration onwards, the historical callback text carries error correction information, guiding the model to adjust its behavior.

[0033] The initialization process described above only involves data preparation and variable assignment. In actual deployment, this system can be called as a function library, with the input parameters provided by the upper-layer application. To ensure versatility, the original input image format supports common formats such as JPEG and PNG, and the original text commands support Chinese, English, and mixed languages.

[0034] Step S20: Generate action text based on the current working image, the original text command, and the historical callback text, wherein the historical callback text is initialized to empty before generating the action text; This embodiment performs action generation operations on a multimodal language model. It receives three output variables: the current working image, the original text instructions, and the historical callback text. The current working image is the original input image in the first round, and in subsequent rounds it becomes a rendered and overlaid visual measurement image. The original text instructions remain unchanged, providing the final goal constraint for the task. The historical callback text records the self-correction content of the model in all previous rounds.

[0035] These three inputs are simultaneously fed into a pre-trained multimodal language model. This model needs to be able to understand both visual and textual content. The inputs are constructed as follows: the current working image is used as the input to the visual channel, and the original text command is concatenated with the historical callback text as the input to the text channel. After forward inference, the model outputs a piece of natural language text, i.e., action text.

[0036] The core characteristic of this action text is that it must contain structured state variables. Structured state variables refer to spatial geometric information expressed in a clear, parsable format. For example, in a maze navigation task, the model might output, "My current coordinates are (120, 340), the target coordinates are (450, 380), and I plan to move to the target along a straight line." Here, "(120, 340)," "(450, 380)," and "move to the target along a straight line" are all instances of structured state variables. These variables can be encoded using JSON format, XML tags, or simple bracket markings.

[0037] The data processing logic described above is divided into three sub-stages. The first sub-stage: The model visually encodes the current working image, extracting key visual elements, including existing graphical primitives such as trajectory lines and markers, as well as obstacles and boundaries in the original scene. The second sub-stage: The model performs cross-modal fusion of the visual encoding results with text instructions and historical callback text, inferring the next action to be taken from the internal latent space. The third sub-stage: The model decodes the inference results into natural language and forces the output of structured state variables according to a predefined format. This forced behavior is obtained through supervised fine-tuning training; the model has learned that state parameters must be explicitly declared when outputting actions.

[0038] In practice, open-source models such as LLaVA and Qwen-VL are used as the base models, and fine-tuning is performed on a specially constructed interleaved inference dataset. The fine-tuned model can recognize the superimposed markings on the image and understand their meaning when it receives a visual measurement image.

[0039] Furthermore, the key technical approach in this process lies in forcing the output of structured state variables. Conventional methods only require the model to output action descriptions, such as "walk forward." This requires the model to simultaneously output quantified state parameters, such as "move forward 30 pixels." This parameter is directly used for subsequent deterministic rendering, forming a crucial link in the closed-loop feedback. Without this parameter, the rendering function cannot draw accurate graphical primitives. The action text generation process described above is the "thinking" stage of closed-loop reasoning, providing quantifiable action planning for subsequent visual measurements.

[0040] Step S30: The structured state variables contained in the action text and the current working image are used as inputs to the rendering function, and a visual measurement image with superimposed graphic primitives is generated through the rendering function. This embodiment performs rendering and generation of a visual measurement image, receives the generated action text, parses out structured state variables from it, and simultaneously receives the current working image as the rendering base map. The structured state variables and the current working image are then input into the rendering function.

[0041] The rendering function is a lightweight, deterministic computer graphics renderer. Its core characteristic is the absence of randomness: given identical input parameters, the rendering function always outputs the exact same image. This is fundamentally different from neural network-based generation methods, which, even with identical input, can produce different outputs due to sampling randomness. The rendering function is implemented without relying on any deep learning model, using only standard graphics rendering algorithms.

[0042] The rendering function works as follows: First, it parses the structured state variables to extract their spatial geometric information. For example, the structured state variables might contain "starting point coordinates (100,100)", "current point coordinates (150,140)", "path point list [(100,100),(120,110),(150,140)]", and "direction vector (30,30)". The parser converts this information into internal drawing instructions. Then, on a copy of the current working image, it sequentially overlays graphic primitives according to the drawing instructions. The type of graphic primitive matches the spatial geometric information: point coordinates correspond to point markers, usually drawn as colored dots or crosses; the path point list corresponds to polylines, connecting the points in sequence and drawing them as broken lines; the direction vector corresponds to arrows, drawing arrows from the current point along the vector direction; bounding boxes or trajectory curves are processed similarly.

[0043] The style parameters of graphic primitives are configurable: points can be colored red for emphasis, polylines can be colored blue with a semi-transparent fill, and arrows can be colored green with a triangular head. All drawing operations do not change the image size or resolution; they simply overlay new pixels on top of the original pixels.

[0044] The output image is the visual measurement image. Essentially, the visual measurement image externalizes the model's internal planning into a visual signal. The model's claimed "current point coordinates (150, 140)" is rendered, resulting in a prominent red dot at that coordinate location. The model can then view this red dot in subsequent steps to verify whether it is indeed in its expected position.

[0045] The data processing logic in this step can be summarized as a "declaration to visualization" transformation. Its key technical approach lies in using deterministic rendering instead of generative models. This choice brings multiple advantages: First, rendering speed is extremely fast, typically completing in milliseconds, far faster than image generation models; second, there are no artifacts, achieving pixel-level accuracy; third, it does not rely on any external tools or network calls; and finally, it offers strong interpretability, allowing users to directly see the state claimed by the model.

[0046] In practice, the rendering function can be implemented using Python's PIL library or the OpenCV library. To improve efficiency, the rendering function is designed as a pure function with no side effects, facilitating parallel calls. For scenarios requiring continuous multi-step rendering, the function supports incremental rendering mode, which only draws newly added graphic primitives instead of redrawing all content each time.

[0047] The visual measurement images described above are the core carrier of closed-loop feedback, replacing the expensive generative visual signals in traditional methods, and providing highly reliable state alignment feedback at extremely low cost.

[0048] Step S40: Generate callback text using the visual measurement image and the action text; This embodiment performs the callback text generation operation, receives visual measurement images and action text, inputs both into a multimodal language model, and requires the model to perform verification and reflection tasks.

[0049] Specifically, the model checks whether the actual superimposed graphic primitives on the visual measurement image are consistent with the expected spatial relationships declared in the action text; this is part of the visual-linguistic alignment verification process. For example, the action text might declare "the current point is in the center of the channel." The model observes the red dot marker drawn on the visual measurement image and finds that the point is indeed located within the empty area of ​​the channel, without touching the walls on either side, thus considering it consistent. Another example: the action text declares "I have reached the target point," but the visual measurement image shows that the overlap area between the red dot marker and the green target area marker is zero, thus considering it inconsistent.

[0050] Based on the above judgments, the model outputs callback text. There are two types of callback text. The first is a confirmatory callback: when the model's judgment is consistent, it outputs text like "Action valid, state variables match visual measurements, continue execution." The second is a corrective callback: when the model's judgment is inconsistent, it outputs text containing correction information, such as "Correction: The actual current point is (160, 145), not (150, 140) declared in the action text; it is recommended to recalculate the path and update the correct coordinates to (160, 145)." The corrective callback text contains the corrected structured state variables.

[0051] This data processing logic specifically refers to the "checksum" mechanism in an automated system. In traditional methods, the model generates actions and executes them directly without a verification step. In this embodiment, the model is forced to perform a fact check based on objective visual measurements before generating the final decision. This check utilizes the model's visual understanding ability: after seeing an image with red dots superimposed, the model can identify the coordinates of the red dots (through visual positioning ability) and compare them with the declared coordinates it remembers.

[0052] In practice, a specific prompt template is constructed to organize the action text and visual measurement images. The prompt template can be designed as follows: "Below is the action text you generated previously, which includes state variables. Now look at this image; graphical markers have been drawn on the image based on your state variables. Please check if the markers on the image match your state variables. If they don't match, please output the corrected state variables." The model generates callback text according to this prompt. Furthermore, during the process of forcing the model to output corrective information, when the model detects inconsistencies, it must point out the error and provide the corrected values. These corrected values ​​are then recorded in the historical callback text and referenced by the model in the next iteration, forming a complete closed-loop correction cycle.

[0053] The callback text generation process described above is the "callback" stage of closed-loop reasoning, which, together with the "thinking" and "observation" stages, forms a three-stage closed loop of "thinking-observation-callback." Each step forces the model to self-correct in the face of objective facts, rather than continuing along erroneous narratives. For more details on the reasoning steps of the above three-stage "thinking-observation-callback" closed loop, please refer to... Figure 6 , Figure 6 This is a schematic diagram of the closed-loop reasoning steps.

[0054] Step S50: Based on the visual measurement image and the callback text, determine whether the current task status meets the termination condition. If it does, end the inference.

[0055] This embodiment performs termination condition determination. It receives the generated visual measurement image and callback text, and based on these two pieces of information, determines whether the current task status meets the termination condition. The termination condition is a preset task completion standard; different types of tasks correspond to different termination condition definitions.

[0056] The first criterion comes from the visual measurement image. The system reads the spatial state of graphic primitives from the visual measurement image. Since the visual measurement image is generated through deterministic rendering, the positional information of the graphic primitives can be accurately obtained. For example, in a maze navigation task, a current point marker and a target region marker are drawn on the visual measurement image. The system calculates the spatial overlap between the two markers: if the pixel coordinates of the current point marker fall within the boundary of the target region marker, or if the intersection area of ​​the two graphic primitives is greater than zero, then the spatial overlap is considered to have reached a threshold. The threshold can be set such that an intersection area greater than one pixel is considered an overlap, or it can require complete containment.

[0057] The second criterion comes from the callback text. The callback text is parsed to look for task completion markers. These markers are typically a set of keywords, such as "task completed," "goal reached," or "success," or a special symbol. If the confirmatory callback text explicitly states that the task is complete, and this statement does not contradict the visual measurement image, the model can terminate based on this. Furthermore, if the callback text contains descriptions like "all sub-goals have been achieved," it is also considered to meet the termination criteria.

[0058] The data processing logic described above employs dual verification, as relying on a single source of information may pose risks: visual measurement alone may misjudge overlapping states due to rendering accuracy issues; model semantics alone may incorrectly claim completion due to model illusions. A combination of visual measurement and semantic judgment is used, terminating when both indicate completion, or when one explicitly points to completion and the other is not contradictory. Specific rules can be configured according to task requirements.

[0059] In the specific implementation process, a priority rule is adopted: First, the spatial overlap of the visual measurement is checked. If the overlap is determined to be true, then the callback text is checked to see if it contains an abnormal rejection signal; if not, the process is terminated. If the visual overlap is false but the callback text explicitly declares completion and provides sufficient reasons, such as "the target area has been satisfied by other conditions," the process can also be terminated. In addition, a fallback termination condition needs to be set: when the number of iterations exceeds the maximum limit, the process is forcibly terminated.

[0060] Additionally, a fallback termination condition needs to be set, which forces termination when the number of iterations exceeds the maximum step limit. This limit is an integer preset by the user or the system (e.g., 50 steps, 100 steps) to prevent infinite iterations due to the model getting stuck in an infinite loop, error state, or the task itself being unreachable. In actual execution, each iteration corresponds to a complete "think-observe-callback" closed loop. The system maintains a step counter, which increments by 1 after each closed loop iteration. When the counter value reaches or exceeds the preset maximum step count, the inference process is forcibly terminated regardless of whether the current visual measurement image and callback text indicate task completion. Upon forced termination, the system should output a timeout or step limit exceedance error message and retain completed intermediate results for analysis. This fallback mechanism ensures that this method will not consume computational resources indefinitely under any extreme circumstances.

[0061] If the determination result is that the termination condition is not met (i.e., the visual measurement has not reached the spatial overlap threshold, the callback text does not explicitly claim completion and does not meet other completion rules, and the current iteration step number has not exceeded the maximum number of steps set manually), then the iteration preparation operation is executed to continue the next round of closed-loop inference.

[0062] Iterative preparation involves two core update actions. The first update action is to use the generated visual measurement image as the new working image for the next inference round. It's important to note that this new working image already contains the graphical primitives rendered and overlaid in all previous rounds. For example, the first round drew the starting point marker, the second round drew the line segment from the starting point to the first turning point, and the third round drew the path that continues to extend. After the update, all these graphical primitives are retained, collectively forming the accumulated visual state. The second update action is to use the generated callback text as the updated historical callback text. Historical callback text is typically stored append-only, rather than overwritten. The historical callback text described above is a list, with each element being a callback text entry. Newly generated callback text is added to the end of the list so that the model can see all historical correction records in the next round.

[0063] After completing the two updates mentioned above, the control flow jumps back to repeatedly executing the entire closed-loop process that started with the generation of the action text. Thus, a typical loop control structure is formed, with each iteration including three phases: "think-observe-callback," as well as termination judgment and state update.

[0064] The data processing logic in this step is the core of the closed-loop inference loop control. Its key technical means lies in passing the visual measurement image as a feedback signal to the next round, while simultaneously passing the callback text as a correction memory. This constitutes a double closed loop: a visual closed loop, because the current working image is constantly replaced by the latest rendering result; and a semantic closed loop, because the historical callback text allows the model to "remember" its previous mistakes and corrections.

[0065] In practical implementation, safety mechanisms need to be set. First, there should be a maximum iteration limit, for example, set to 50 iterations. Exceeding this limit should force termination and report a timeout error. This is a manually set maximum number of steps, consistent with the aforementioned fallback termination condition: when the actual number of iterations exceeds this limit, the inference should be forcibly terminated to prevent system hangs or resource exhaustion. Second, there should be state change detection. If the current working image shows no substantial change for several consecutive rounds, it indicates the model is trapped in an infinite loop and should also be terminated early.

[0066] The iterative process described above is the fundamental guarantee for achieving robust inference over long horizons in this application. Even if the model generates incorrect action text at a certain step, as long as the inconsistency is detected by visual measurement in subsequent callback stages and correction information is output, this correction information will be added to the historical callback text and affect the next round of action generation. The error is confined to a single step and will not cascade like traditional methods. The entire process does not rely on any image generation model or external tools, and can be completed solely through lightweight rendering and a standard multimodal model.

[0067] Furthermore, you can also view Figure 2 , Figure 2 This is a detailed process diagram based on step S30 in the first embodiment. Figure 2 The step of using the structured state variables contained in the action text and the current working image as input to the rendering function, and generating a visual measurement image superimposed with graphic primitives through the rendering function, includes S31~S32: Step S31: Based on the spatial geometric information carried in the structured state variables, superimpose graphic primitives matching the type of spatial geometric information at the corresponding positions of the current working image, wherein the graphic primitives are selected from one or more of point markers, line segments, polylines, directional arrows, bounding boxes, or trajectory curves. Step S32: Use a deterministic algorithm to draw the graphic primitives on the current working image to generate the visual measurement image.

[0068] This embodiment further defines the specific execution method of the above-mentioned rendering function, that is, how to convert structured state variables into visual measurement images. This process is divided into two closely related sub-steps: graphics primitive type matching and deterministic rendering.

[0069] First, the structured state variables extracted from the action text are parsed to obtain the spatial geometric information they carry. Spatial geometric information is a set of data describing position, shape, direction, or trajectory, such as the coordinates of a point, a line segment between two points, a polyline connecting multiple points, a direction vector, a rectangular bounding box, or a smooth curve. The data format of the above information can be an array of coordinate pairs, vector pairs, or parametric equations.

[0070] Based on the type of spatial geometry information, the rendering function overlays a matching graphic primitive at the corresponding position in the current working image. The graphic primitives are selected from a predefined set, including point markers, line segments, polylines, directional arrows, bounding boxes, and trajectory curves. There is a one-to-one correspondence between different types of spatial geometry information and graphic primitives: a single point coordinate corresponds to a point marker; two coordinates correspond to a line segment; three or more ordered coordinates correspond to a polyline; a direction vector corresponds to a directional arrow; a rectangular region corresponds to a bounding box; and a series of consecutive position points or curve parameters correspond to a trajectory curve. This matching rule is pre-hard-coded in the rendering function, requiring no model inference. For example, when the structured state variable contains "current point coordinates (150, 140)," it is identified as a single point type, and a point marker is selected as the graphic primitive. The specific style of the point marker can be a red dot with a diameter of 8 pixels or a crosshair, depending on the configuration. For example, when the structured state variable contains "path point list [(100,100),(120,110),(150,140)]", the system recognizes it as a multi-point sequence type, selects a polyline as the graphic primitive, and draws a blue polyline with a line width of 3 pixels by connecting the three points in sequence. As another example, when the structured state variable contains "movement direction vector (30,30)", the system recognizes it as a direction type, selects a direction arrow as the graphic primitive, and draws an arrow starting from the current point coordinates along the vector direction, with the arrowhead being a triangle.

[0071] The data processing logic described above is essentially a type dispatcher. This dispatcher iterates through all entries in the structured state variables, performs type determination on each entry, and generates a list of drawing instructions. Each instruction contains the graphics primitive type, position parameters, and style parameters.

[0072] Next, using the drawing instruction list generated in the previous step as input, a deterministic algorithm is employed to draw the aforementioned graphic primitives on the current working image. A deterministic algorithm means that the drawing process does not rely on any random numbers, neural network sampling, or the state of the external environment. Given the same current working image and the same drawing instructions, the algorithm always outputs the exact same visual measurement image.

[0073] In practice, deterministic algorithms can be based on the Bresenham line algorithm to draw line segments, the midpoint circle algorithm to draw circular dot markers, and the polygon fill algorithm to draw arrow heads. All of these algorithms are classic computer graphics algorithms, possessing determinism and repeatability. For example, when drawing a line segment from (100, 100) to (150, 140), the Bresenham algorithm always selects the same set of pixels, unaffected by runtime or system load. Similarly, when drawing a circle with a radius of 4 pixels, the midpoint circle algorithm always illuminates the same pixel position.

[0074] To avoid modifying the original image, the deterministic algorithm first creates a deep copy of the current working image and then performs all drawing operations on this copy. The drawing order follows the sequence in the drawing instruction list; later drawn primitives may cover earlier drawn parts, which is normal behavior for transparent overlay. For primitives requiring a semi-transparent effect, the algorithm uses alpha blending to calculate the final color value for each pixel.

[0075] After all the drawing is complete, the generated image is the visual measurement image. The entire rendering process described above is typically completed within milliseconds and does not involve any model calls or network requests. These limitations clarify the specific implementation method of visual measurement: based on the overlay of type-matched graphical primitives, combined with deterministic algorithms for drawing. This design ensures that visual measurement is lightweight, noise-free, and highly reproducible, fundamentally different from generative methods.

[0076] Furthermore, you can also view Figure 3 , Figure 3 This is a detailed process diagram based on step S40 in the first embodiment. Figure 3 The step of generating callback text using the visual measurement image and the action text includes S41~S43: Step S41: Input the visual measurement image and the action text together into the multimodal language model; Step S42: Detect whether the graphic primitives in the visual measurement image are consistent with the expected spatial relationship declared in the action text using the multimodal language model; Step S43: If they match, output a callback text confirming the validity of the action text. Step S44: If there is a discrepancy, output a callback text containing the corrected structured state variable corrections.

[0077] This embodiment further defines the specific execution method of generating callback text, that is, how the multimodal language model performs consistency detection based on visual measurement images and action text and outputs the corresponding callback results. This process is divided into four ordered sub-steps: input loading, consistency detection, confirmation output, and correction output.

[0078] First, the generated visual measurement image and the generated action text are input into the same multimodal language model. The input is constructed as follows: the visual measurement image is scaled or cropped to a fixed size required by the model and serves as input to the visual encoder. The action text is directly concatenated into a pre-defined cue template and serves as input to the text encoder. The cue template is designed to explicitly inform the model of the task to be performed, such as: "Please check whether the spatial relationships declared in the following action text are consistent with the graphic markings drawn on the image." To maintain the model's attention, the positions of the visual measurement image and action text in the input sequence can be flexibly arranged; a common practice is to place the image before the text. After receiving the input, the model simultaneously extracts visual and text features and establishes a correspondence between the two in an internal cross-modal attention layer.

[0079] Then, the multimodal language model described above is used to detect whether the graphic primitives in the visual measurement image are consistent with the expected spatial relationships declared in the action text. This detection process is essentially a visual-language alignment reasoning task. The model needs to complete the following sub-tasks: First, identify which graphic primitives exist in the visual measurement image, including the position of points, the direction of line segments, the shape of polylines, the direction of arrows, etc. Second, parse the expected spatial relationships previously declared by the model from the action text, such as "the current point is located in the center of the channel" or "the arrow points to the red area". Third, compare the results of visual recognition with the expected text declaration item by item to determine whether there is any deviation. The model's internal computation utilizes its visual localization and spatial reasoning capabilities acquired during the pre-training phase. For example, when the action text declares "the current point coordinates are (150, 140)," the model needs to observe whether the point marker on the visual measurement image actually appears near the coordinates (150, 140). Since the model does not have precise pixel coordinate perception, it usually uses relative position judgment: whether the point marker matches the declared spatial position relative to the environmental features in the image. To improve detection accuracy, the prompt template can be used to require the model to output structured judgment results, such as "consistent" or "inconsistent", as well as the specific manifestations of inconsistency.

[0080] Next, if the above detection results are consistent—that is, the graphic primitives in the visual measurement image match the expected spatial relationship of the action text declaration—the model outputs a confirmatory callback text. This callback text can be designed as "Confirmation: Action text is valid, state variables match visual measurement," or more simply, "Valid." The confirmatory callback text does not need to contain any modification information; it only indicates that the model acknowledges the correctness of the current action. After receiving the confirmatory callback, the system retains the current visual measurement image and action text for subsequent processing.

[0081] Finally, if the above detection results are inconsistent—that is, there is a deviation between the graphic primitives in the visual measurement image and the expected spatial relationship declared in the action text—the model outputs corrective callback text. This corrective callback text must contain corrected structured state variables. For example, the action text declares "Current point is (150, 140)," but the model observes that the point marker in the visual measurement image is actually located near (160, 145). The model can output: "Correction: The current point in the action text is incorrect; the actual current point should be (160, 145). Please update the state." The corrected structured state variables use the same format as the action text, ensuring that the system can parse them and use them for the next round of rendering. The key feature of this application is that when the model outputs a corrective callback, it not only points out the error but also provides specific correction values. The underlying technique is to train the model through supervised fine-tuning to learn to actively generate correction parameters when inconsistencies are detected.

[0082] The above processes together constitute the complete logic for callback text generation. This logic forces the model to perform a fact check based on objective visual measurements before executing any action, and decides whether to confirm or correct based on the check results. This mechanism effectively prevents the propagation of single-step errors to the entire trajectory and is the core element for achieving closed-loop interleaved reasoning. In practical implementation, the above prompt templates and output formats can be solidified through a small number of examples or fine-tuning instructions to ensure stable system operation.

[0083] In addition, before the step of inputting the visual measurement image and the action text together into the multimodal language model, the method further includes: Action text, visual measurement images, and corresponding callback text are used as supervisory data to perform supervised fine-tuning of the initial multimodal language model; To reinforce learning, verifiable reward signals are set, including accuracy rewards and efficiency rewards; The multimodal language model is obtained by using a group-relative policy optimization algorithm and the verifiable reward signal to train the supervised fine-tuned multimodal language model through reinforcement learning.

[0084] This embodiment further defines the training method of the aforementioned multimodal language model, namely, how to make the model fully adapt to the interleaved reasoning process of "thinking-observation-callback" through two-stage optimization. This process includes two stages: supervised fine-tuning and reinforcement learning.

[0085] Construct the training dataset required for supervised fine-tuning. Each training dataset contains three parts: action text, a visual measurement image, and the correct callback text corresponding to that action text. The action text is generated by the model or manually annotated and contains structured state variables. The visual measurement image is drawn using the deterministic rendering function described above based on the structured state variables in the action text. The callback text is the standard answer corresponding to the action text: if the state variables in the action text match those in the visual measurement image, the callback text is confirmatory; otherwise, it is corrective, containing the corrected structured state variables. The supervised data described above is essentially a process-level alignment dataset that teaches the model what callback text it should output when it sees a given action text and its corresponding visual measurement image.

[0086] The above supervised data is then input into the initial multimodal language model for supervised fine-tuning. The goal of fine-tuning is to enable the model to learn to generate output consistent with the standard callback text based on the input visual measurement image and action text. This fine-tuning stage does not require the model to learn how to generate the visual measurement image, as the image is generated by a deterministic rendering function. The model only needs to learn a conditioned reflex: given action text and the rendered image, output the correct callback. Through training on a large amount of such interleaved data, the model gradually masters the ability to "verify" and "correct".

[0087] Next, verifiable reward signals are set for the reinforcement learning phase. These signals have two dimensions. The first is an accuracy reward: this reward is based on the final completion state of the task. For example, in a maze navigation task, a positive reward is given if the model successfully reaches the target area from the starting point within a finite number of steps; a negative reward is given if it times out, goes out of bounds, or fails to reach the target. The accuracy reward is sparse and is only provided at the end of the trajectory. The second is an efficiency reward: this reward encourages the model to use fewer inference steps while maintaining correctness. Specifically, the total length of the action text and callback text generated in the current inference trajectory is calculated, such as the total number of steps or tokens, and then a length penalty factor is introduced. When the accuracy reward is positive, the efficiency reward is positively correlated with the length penalty factor; that is, the shorter the trajectory, the higher the efficiency reward. When the accuracy reward is negative, the efficiency reward can be set to zero or a negative value to avoid the model sacrificing correctness to shorten the number of steps.

[0088] A key characteristic of verifiable reward signals lies in their computability. Accuracy rewards can be objectively verified by examining the spatial relationship between the current position and the target region in the final visual measurement image, without requiring human evaluation. Efficiency rewards are calculated directly from the number of steps recorded by the system. Neither relies on additional models or human feedback, and therefore can be generated automatically at scale.

[0089] Finally, a group-relative policy optimization (PRC) algorithm is used to train the supervised fine-tuned model for reinforcement learning. PRC is a policy gradient method that estimates the dominance function by comparing multiple sampled trajectories within a batch, exhibiting high training stability and good sample efficiency. In this reinforcement learning framework, the model is treated as a policy network, and its action space is a sequence of tokens that generate callback text. The environment calculates a weighted sum of the accuracy and efficiency rewards based on the complete trajectory generated by the model as the final reward. During training, the model adjusts its parameters by maximizing the expected reward, thereby learning to proactively perform effective callbacks in long-range tasks, reducing redundant steps, and improving the final task success rate.

[0090] To mitigate catastrophic forgetting and reduce training costs, a low-rank adaptation parameter fine-tuning technique is introduced during the reinforcement learning phase. Specifically, a low-rank adaptation module is inserted in parallel alongside the query mapping matrix and value mapping matrix of each layer's attention computation module in the multimodal language model. This module contains a dimensionality reduction matrix and an dimensionality increase matrix, compressing the original hidden states to a low-dimensional space before restoring them. During reinforcement learning training, the model's original pre-trained parameters are frozen, and gradient updates are performed only on the parameters in the aforementioned low-rank adaptation module. This preserves the basic capabilities gained during supervised fine-tuning while enabling deeper self-reflection and error correction through reinforcement learning, significantly reducing memory usage and computational overhead.

[0091] Through the two-stage training pipeline described above, the model ultimately acquires the ability to generate accurate callback text based on visual measurements in closed-loop inference. This training design is specifically tailored to the interleaved process of "think-observe-callback," with procedural supervised fine-tuning providing initial behavioral guidelines and reinforcement learning further optimizing the robustness and efficiency of long-field decision-making.

[0092] Furthermore, the verifiable reward signal includes accuracy reward and efficiency reward, and the step of setting a verifiable reward signal for reinforcement learning includes: Based on whether the final state of the task at the end of the current reasoning round has reached the target state, a positive or negative reward is assigned according to the judgment result. A length penalty factor is calculated based on the total length of the action text and callback text generated in the current inference trajectory. When the accuracy reward is determined to be positive, the efficiency reward is set based on the length penalty factor. The length penalty factor is positively correlated with the efficiency reward.

[0093] This embodiment further defines the specific setting method of the verifiable reward signal in the above-mentioned reinforcement learning, that is, how to calculate the accuracy reward and efficiency reward based on the final state of the task and the length of the inference trajectory, respectively. This process includes two parallel reward calculation sub-steps.

[0094] The accuracy reward is determined based on whether the final task state at the end of the current inference round reaches the target state. The final task state is obtained using a visual measurement image. After the inference round ends, the visual measurement image output in the last iteration is examined. The spatial relationship between the current location marker and the target region marker is read from this image. The specific determination method follows the overlap detection logic: if the spatial overlap between the current location marker and the target region marker reaches a preset threshold, the task is considered successful; otherwise, it is considered a failure. This determination is entirely based on the geometric information from deterministic rendering and does not rely on any model judgment, therefore it is objectively verifiable.

[0095] Based on the above judgment results, corresponding reward values ​​are assigned. If the task successfully reaches the target state, a positive reward is given. The specific value of the positive reward can be set to a fixed positive value, such as +1.0. If the task does not reach the target state, for example, if the model fails to reach the target area within the maximum number of steps, or exits midway, a negative reward is given. The value of the negative reward can be set to -1.0. In one embodiment, tiered penalties can also be applied based on the degree of failure; for example, if the model deviates completely from the trajectory, a lower reward value is given, but a binary reward is sufficient in the basic scheme. The accuracy reward described above is sparse, provided only at the end of the entire trajectory, which forces the model to plan a complete sequence of actions to obtain positive feedback.

[0096] The length penalty factor is calculated based on the total length of the action text and callback text generated in the current inference trajectory. The total length can be measured in one of two ways: either by the number of inference steps (the total number of iterations executed from the first to the last round) or by the number of tokens generated (the total number of tokens contained in all action text and callback text). For simplicity, step counting is preferred. Let T be the number of steps in the current trajectory. The length penalty factor can be defined as a function of T, for example, penalty factor = 1 / T, or penalty factor = -α * T, where α is an adjustable hyperparameter. The design goal of the penalty factor is to encourage the model to complete inference with fewer steps while still meeting correctness requirements.

[0097] The efficiency reward setting has an important precondition: it is only set based on the length penalty factor when the accuracy reward is positive. In other words, if the model fails to complete the task, regardless of how short its inference trajectory is, the efficiency reward is zero or ignored. This design prevents the model from prematurely terminating or taking shortcuts to obtain a high efficiency reward, leading to task failure. Specifically, when the accuracy reward is positive, the efficiency reward equals the length penalty factor, and the length penalty factor is positively correlated with the efficiency reward. For example, if the penalty factor is set to 1 / T, then the efficiency reward = 1 / T, meaning fewer steps result in a higher efficiency reward; if the penalty factor is set to -α * T, then the efficiency reward = -α * T, meaning more steps result in a lower efficiency reward (a larger negative value). The latter method is essentially a step-count penalty. Both methods can encourage shorter trajectories. Optionally, the efficiency reward can also be set as a base value minus a step-count penalty.

[0098] In practice, the total length calculation also includes the length of the correction content in the callback text. The model might increase the number of steps by generating lengthy correction instructions, but efficiency rewards can suppress this behavior. A balance between accuracy and efficiency can be achieved by adjusting the hyperparameter α. For example, when α is 0.01, approximately 0.01 of the reward is deducted for each additional step. Compared to the accuracy reward of +1.0, this is still acceptable for the model if it can be completed in a few steps; however, if hundreds of steps are required, the penalty will exceed the positive reward, prompting the model to seek a better strategy.

[0099] The accuracy and efficiency rewards mentioned above together constitute a verifiable multi-objective reward signal. Accuracy rewards ensure the effectiveness of task completion, while efficiency rewards encourage the simplicity of the model's reasoning. This combination ensures that the model, during reinforcement learning training, neither sacrifices correctness for shorter trajectories nor infinitely extends trajectories to fine-tune actions. Key technical means lie in using deterministic rendering results to objectively verify the task completion status and in automatic penalty calculation based on trajectory length; both require no manual intervention, making large-scale reinforcement learning possible. Through this reward design, the model can gradually learn to perform closed-loop interleaved reasoning efficiently and accurately in a long-viewpoint space.

[0100] The step of obtaining the multimodal language model by employing a group-relative policy optimization algorithm and using the verifiable reward signal to train the supervised fine-tuned multimodal language model through reinforcement learning includes: A low-rank adaptation module is inserted in parallel next to the query mapping matrix and value mapping matrix of each layer attention calculation module in the multimodal language model. The low-rank adaptation module contains a dimension reduction matrix and an dimension increase matrix. The dimension reduction matrix compresses the original hidden state to a low-dimensional space, and the dimension increase matrix restores the low-dimensional features to the original dimension. During the training process of the reinforcement learning, the original pre-training parameters of the multimodal language model are frozen, thereby performing gradient updates on the dimensionality reduction matrix parameters and dimensionality increase matrix parameters in the low-rank adaptation module.

[0101] This embodiment further defines the parameter update method in the reinforcement learning training process described above, namely, how to efficiently fine-tune the multimodal language model by using a low-rank adaptation module while freezing the original pre-training parameters. This process includes two core operations: the insertion of the adaptation module and the freezing and updating of training parameters.

[0102] In each layer of the attention computation module of the supervised fine-tuned multimodal language model described above, a low-rank adaptation module is inserted in parallel next to the query mapping matrix and the value mapping matrix. The attention computation module described above is a core component of the Transformer model. The query mapping matrix is ​​responsible for transforming the input hidden state into a query vector, and the value mapping matrix is ​​responsible for transforming it into a value vector. The low-rank adaptation module is a compact, trainable structure that does not change the size or function of the original matrices; instead, it is computed in parallel with them and then summed to output the result.

[0103] Each low-rank adaptation module contains two consecutive linear transformation matrices: a dimension reduction matrix and an dimension increase matrix. Let the original hidden state have dimension *d*, and the dimension reduction matrix have the shape *d × r*, where *r* is much smaller than *d*, typically taking the form of 4, 8, or 16. The dimension increase matrix has the shape *r × d*. For a hidden state vector *h* (of dimension *d*) input to this module, the module's computation process is as follows: first, *h* is compressed to a lower-dimensional space using the dimension reduction matrix, resulting in an intermediate vector of dimension *r*; then, the intermediate vector is restored to the original dimension *d* using the dimension increase matrix. The final output is the sum of the output of the original mapping matrix and the output of the low-rank adaptation module. Since *r* is much smaller than *d*, the number of trainable parameters introduced by the low-rank adaptation module is only 2 × *d × r*, while the number of parameters in the original matrix is ​​*d × d*. When *d* is 4096 and *r* is 8, the number of parameters in the adaptation module is only about 0.4% of that in the original matrix, significantly reducing the training burden.

[0104] The dimensionality reduction and dimensionality increase matrices described above are typically initialized using a random Gaussian distribution, with the dimensionality increase matrix initialized to zero. This ensures that the adapter module outputs zero at the start of training, without affecting the initial behavior of the original model. This parallel insertion method allows low-rank adapter modules to be flexibly added to the pre-trained model without modifying the original network structure.

[0105] In the reinforcement learning training process based on group-relative policy optimization described above, the system freezes all original pre-trained parameters of the multimodal language model, preventing them from receiving gradient updates during backpropagation. That is, the model's original attention matrix, feedforward network, layer normalization, and other parameters remain fixed and do not participate in training. Simultaneously, normal gradient updates are performed on the dimensionality reduction and elevation matrix parameters in all the inserted low-rank adaptation modules. Specifically, the optimizer only adds the parameters of the low-rank adaptation modules to the parameter set, calculates the gradient of the loss function with respect to these parameters, and performs parameter updates. Although the original parameters participate in the forward computation, they are marked as `requires_grad=False`, do not generate gradients, and are not updated by the optimizer.

[0106] The benefits of this freeze update are multifaceted. First, it significantly reduces GPU memory usage: there's no need to store the gradients of the original parameters, nor to retain the momentum information of the original parameters for the optimizer. This allows for reinforcement learning training of large models even with limited hardware resources (e.g., a single consumer-grade graphics card with 24GB of VRAM). Second, it prevents catastrophic forgetting: because the original pre-trained parameters never change, the model's original language and visual understanding capabilities are fully preserved, and the low-rank adaptation module simply adds fine-tuning for the "think-observe-callback" task on top of the original capabilities. Third, it speeds up training: the number of parameters that need updating is minimal, and the computational and communication costs per iteration are significantly reduced.

[0107] The specific implementation process is as follows: First, load the complete model parameters after supervised fine-tuning. Then, iterate through all attention layers of the model, creating a corresponding low-rank adaptor module for each query mapping matrix and value mapping matrix and inserting it next to the original computation graph. Set the `requires_grad` attribute of the original parameters to `False`, and set the `requires_grad` attribute of the low-rank adaptor module parameters to `True`. Then, start a training loop based on group-relative policy optimization, sampling trajectories and calculating the loss in each batch. During backpropagation, only the gradients of the low-rank adaptor modules are updated. After training, the parameters of the low-rank adaptor modules can be merged and saved with the original model parameters, or they can be saved separately for dynamic loading during deployment.

[0108] The above constraints clarify the efficient fine-tuning of parameters during reinforcement learning training. This approach, while maintaining the original model's capabilities, teaches the model to make better callback decisions in closed-loop inference with extremely low computational cost. Key technical means lie in low-rank decomposition and parallel structures, enabling the model to adapt to long-field-of-view spatial inference tasks with a small number of trainable parameters without introducing generative noise.

[0109] Furthermore, you can also view Figure 4 , Figure 4 This is a detailed process diagram based on step S50 in the first embodiment. Figure 4 The step of determining whether the current task state meets the termination condition based on the visual measurement image and the callback text, and ending the inference if the condition is met, includes S51~52: Step S51: Read the spatial overlap state between the current location marker and the target area marker from the visual measurement image, or extract the task completion identifier from the callback text; Step S52: If the spatial overlap state reaches a preset threshold or the task completion flag is true, then the current task state is determined to meet the termination condition.

[0110] This embodiment further defines the specific execution method of the above termination condition determination, namely, how to determine whether the task should end based on the visual measurement image and callback text. This process includes two core sub-steps: reading the status information and logically determining the termination condition.

[0111] First, the information needed for termination judgment is extracted from two independent data sources. The first data source is a visual measurement image. The spatial overlap state between the current location marker and the target region marker is read from this image. The current location marker described above is a graphical primitive rendered on the image, such as a red dot or a crosshair. The target region marker is also a graphical primitive, typically rendered as a green semi-transparent rectangle or a closed boundary outline. The spatial overlap state is read without relying on any machine learning model, but rather through pure image analysis or by directly utilizing the drawing records of the rendering function. In practice, an efficient method is as follows: since the rendering function records the coordinate position and geometry of each primitive when drawing it, the system can directly obtain the center coordinates of the current location marker and the boundary coordinates of the target region marker from these records, and then calculate whether the two regions intersect or contain each other. For example, if the current location marker is a dot with a radius of 4 pixels, and the target region marker is a rectangle from (400, 300) to (500, 400), it is only necessary to determine whether the center of the dot falls inside the rectangle, or whether the boundary of the dot intersects with the rectangle. This rendering-based recording method avoids post-processing image recognition, is extremely fast, and is 100% accurate.

[0112] The second data source is the callback text. Task completion identifiers are extracted from the generated callback text. These identifiers can be a set of predefined keywords, such as "task completed," "goal achieved," or "successfully completed." They can also be special markers output by the model according to a fixed format, such as "status:completed." Extraction is performed using string matching or regular expressions. For example, if the callback text reads "Confirmation: Action valid, target area reached, task completed," searching for the keyword "task completed" will determine it as true. These two data sources can be used individually or in combination. In some tasks, visual measurements are sufficient to determine completion status, such as in maze navigation where the current point falling into the target area signifies completion. In other, more semantically complex tasks, the model's textual declarations may be more direct.

[0113] Based on the acquired spatial overlap status and task completion flag, a termination condition is logically determined. The determination rule uses OR logic: the current task status is considered to meet the termination condition as long as the spatial overlap status reaches a preset threshold or the task completion flag is true. The preset threshold can be configured according to task requirements. For the overlap between a point and a rectangle, the threshold can be set so that the center coordinates of the point are completely inside the rectangle; for the overlap between a point and an irregular region, the threshold can be set so that the intersection area of ​​the point and the region is greater than zero. The thresholds mentioned above can also include a tolerance parameter, such as considering completion when the distance between the current point and the boundary of the target region is less than 5 pixels, to adapt to the requirements of rendering precision or task leniency.

[0114] The specific decision-making process is as follows: First, determine the spatial overlap state. If the overlap state reaches a threshold, return true directly without further checking the callback text. If the overlap state does not reach the threshold, continue checking the task completion flag in the callback text. If the completion flag is true, return true; otherwise, return false. This priority design is based on the objectivity of visual measurement taking precedence over the subjective declaration of model semantics. However, to prevent situations where the model has completed the task ahead of time but the visual measurement fails to reflect it in time due to rendering delays, a two-way confirmation AND logic can also be used, i.e., termination only occurs when both are true. This application does not limit the fixed decision logic but allows for flexible configuration based on task characteristics.

[0115] Based on the above limitations, a reliable and efficient termination condition determination mechanism is provided. This mechanism combines quantitative visual measurement with qualitative semantic judgment, utilizing the accuracy of deterministic rendering while retaining the flexibility of language models in handling complex task completion conditions. The key technical approach lies in directly reading the overlapping state from the rendering record, avoiding additional calls to the visual understanding model, while retaining semantic completion markers in the callback text as alternatives or supplements.

[0116] Furthermore, you can also view Figure 5 , Figure 5 This is a flowchart illustrating the second embodiment of the long-field-of-view spatial long-field-of-view visual measurement feedback closed-loop staggered reasoning method of this application. The long-field-of-view visual measurement feedback closed-loop staggered reasoning method further includes steps S60~S80: Step S60: In the next inference round, the visual measurement image is used as the updated current working image; Step S70: Use the updated current working image as the input base map of the rendering function, so that the newly generated visual measurement image inherits the graphic primitives superimposed in all previous rounds; Step S80: The updated historical callback text and the original text instruction are used together as text conditions for generating the next round of action text, so that the multimodal language model can adjust subsequent actions based on historical correction information.

[0117] This embodiment further defines the specific implementation of the above iterative update process, namely, how to pass the visual measurement image and callback text of the current round to the next inference round, and ensure that the model can utilize accumulated visual information and historical correction experience in subsequent steps. This process includes two core update operations.

[0118] First, at the start of the next inference round, the current working image updated at the end of the previous round is used as the input base map for the rendering function. The current working image described above was replaced with the generated visual measurement image in the previous round. This image already has all the graphic primitives drawn based on the structured state variables in the previous round overlaid on it, such as dot markers, polylines, and directional arrows. When passed as the input base map to the rendering function in this round, the rendering function continues to overlay new graphic primitives on top of this base map, rather than redrawing from the original input image. This design allows the newly generated visual measurement image to inherit the graphic primitives overlaid in all previous rounds.

[0119] The specific implementation of the above inheritance mechanism is as follows: Each time the rendering function is called, the base map parameter received is the image rendered in the previous round, not the original image. The rendering function first creates a copy of this base map, then iterates through the drawing instruction list for the current round, drawing new graphic primitives onto the copy. The final output visual measurement image contains the cumulative sum of all graphic primitives from the first round to the current round. For example, the first round draws the starting point marker, the second round draws the line segment from the starting point to the first turning point, and the third round draws the line segment from the first turning point to the second turning point. At the end of the third round, the output visual measurement image simultaneously contains the starting point marker, the first line segment, and the second line segment, fully displaying the entire movement trajectory. This cumulative drawing method avoids repeatedly drawing historical paths, ensuring the visual continuity of the reasoning process.

[0120] The key technical approach in this step lies in using the current working image as the recursive input base map to form a visual state accumulator. Unlike conventional methods that only display the latest state at a time, this application allows the model to "see" the complete operation history through cumulative rendering. When the model observes the visual measurement image in subsequent rounds, it can intuitively identify which areas have been explored and its current position on the path, thus making more informed decisions.

[0121] Simultaneously, the updated historical callback text, along with the original text instruction, serves as the text condition for generating the action text for the next round. The historical callback text is stored as a cumulative list, containing corrective or confirmatory callbacks for all model outputs from the first round to the previous round. In the next round, the text input received by the multimodal language model is composed of two parts: the original text instruction remains unchanged and is located at the beginning of the sequence; the historical callback text is appended sequentially in chronological order.

[0122] The historical callback text described above provides the model with a complete error-correction memory. For example, in the second round, the model output a corrective callback "Correction: Actual current point is (160, 145)", and this information is added to the historical callback text. In the third round, the model will see this record when generating action text, thus knowing that a coordinate deviation has occurred before, and proactively compensating when planning new actions to avoid repeating the same mistake. This design gives the model a "learning" ability: it not only makes decisions based on the current visual state, but also adjusts its behavior based on past corrective experiences.

[0123] This data processing step involves passing semantic-level feedback memory and visual-level accumulated states to the model in parallel, complementing visual inheritance. Together, they form a dual closed loop: the visual loop provides spatial continuity through accumulated rendering, while the semantic loop provides correctability in inference through accumulated callbacks. In practice, the accumulation method for historical callback texts requires careful length control. When the number of inference steps is large, the text sequence may become too long, exceeding the model's context window. To address this, a sliding window strategy can be introduced, retaining only the callback text from the most recent K steps, or summarizing and compressing older callback texts. However, the core inheritance mechanism remains unchanged: each new round can access the correction information from previous rounds, allowing for more accurate adjustments.

[0124] The above constraints clarify two key inheritance relationships in the iterative process. Visual inheritance ensures the model sees the complete historical trajectory, while semantic inheritance ensures the model understands past errors and corrections. Working together, these two mechanisms prevent the accumulation of errors in long-vision inference, as any single-step error is recorded and corrected in subsequent rounds. This mechanism does not rely on any external tools or additional training; robust closed-loop control is achieved solely through a clever input update strategy.

[0125] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the long-field visual measurement feedback closed-loop interleaved reasoning method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0126] This application provides a long-field-of-view visual measurement feedback closed-loop interleaved inference device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the long-field-of-view visual measurement feedback closed-loop interleaved inference method in the above embodiment 1.

[0127] The following is for reference. Figure 7 This document illustrates a structural schematic diagram of a long-field-of-view visual measurement feedback closed-loop interleaved inference device suitable for implementing embodiments of this application. The long-field-of-view visual measurement feedback closed-loop interleaved inference device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The long-field-of-view visual measurement feedback closed-loop interleaved inference device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0128] like Figure 7As shown, the long-field-of-view visual measurement feedback closed-loop interleaved inference device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the long-field-of-view visual measurement feedback closed-loop interleaved inference device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the long-field-of-view visual measurement feedback closed-loop interleaved inference device to wirelessly or wiredly communicate with other devices to exchange data. Although various long-field-of-view visual measurement feedback closed-loop interleaved inference devices are shown in the figures, it should be understood that it is not required to implement or possess all of them. More or fewer may be implemented alternatively.

[0129] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0130] The long-field-of-view visual measurement feedback closed-loop interleaved inference device provided in this application employs the long-field-of-view visual measurement feedback closed-loop interleaved inference method in the above embodiments. This solves the technical problem of existing visual language models failing in multi-step inference due to the lack of closed-loop feedback strictly aligned with the real visual state, leading to the accumulation of intermediate deviations and resulting in trajectory-level task failure. Compared with the prior art, the beneficial effects of the long-field-of-view visual measurement feedback closed-loop interleaved inference device provided in this application are the same as those of the long-field-of-view visual measurement feedback closed-loop interleaved inference method provided in the above embodiments. Furthermore, other technical features of this long-field-of-view visual measurement feedback closed-loop interleaved inference device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0131] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0133] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the long field-of-view visual measurement feedback closed-loop interleaved inference method in the above embodiments.

[0134] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be executed by instructions, used by devices, or used in conjunction with them. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0135] The aforementioned computer-readable storage medium may be included in the long-field-of-view visual measurement feedback closed-loop interleaved inference device; or it may exist independently and not be assembled into the long-field-of-view visual measurement feedback closed-loop interleaved inference device.

[0136] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the long-field-of-view visual measurement feedback closed-loop interleaved inference device, enable the long-field-of-view visual measurement feedback closed-loop interleaved inference device to implement the technical content of the long-field-of-view visual measurement feedback closed-loop interleaved inference method embodiment as shown above.

[0137] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based implementations that perform the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0139] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0140] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned long-field-of-view visual measurement feedback closed-loop interleaved inference method. This solves the technical problem that existing visual language models, in multi-step inference, suffer from trajectory-level task failure due to the accumulation of intermediate deviations caused by the lack of closed-loop feedback strictly aligned with the actual visual state. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the long-field-of-view visual measurement feedback closed-loop interleaved inference method provided in the above embodiments, and will not be elaborated upon here.

Claims

1. A long-field-of-view visual measurement feedback closed-loop interleaved reasoning method, characterized in that, The long-field-of-view visual measurement feedback closed-loop interleaved inference method includes the following steps: Set the original input image as the current working image for the first reasoning round; Based on the current working image, the original text command, and the historical callback text, an action text is generated, wherein the historical callback text is initialized to empty before the action text is generated; The structured state variables contained in the action text and the current working image are used as inputs to the rendering function, which generates a visual measurement image superimposed with graphic primitives. The callback text is generated using the visual measurement image and the action text. Based on the visual measurement image and the callback text, determine whether the current task status meets the termination condition; if it does, end the reasoning.

2. The long-field-of-view visual measurement feedback closed-loop staggered reasoning method as described in claim 1, characterized in that, The step of using the structured state variables contained in the action text and the current working image as input to a rendering function, and generating a visual measurement image superimposed with graphic primitives through the rendering function, includes: Based on the spatial geometric information carried in the structured state variables, graphic primitives matching the type of spatial geometric information are superimposed on the corresponding positions of the current working image, wherein the graphic primitives are selected from one or more of point markers, line segments, polylines, directional arrows, bounding boxes, or trajectory curves; The graphic primitives are drawn on the current working image using a deterministic algorithm to generate the visual measurement image.

3. The long-field-of-view visual measurement feedback closed-loop staggered reasoning method as described in claim 1, characterized in that, The step of generating callback text using the visual measurement image and the action text includes: The visual measurement image and the action text are input together into a multimodal language model; The multimodal language model is used to detect whether the graphic primitives in the visual measurement image are consistent with the expected spatial relationship declared in the action text; If they match, output a callback text confirming the validity of the action text; If there is a discrepancy, output a callback text containing corrective information about the structured state variables.

4. The long-field-of-view visual measurement feedback closed-loop staggered reasoning method as described in claim 3, characterized in that, Before the step of inputting the visual measurement image and the action text together into the multimodal language model, the method further includes: Action text, visual measurement images, and corresponding callback text are used as supervisory data to perform supervised fine-tuning of the initial multimodal language model; To reinforce learning, verifiable reward signals are set, including accuracy rewards and efficiency rewards; The multimodal language model is obtained by using a group-relative policy optimization algorithm and the verifiable reward signal to train the supervised fine-tuned multimodal language model through reinforcement learning.

5. The long-field-of-view visual measurement feedback closed-loop interleaved reasoning method as described in claim 4, characterized in that, The verifiable reward signal includes accuracy reward and efficiency reward, and the step of setting a verifiable reward signal for reinforcement learning includes: Based on whether the final state of the task at the end of the current reasoning round has reached the target state, a positive or negative reward is assigned according to the judgment result. A length penalty factor is calculated based on the total length of the action text and callback text generated in the current inference trajectory. When the accuracy reward is determined to be positive, the efficiency reward is set based on the length penalty factor. The length penalty factor is positively correlated with the efficiency reward.

6. The long-field-of-view visual measurement feedback closed-loop staggered reasoning method as described in claim 4, characterized in that, The step of using a group-relative policy optimization algorithm and the verifiable reward signal to train the supervised fine-tuned multimodal language model through reinforcement learning to obtain the multimodal language model includes: A low-rank adaptation module is inserted in parallel next to the query mapping matrix and value mapping matrix of each layer attention calculation module in the multimodal language model. The low-rank adaptation module contains a dimension reduction matrix and an dimension increase matrix. The dimension reduction matrix compresses the original hidden state to a low-dimensional space, and the dimension increase matrix restores the low-dimensional features to the original dimension. During the training process of the reinforcement learning, the original pre-training parameters of the multimodal language model are frozen, thereby performing gradient updates on the dimensionality reduction matrix parameters and dimensionality increase matrix parameters in the low-rank adaptation module.

7. The long-field-of-view visual measurement feedback closed-loop staggered reasoning method as described in claim 1, characterized in that, The step of determining whether the current task state meets the termination condition based on the visual measurement image and the callback text, and ending the inference if the condition is met, includes: Read the spatial overlap state between the current location marker and the target area marker from the visual measurement image, or extract the task completion identifier from the callback text; If the spatial overlap state reaches a preset threshold or the task completion flag is true, then the current task state is determined to meet the termination condition.

8. The long-field-of-view visual measurement feedback closed-loop staggered reasoning method as described in claim 1, characterized in that, The long-field-of-view visual measurement feedback closed-loop interleaved inference method further includes: In the next inference round, the visual measurement image is used as the updated current working image; The updated current working image is used as the input base map for the rendering function, so that the newly generated visual measurement image inherits the graphic primitives superimposed in all previous rounds; The updated historical callback text and the original text instruction are used together as text conditions for generating the next round's action text, enabling the multimodal language model to adjust subsequent actions based on historical correction information.

9. A long-field-of-view visual measurement feedback closed-loop interleaved reasoning device, characterized in that, The long field of view visual measurement feedback closed-loop interleaved inference device stores a computer program, which, when executed by a processor, implements the long field of view visual measurement feedback closed-loop interleaved inference method according to any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the long-field-of-view visual measurement feedback closed-loop interleaved inference method according to any one of claims 1-8.

Images