A Visual Autonomous Localization and Navigation Method and System for Large-Scale Visual Language Action Models

By introducing visual anchor descriptors and matchers into the large-scale visual language action model, and combining them with a three-level drift hierarchical response mechanism, the semantic drift problem in long-range autonomous navigation is solved, the navigation success rate is improved, and efficient matching of visual scenes and language commands is achieved.

CN122799403BActive Publication Date: 2026-11-17ZHEJIANG KECONG CONTROL TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611257815.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-19
Publication Date
2026-11-17
Estimated Expiration
2046-08-19

AI Technical Summary

Technical Problem

Existing large-scale visual-language-action models suffer from semantic drift in long-range autonomous localization and navigation. When performing long-range tasks, robots cannot effectively maintain the matching between visual scenes and language commands, resulting in a decrease in navigation accuracy.

Method used

By introducing a sequence of visual anchor descriptors into the large visual language action model and using a visual language matcher for literal verification, combined with a three-level drift graded response mechanism, the robot's motion control is corrected in real time to ensure the matching between the visual scene and the language instructions.

Benefits of technology

It significantly reduces semantic drift accumulation in long-range tasks, improves navigation success rate from 62.4% to 87.1%, and does not require modification or retraining of the visual language action model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122799403B_ABST
    Figure CN122799403B_ABST
Patent Text Reader

Abstract

This invention discloses a visual autonomous localization and navigation method and system for visual language action models. Addressing the technical problem of semantic drift accumulating due to the failure to simultaneously output corresponding visual verification conditions when outputting action intentions in long-range, multi-step navigation, this invention enables the visual language action model to simultaneously output a sequence of visual anchor descriptors while outputting action intentions. During the underlying execution phase, a pre-trained comparative language image matching model is used to calculate literal matching scores. After categorizing these into hard drift, soft drift, and no drift types, three responses are executed: callback large model replanning, visual odometry fusion correction, and continued execution. This improves the success rate in long-range navigation tasks from a baseline of 62.4% to 87.1%.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of visual autonomous positioning and navigation and embodied intelligence, specifically to a method and system for visual autonomous positioning and navigation of service robots that uses a large visual language action model as a high-level decision-maker. Background Technology

[0002] Visual autonomous localization and navigation is a core technology for intelligent agents such as service robots, home robots, and inspection robots to autonomously reach their commanded target locations in unknown or semi-unknown indoor and outdoor environments. Its engineering implementation has long relied on geometric methods such as real-time localization and mapping (RTL) technology, visual inertial odometry (VIO), and bag-of-words-based visual relocalization. This involves pre-collecting environmental feature points to construct a dense map before performing real-time pose tracking. Traditional geometric methods can achieve centimeter-level positioning accuracy in strictly static, textured, and stable lighting scenarios. However, they depend on pre-built maps and manually labeled target coordinates. For user commands in natural language form, such as "go to the kitchen to get a glass of water" or "wait for me outside the red door," they cannot bridge the semantic gap between natural language and geometric coordinates. Therefore, in consumer-facing service robot products, the deployment cost of manually pre-recorded target points has long been a constraint.

[0003] In recent years, the rise of visual language models and large-scale visual language action models has provided new solutions to the aforementioned semantic gap. These include open-vocabulary visual language navigation networks, target reasoning navigation based on large language models, and end-to-end large-scale models integrating visual, language, and action modalities, such as open visual language action models and similar follow-up work. This mainstream approach compresses three originally independent functions—natural language instruction understanding, visual feature extraction, and action intent generation—into a single large-scale model for end-to-end reasoning. This allows robots to directly receive natural language instructions and output underlying action intents without the need for pre-mapping or manual annotation of target points. Large-scale visual language action models, after pre-training on large-scale teleoperation demonstration datasets, only require minor fine-tuning on the target robot itself before deployment, significantly reducing deployment costs. This approach is widely recognized in the industry as the next-generation core architecture for consumer service robots.

[0004] However, visual-language-action large models exhibit a phenomenon in long-range, multi-step navigation tasks that has been repeatedly observed in the industry but has long remained unengineered: when a robot receives a long-range instruction such as "go to the kitchen to get a glass of water," which requires 8 to 15 consecutive decisions to complete, the robot often "forgets where it is" in the middle of the journey. For the first few meters, the robot can clearly move towards the kitchen, but after walking a dozen meters, it faces an unfamiliar wall or a furniture arrangement it has never seen in the training data, and begins to circle, pause, or choose the wrong direction. This is like a visitor who has only heard a verbal description once trying to find a meeting room in a multi-story office building: for the first few steps, it remembers "take the elevator to the third floor and turn right," but after getting off the elevator and walking through two similar corridors, the anchor of the verbal instruction in the visual scene is lost, and it continues to stray further and further from the goal. Although some existing methods attempt to alleviate the above phenomenon by periodically replaying the original instruction, expanding the context window, and introducing additional historical frame memory, they still cannot solve the core problem.

[0005] The root cause of the aforementioned long-range semantic drift phenomenon lies in the fact that existing methods only concern themselves with how the visual-language-action model outputs the target pose and action intent, but neglect whether the visual-language-action model can simultaneously output the corresponding visual verification condition when outputting the intent. The visual-language-action model itself has powerful language generation capabilities and could have simultaneously output a natural language-form visual verification condition, "After moving to point A, there should be a white door in front, with the doorknob on the right," when outputting the action intent "move forward to point A." However, the reasoning process of the existing visual-language-action model lacks this output channel, causing the underlying motion controller to blindly trust the accumulated odometer value during execution and failing to perform visual verification. Once the robot deviates from the target position due to any reason such as slipping on the ground, visual similarity deception, or sensor drift, this deviation is silently swallowed by the visual-language-action model when taking the next frame. The robot still continues to output the next intent based on the assumption that "the target position has been reached," and semantic drift accumulates step by step. In the fields of cognitive science and human-computer interaction, it is a well-known fact that people continuously make literal comparisons between "expected visual scenes and actual visual scenes" while walking. However, this a priori fact has not been explicitly mapped into the engineering criteria for autonomous positioning and navigation systems based on large-scale visual language action models for a long time.

[0006] To address the long-range semantic drift problem in the autonomous localization and navigation of large visual language action models, there is an urgent need for a visual autonomous localization and navigation method for large visual language action models that does not modify the main network structure of the large visual language action model, does not require retraining or fine-tuning of the large model, and can run in real time on an embedded intelligent robot platform. Summary of the Invention

[0007] This invention provides a visual autonomous localization and navigation method for large-scale visual language action models to solve the aforementioned long-range semantic drift problem. The method operates according to the following steps: S0, receiving the user's natural language navigation command and the current field of view image, and calling the visual language action model to generate an action intent and a sequence of visual anchor descriptors; the sequence of visual anchor descriptors contains 2 to 5 visual anchor descriptors, each of which is a natural language literal description of the expected field of view feature object; S1, the underlying motion controller executes the action intent, during which the current real-shot image is acquired at a preset visual acquisition frame rate; S2, for each current real-shot image and each visual anchor descriptor, the visual language matcher is called to output a visual language matching score with a value in [0, 1], and all visual language matching scores constitute an anchor matching score set; S3, the anchor matching score set is classified according to three categories in the order of hard drift, soft drift, and no drift to obtain an anchor drift grade, which takes values ​​of 2, 1, and 0 respectively; S4, responding according to the anchor drift grade: when the value is 0, continue execution; when the value is 1, continue execution after visual odometry fusion correction; when the value is 2... The callback visual language action big model regenerates the next action intent and the next visual anchor descriptor sequence.

[0008] The visual anchor descriptor is a natural language literal description of the expected visual field feature object during the execution of the action intention, and has two properties: it is a readable string and it can be literally verified by a visual language matcher.

[0009] The visual anchor descriptor sequence refers to an ordered set of multiple visual anchor descriptors output by the large model decision for the same visual language action. The sequence length is preset by the user, typically ranging from 2 to 5.

[0010] The visual language matcher refers to a lightweight contrastive language image pre-trained matching model that receives a real-world image and a natural language description, and outputs a visual language matching score with values ​​in the range [0, 1].

[0011] The visual language matching score refers to the literal matching score of the visual language matcher for a single anchor point and a single real-world image. The closer the value is to 1, the better the visual scene matches the natural language description; the closer the value is to 0, the worse the match.

[0012] The anchor point drift classification refers to the result of classifying the anchor point matching score set into three levels, with values ​​of 0, 1, and 2, corresponding to the three categories of no drift, soft drift, and hard drift, respectively.

[0013] The preset high-level anchor point matching threshold refers to the upper limit of the visual language matching score preset by the user, typically 0.7.

[0014] The preset low-position anchor point matching threshold refers to the lower limit of the visual language matching score preset by the user, typically 0.4.

[0015] The preset failure rate threshold refers to the upper limit of the proportion of the number of times the matching score within the sliding window is lower than the preset low anchor point matching threshold to the total number of verifications, typically set to 0.3.

[0016] The preset sliding window length refers to the window length based on the statistical count of the most recent visual language matching events, typically set to 10 times.

[0017] The preset visual acquisition frame rate refers to the fixed frequency at which the underlying motion controller acquires the current real-time image during the execution of the action intention, typically 10 frames per second.

[0018] The action intent refers to the target pose or desired trajectory tracked by the underlying motion controller, which is output by the visual language action big model at each decision.

[0019] The underlying motion controller refers to a position-velocity-acceleration three-loop controller that performs high-frequency tracking of the motion intent, with a typical operating frequency of no less than 50 times per second.

[0020] The visual odometry fusion correction refers to the action of correcting the odometry cumulative pose using the visual features of the current real-shot image. Its specific sub-steps include four items: local feature point extraction, nearest neighbor matching, epipolar geometric pose increment estimation, and increment replacement.

[0021] The historical action summary refers to a three-field structured record consisting of the cumulative action intent number, cumulative travel distance, and the most recent hard drift trigger pose since the method was started.

[0022] The beneficial effects of this invention include: First, by synchronously outputting visual anchor descriptors and performing literal verification, the open-loop action generation of the visual language action model is transformed into autonomous localization and navigation with closed-loop verification using visual anchor literal evidence, significantly reducing semantic drift accumulation in long-range tasks; Second, literal verification relies on a pre-trained contrastive language image matching model, without modifying the main network structure of the visual language action model, allowing for deployment with zero training cost; Third, the three-level graded response triggers only local visual odometry fusion correction without re-invoking the visual language action model when there are slight visual deviations, greatly saving high-level inference resources; Fourth, on the visual language action model with the NaVILA-8B backbone structure, the success rate of a typical 30-meter long-range navigation task is increased from 62.4% to 87.1%. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the method described in this invention.

[0024] Figure 2 This is a schematic diagram of the module structure of the system described in this invention.

[0025] Figure 3 This is a schematic diagram showing the comparison between the visual anchor descriptor sequence and the visual language matching score.

[0026] Figure 4 This is a schematic diagram illustrating the hierarchical classification of the preset high-level anchor matching threshold and the preset low-level anchor matching threshold on the visual language matching score axis.

[0027] Figure 5 This is a schematic diagram of a decision tree for the three-level classification criteria of anchor point drift.

[0028] Figure 6 This is a schematic diagram of the visual odometry fusion correction sub-step process.

[0029] Figure 7 This is a schematic diagram illustrating the deployment of a service robot to fetch water from the kitchen, as an example.

[0030] Figure 8 This is a schematic diagram of the deployment of a warehouse inspection robot in a long-range inspection scenario, as shown in Example 2.

[0031] Figure 9 This is a schematic diagram comparing the long-range navigation success rates of the method described in this invention with the baseline scheme.

[0032] Figure 10 This is a heatmap showing the success rate under different combinations of preset high-level anchor point matching thresholds and preset low-level anchor point matching thresholds.

[0033] Figure 11 This is a schematic diagram comparing the single inference delay of the visual language matcher and the response frequency of the underlying motion controller in the method described in this invention on two axes. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention. The specific threshold values, number of dimensions, time window lengths, sampling times, etc., used in the following embodiments are illustrative examples and do not constitute a limitation on the scope of protection of this invention; those skilled in the art can adjust the corresponding parameters according to actual application scenarios, and the adjusted implementation methods still fall within the scope of protection of this invention. The following specific embodiments are described in the following order: First, combined with... Figure 1 The overall flow of the method described in this invention is summarized, and then combined with... Figure 2 The modular structure of the system described in this invention will be introduced, and then combined with... Figures 3 to 6The core terms and underlying steps of the core algorithm involved in this invention are explained in detail. Then, four specific embodiments and two non-triggering control branches are presented in sequence. Finally, a general implementation overview of each step of the method, experimental verification and performance comparison, boundary cases and exception handling are given.

[0035] Overall process overview. Figure 1 As shown, the overall process of the method described in this invention includes the following five main steps: receiving the user's natural language navigation instructions and the current field of view image; calling the visual language action model to generate action intentions and visual anchor point descriptor sequences; the underlying motion controller executing the action intentions and periodically acquiring the current real-world images; the visual language matcher calculating the visual language matching score for each current real-world image and each visual anchor point descriptor; classifying the anchor point matching score set according to the three categories of hard drift, soft drift, and no drift to obtain the anchor point drift classification, and executing the corresponding response according to the value of the anchor point drift classification.

[0036] System module structure introduction. For example... Figure 2 As shown, the system of the present invention includes five modules: an instruction receiving and initial planning module, a low-level execution and image acquisition module, a visual language literal matching module, an anchor point drift three-level hierarchical module, and a hierarchical response execution module. The five modules are connected sequentially according to the data flow to form a closed-loop execution chain of "instruction input to hierarchical response". Among them, the instruction receiving and initial planning module is responsible for the encapsulation and interface of the visual language action model; the low-level execution and image acquisition module is responsible for the encapsulation of the low-level motion controller and the synchronous acquisition of the visual sensor; the visual language literal matching module is responsible for the encapsulation of the visual language matcher and the batch calculation of matching scores; the anchor point drift three-level hierarchical module is responsible for the three-level classification and sliding window statistics; and the hierarchical response execution module is responsible for the execution of the state machine of the three-level corresponding response.

[0037] A diagram illustrating the comparison between visual anchor descriptor sequences and visual language matching scores. For example... Figure 3 As shown, the visual anchor descriptor sequence of the present invention includes three visual anchor descriptors in a specific embodiment: "A white door should be visible in front, with the doorknob on the right side," "A brown wooden dining table should be visible to the left front, with a blue tablecloth on the table," and "A silver-white refrigerator should be visible to the right front, with a green potted plant on top of the refrigerator." Each descriptor and the current real-shot image at the same time are output by the visual language matcher with a visual language matching score ranging from [0, 1]. The three scores constitute the anchor matching score set. The closer the visual language matching score is to 1, the better the visual scene matches the natural language description; the closer the score is to 0, the worse the match.

[0038] The preset high-level anchor matching threshold and the preset low-level anchor matching threshold are graded on the visual-language matching score axis. For example... Figure 4 As shown, the visual language matching score axis, ranging from 0 to 1, is divided into three segments by a preset high-level anchor matching threshold and a preset low-level anchor matching threshold: segments with values ​​not less than the preset high-level anchor matching threshold correspond to reliable visual matching; values ​​in the range of (preset low-level anchor matching threshold, preset high-level anchor matching threshold) correspond to slight visual deviation; and segments with values ​​lower than the preset low-level anchor matching threshold correspond to significant visual drift. A typical value combination is a preset high-level anchor matching threshold of 0.7 and a preset low-level anchor matching threshold of 0.4.

[0039] Anchor point drift three-level hierarchical decision tree. For example... Figure 5 As shown, the decision order of the three-level anchor point drift criterion decision tree of the present invention is as follows: First, check whether any visual language matching score in the anchor point matching score set is lower than a preset low-level anchor point matching threshold, or whether the proportion of times it is lower than the threshold within the most recent preset sliding window length is not less than a preset failure rate threshold. If either condition is met, it is classified as a hard drift, and the anchor point drift grading value is 2. Otherwise, check whether there is at least one visual language matching score in the anchor point matching score set that is in the range of (preset low-level anchor point matching threshold, preset high-level anchor point matching threshold). If it is met, it is classified as a soft drift, and the anchor point drift grading value is 1. The remaining cases are classified as no drift, and the anchor point drift grading value is 0. The order of the three categories avoids the ambiguity of satisfying two categories at the same time in certain boundary cases. For example, when the anchor point matching score set contains both scores lower than the preset low-level anchor point matching threshold and scores in the open interval, it is determined as hard drift rather than soft drift according to the order.

[0040] Visual odometry fusion correction sub-step flow. (e.g.) Figure 6 As shown, the visual odometry fusion correction specifically includes the following sub-steps: First, local feature point descriptors are extracted from the current real-world image. These local feature point descriptors can be selected from one of three types: histogram of orientation gradients, acceleration segment test corner feature descriptors, and binary robust independent primitive feature descriptors. Second, the local feature point descriptors are matched with the local feature point descriptors of the previous keyframe to obtain a set of matching point pairs. Third, the inter-frame relative pose increment is estimated based on epipolar geometry constraints for the set of matching point pairs, typically using an eight-point or five-point method to solve the essential matrix. Finally, the inter-frame relative pose increment replaces the accumulated odometry pose increment within the corresponding time interval to obtain the corrected pose.

[0041] The core terminology is explained in detail. The visual anchor descriptor has a dual meaning: the first meaning is a literal string, that is, a natural language description of the expected visual field feature object; the second meaning is a literal verification target, that is, the reference point used by the visual language matcher to calculate the matching score between the real image and the descriptor. The visual anchor descriptor is like a tour guide telling a tourist, "There should be a white pavilion ahead." The tourist walks and compares it with the real scene. If the white pavilion is not seen, they stop and ask the tour guide. In this invention, the visual language action model plays the role of the tour guide, the visual language matcher plays the role of the tourist's eyes, and the robot plays the role of the tourist. Each of the aforementioned visual anchor descriptors consists of four fields in its data structure: anchor object name field, relative orientation field, relative distance field, and significant attribute field; taking "There should be a white door ahead, with the doorknob on the right" as an example, the anchor object name field is "door", the relative orientation field is "ahead", the relative distance field is not explicitly stated and is taken according to the distance implied by the action intention, and the significant attribute field is the color attribute "white" and the position attribute "doorknob on the right". If the visual anchor descriptor output by the visual language action big model is incorrect, for example, it should describe "white door" but is described as "red door", the error will be reflected in the visual language matching module as a low visual language matching score for the descriptor. This will trigger the anchor drift classification as soft drift or hard drift, which will then be corrected by the classification response execution module in the local visual odometry fusion correction or the callback visual language action big model replanning.

[0042] The visual language matcher is implemented using a pre-trained contrastive language image matching model, typically selected from one of the following three types: a ViT-B / 32 backbone structure contrastive language image pre-trained matching model with approximately 151 megabytes of parameters and a single inference latency of approximately 12 milliseconds; a ViT-L / 14 backbone structure contrastive language image pre-trained matching model with approximately 428 megabytes of parameters and a single inference latency of approximately 28 milliseconds; or a language image pre-trained matching model using Sigmoid loss with approximately 380 megabytes of parameters and a single inference latency of approximately 22 milliseconds. The visual language matching score is obtained by linearly mapping the cosine similarity between the image embedding vector and the text embedding vector output by the visual language matcher. A typical linear mapping method is to linearly map the cosine similarity from -1 to 1 to a range of 0 to 1. In terms of physical meaning, a visual-language matching score of 0.7 means that the similarity between the current real-world image and the visual anchor descriptor in the semantic embedding space is high enough, and it can be assumed that the robot has indeed seen the feature object described by the visual anchor descriptor. A visual-language matching score of 0.4 means that the similarity is in a fuzzy area, and there may be partial semantic matching but not complete matching due to changes in lighting or viewpoint shifts. A visual-language matching score of 0.1 means that there is no match at all, and the feature object currently seen by the robot is very different from the feature object described by the visual anchor descriptor, and there is a high probability that there has been a significant positional drift.

[0043] Interpretation of the physical meaning of the preset threshold set. The preset high-level anchor point matching threshold is the lower limit for determining sufficient match between the visual scene and the natural language description, typically 0.7; a larger value requires stricter visual matching, but excessively large values ​​can lead to frequent false alarms and drift; the typical value range is a closed interval between 0.6 and 0.85. The preset low-level anchor point matching threshold is the upper limit for determining complete mismatch between the visual scene and the natural language description, typically 0.4; a smaller value allows for wider tolerance of drift, but excessively small values ​​can lead to missed false alarms and drift; the typical value range is a closed interval between 0.25 and 0.55. The preset failure rate ratio threshold is the upper limit of the cumulative failure rate within the sliding window, typically 0.3, meaning that more than 3 failures in the last 10 matches trigger a hard drift; a larger value allows for wider tolerance of occasional failures, but excessively large values ​​can delay the triggering of hard drift; the typical value range is a closed interval between 0.2 and 0.5. The preset sliding window length is typically set to 10 times, which is the cumulative statistics based on the most recent 10 visual language matching times. The larger the value, the stronger the robustness to short-term fluctuations, but too large a value will lead to a delay in the response of drift detection. The typical value range is no less than 5 times and no more than 30 times.

[0044] Specific Implementation Example 1: A scenario where a home service robot fetches water from the kitchen. For example... Figure 7As shown, in this embodiment, the home service robot has a height of 1.2 meters and is equipped with a forward-facing RGB monocular camera with an image resolution of 1280p×720p and a visual acquisition frame rate of 10 frames per second. The large-scale visual language action model adopts the NaVILA-8B backbone structure and is deployed on the NVIDIA Jetson AGX Orin edge platform embedded in the robot body, with an inference frequency of approximately 2 times per second. The visual language matcher adopts a contrastive language image pre-trained matching model with the ViT-B / 32 backbone structure and is deployed on the same host as the large-scale visual language action model. The underlying motion controller is a differential drive two-wheel chassis with an operating frequency of 100 times per second. The preset high-level anchor point matching threshold is 0.7, the preset low-level anchor point matching threshold is 0.4, the preset failure rate threshold is 0.3, and the preset sliding window length is 10 times. The robot receives the user's voice command "Go to the kitchen to get a glass of water," which is then transcribed into a natural language navigation command through speech recognition. This command, along with the robot's current field of view image, is input into the visual language action model. The visual language action model generates the first action intention, "Move forward 4.2 meters to the doorway where the living room and kitchen meet," and the first visual anchor point descriptor sequence, which includes three items: "A white door should be visible ahead, with the doorknob on the right side," "A brown wooden dining table should be visible to the left front, with a blue tablecloth on the table," and "A silver-white refrigerator should be visible to the right front, with a green potted plant on top." The underlying motion controller executes the intended action. For the first 2.3 seconds, the robot moves along the corridor. The visual-language matcher calculates the matching score frame by frame for 23 images. The visual-language matching score sequences of the three anchor points fluctuate within the ranges of [0.78 to 0.85], [0.71 to 0.82], and [0.69 to 0.79], respectively, all of which are not less than the preset high-level anchor point matching threshold. The anchor point drift level is set to 0, and the robot continues to execute the original intention. In the 2.4-second period, the robot deviates from its original trajectory by about 15 centimeters due to slipping on the carpet. The visual-language matching score of the first anchor point drops to 0.52, which is between the preset low-level anchor point matching threshold of 0.4 and the preset high-level anchor point matching threshold of 0.7. The anchor point drift level is set to 1, triggering visual odometry fusion correction: extracting the frame from the 2.4-second period. There are 1856 directional gradient histogram feature descriptors. Nearest neighbor matching was performed with the 1742 descriptors in the 2.3-second keyframe to obtain 638 matching points. Based on epipolar geometry constraints, the inter-frame relative pose increment was estimated to be 0.18 meters forward and 2.3 degrees left yaw. This was used to replace the odometry-accumulated pose increment within the corresponding interval, and the original intent continued. At 3.8 seconds, the robot reached the expected doorway position, and the matching scores of the three anchor points rose to 0.81, 0.79, and 0.74. The anchor point drift grade was set to 0, and execution continued until the entire 12.6-meter path was completed and stopped in front of the kitchen sink. The visual-language-action model was called 4 times, averaging once every 3.15 meters. Visual odometry fusion correction was triggered 7 times, and no hard drift was triggered. The total time was 18.4 seconds, and the task was successful.

[0045] Specific Implementation Example 2: Long-range inspection scenario of warehouse inspection robot. For example... Figure 8 As shown, in this embodiment, the warehouse inspection robot has a height of 0.9 meters and is equipped with two forward-facing and left-side RGB cameras, each with an image resolution of 1920p×1080p and a visual acquisition frame rate of 15 frames per second. The visual language action model uses a NaVILA-8B backbone structure with an inference frequency of approximately 1.5 times per second. The visual language matcher uses a contrastive language image pre-trained matching model with a ViT-L / 14 backbone structure, deployed on the robot's RTX4070 mobile industrial PC, with a single inference latency of 28 milliseconds. The underlying motion controller is a four-wheel omnidirectional chassis with an operating frequency of 80 times per second. The preset high-level anchor point matching threshold is 0.65, the preset low-level anchor point matching threshold is 0.35, the preset failure rate threshold is 0.4, and the preset sliding window length is 15 times. The robot receives a text instruction from a remote inspector: "Inspect along row A of the East Zone to shelf number 14 and record all locations with missing labels." The target path length of this instruction is 47 meters, requiring the visual language action model to make approximately 12 consecutive decisions. After the robot is started, the visual language action model outputs the first action intention "move forward 8.3 meters to the front of shelf number 3 in row A" and the first visual anchor point descriptor sequence of four items: "A light gray shelf column should be visible in front, with yellow and black warning strips on the column"; "A red fire cabinet should be visible on the right, with an indicator sign above the fire cabinet"; "A yellow inspection line should be visible on the ground to the left front"; "A white LED strip light should be visible above in front". The underlying motion controller executes the action intent. During this process, the visual language matcher calculates the matching scores of four anchor points for each of the two cameras at 15 frames per second. At any given moment, the maximum score from both cameras is used to form the anchor point matching score set. For the first 5 seconds, the anchor point matching score sequence is not less than the preset high-level anchor point matching threshold of 0.65, and the anchor point drift level is set to 0. At the 5.2-second mark, the robot approaches a visually similar fork in an adjacent passageway. The matching score of the "red fire cabinet" anchor point in the anchor point matching score set drops to 0.28, which is lower than the preset low-level anchor point matching threshold of 0.35. The anchor point drift level is set to 2, triggering a visual language callback. Large-scale motion model: Taking the 5.2-second frame, the original inspection command, and the summary of executed historical actions (executed action intention number 1, cumulative travel distance 5.18 meters, and the pose space of the most recent hard drift trigger) as input, the visual language motion model regenerates the next action intention "turn 38 degrees to the right and continue forward 5 meters to the front of shelf number 3 in row A" and the next visual anchor descriptor sequence; the robot continues to execute according to the regenerated intention until it completes the entire 47-meter path; the large-scale visual language motion model is called 13 times, visual odometry fusion correction is triggered 19 times, and hard drift is triggered 2 times, with a total time of 52.3 seconds, and the mission is successful.

[0046] Specific Implementation Example 3: Outdoor Park Delivery Scenario for Unmanned Delivery Vehicles. In this embodiment, the unmanned delivery vehicle is 1.05 meters high and equipped with a forward-facing RGB-D camera and a surround-view fisheye camera. The RGB-D camera has a resolution of 1280p×800p and a visual acquisition frame rate of 20 frames per second. The large-scale visual language action model uses a NaVILA-8B backbone structure with an inference frequency of approximately 1.8 times per second. The visual language matcher uses a language image pre-trained matching model with Sigmoid loss, with a single inference latency of 22 milliseconds. The underlying motion controller is an Ackermann front-wheel steering rear-wheel drive chassis with an operating frequency of 50 times per second. The preset high-level anchor point matching threshold is 0.75, the preset low-level anchor point matching threshold is 0.45, the preset failure rate threshold is 0.25, and the preset sliding window length is 20 times. The delivery vehicle receives an order from the system to "deliver the express to the door of Room 305, Building 12," with a target path length of approximately 230 meters, involving an outdoor park passage, an underground parking garage passage, and an indoor building lobby. After the delivery vehicle started, the visual language action model output the first action intent, "Proceed 75 meters along the main road of the park to the east gate of Building 12," along with the first sequence of five visual anchor point descriptors. For the first 38 seconds, the anchor point matching score sequence was above the preset high anchor point matching threshold of 0.75, and the anchor point drift level was 0. Between the 39th and 41st seconds, due to strong afternoon light, all anchor point matching scores collectively dropped to between 0.55 and 0.68, falling within the range of (preset low anchor point matching threshold, preset high anchor point matching threshold). The anchor point drift level was 1, triggering visual odometry fusion correction 5 times. The robot continued to execute the original intent according to the corrected pose without calling back the visual language action model. The visual language action model was called a total of 6 times, visual odometry fusion correction was triggered 31 times, and hard drift was triggered once, with a total time of 4 minutes and 22 seconds. The task was successful.

[0047] Specific Implementation Example 4: Long-range Guidance Scenario in a Hospital Lobby Using a Guidance Robot. In this example, the guidance robot is equipped with a forward-facing RGB monocular camera with an image resolution of 1920p×1080p and a visual acquisition frame rate of 8 frames per second. The large-scale visual language action model uses a NaVILA-8B backbone structure with an inference frequency of approximately 1.2 times per second. The visual language matcher uses a contrastive language image pre-trained matching model with a ViT-B / 32 backbone structure, with a single inference latency of 12 milliseconds. The underlying motion controller is a differential-driven two-wheeled chassis with an operating frequency of 100 times per second. The preset high-level anchor point matching threshold is 0.6, the preset low-level anchor point matching threshold is 0.3, the preset failure rate threshold is 0.35, and the preset sliding window length is 8 times. The robot receives a patient's voice inquiry, "I want to go to the pediatric clinic," and the target path involves three sections: the lobby, corridor 2, and the east stairwell, totaling 38 meters. The visual language action model outputs the initial action intent "move forward 12 meters, pass the registration desk, then turn right into corridor number two" and the initial visual anchor point descriptor sequence: "Ahead, there should be a milky white registration desk with a display screen"; "To the right, there should be a green sign with white lettering for pediatrics." At 4.6 seconds, the robot deviates from its original trajectory due to dense crowds ahead. The registration desk anchor point matching score drops to 0.42, falling between the preset low anchor point matching threshold of 0.3 and the preset high anchor point matching threshold of 0.6. The anchor point drift level is set to 1, triggering visual odometry fusion correction. At 5.1 seconds, the pediatrics sign anchor point matching score drops to 0.18, below the preset low anchor point matching threshold of 0.3. The anchor point drift level is set to 2, triggering a callback to the visual language action model to regenerate the next action intent and the next visual anchor point descriptor sequence. Finally, the robot completes the 38-meter path in 38.6 seconds, and the mission is successful.

[0048] Specific Implementation Example 5: Multi-floor delivery scenario of intelligent hotel delivery robot. In this embodiment, the intelligent hotel delivery robot has a height of 0.85 meters and is equipped with one forward-facing RGB camera and one downward-facing RGB camera. The resolution of the forward-facing camera is 1280p×720p, and the resolution of the downward-facing camera is 640p×480p. The visual acquisition frame rate of the forward-facing camera is 12 frames per second, and the visual acquisition frame rate of the downward-facing camera is 5 frames per second. The large-scale visual language action model adopts the NaVILA-8B backbone structure and is deployed on the edge server in the guest room floor corridor. The robot communicates with the edge server via Wi-Fi 6 wireless network. The visual language matcher adopts the contrastive language image pre-trained matching model with the ViT-B / 32 backbone structure and is locally deployed on the JetsonOrinNano platform of the robot body, with a single inference latency of 14 milliseconds. The underlying motion controller is a differential-driven two-wheeled chassis with an elevator button interactive robotic arm, with an operating frequency of 80 times per second. The preset high-level anchor point matching threshold is set to 0.68, the preset low-level anchor point matching threshold is set to 0.38, the preset failure rate threshold is set to 0.3, and the preset sliding window length is set to 12 times. The robot receives an order from the hotel front desk to "deliver a toothbrush set to room 1208." The target path involves six sections: the lobby, the elevator entrance, the elevator car, the 12th-floor elevator entrance, the 12th-floor corridor, and the door of room 1208, with a total path length of approximately 65 meters. After the robot starts, the visual language action model outputs the first action intent "move 18 meters to the elevator entrance on the east side of the lobby" and the first visual anchor point descriptor sequence of three items: "A silver metal elevator door should be visible ahead, with a floor display screen above the door"; "A green potted plant should be visible on the left, with a height of approximately 1.5 meters"; "A hotel logo sign with a dark brown background should be visible to the right front." At 12.8 seconds, the robot approaches 2 meters from the elevator entrance. Due to a passenger pushing a suitcase obstructing its view, the elevator door anchor point matching score drops to 0.51, falling between the preset low anchor point matching threshold of 0.38 and the preset high anchor point matching threshold of 0.68. The anchor point drift level is set to 1, triggering visual odometry fusion correction. At 14.3 seconds, the robot reaches the elevator entrance, presses the elevator button, waits 7 seconds, and then enters the elevator. Inside the elevator, the visual language action model re-outputs the action intent: "Go up to the 12th floor and wait for the elevator door to open." At 31.6 seconds, the elevator reaches the 12th floor and the door opens. The robot continues to execute the re-output action intent until it completes the 65-meter delivery task in 1 minute and 18 seconds. Throughout the process, the visual language action model is called 9 times, visual odometry fusion correction is triggered 14 times, and hard drift is triggered once.

[0049] Non-triggering control branch A: Short-distance single-segment navigation scenario. In this control branch, the robot receives the instruction "move forward 1.5 meters to stop in front of the sofa". The target path is only 1.5 meters. After the visual language action model outputs the first action intention and the first visual anchor point descriptor sequence, the robot completes the execution within 2.1 seconds. During this period, the anchor point matching score sequence is always not less than the preset high-level anchor point matching threshold, the anchor point drift level is always 0, and neither visual odometry fusion correction nor callback of the visual language action model is triggered. This contrasts with the multiple three-level response in Examples 1 to 4, showing that the method described in this invention does not introduce unnecessary overhead in short-distance single-segment navigation scenarios.

[0050] Non-triggering control branch B: Long-range navigation scenario under ideal visual conditions. In this control branch, the home service robot described in Example 1 executes the same "go to the kitchen to get a glass of water" command in an ideal lighting and undisturbed home environment at dawn. Throughout the process, the visual language matching score sequence is not less than the preset high-level anchor point matching threshold, and the anchor point drift level is always 0. This is compared with the 7 visual odometry fusion corrections triggered in Example 1 under direct sunlight in the afternoon, showing that the additional energy consumption of the method described in this invention is almost zero under ideal visual conditions. Local correction or callback of the large visual language action model is only triggered when visual conditions actually drift, achieving "on-demand compensation".

[0051] General Implementation Overview. The general implementation of the method described in this invention unfolds end-to-end in five steps, S0 to S4. The inputs, outputs, core algorithms, and adjustable parameters for each step are as follows: Step S0: Inputs are the user's natural language navigation commands and the robot's current field of view image. Outputs are the action intent and a sequence of visual anchor descriptors. The core algorithm is visual language action large-scale model inference. Adjustable parameters are the selection of the main structure of the visual language action large-scale model and the length of the visual anchor descriptor sequence. Step S1: Input is the action intent. Outputs are the pose tracking commands from the underlying motion controller and a sequence of currently captured images acquired at a preset visual acquisition frame rate. The core algorithm is position-velocity-acceleration three-loop control and synchronous visual acquisition. Adjustable parameter is the preset visual acquisition frame rate. Step S2: Inputs are the current captured image and visual anchor descriptors. Output is the visual language matching score. The core algorithm is the calculation of cosine similarity of image-text embedding and linear mapping in the comparison of the pre-trained matching model of the language image. Adjustable parameter is the selection of the main structure of the visual language matcher. Step S3 takes the anchor point matching score set as input and outputs the anchor point drift classification. The core algorithm involves comparing literal thresholds in three order and statistically analyzing the proportion of failed sliding window attempts. Adjustable parameters include preset high-order anchor point matching thresholds, preset low-order anchor point matching thresholds, preset sliding window length, and preset failure rate thresholds. Step S4 takes the anchor point drift classification as input and outputs the corresponding execution response. The core algorithm is a three-branch state machine. Adjustable parameters include the selection of local feature point descriptors and the pose increment estimation method for visual odometry fusion correction.

[0052] S0 Step General Implementation Details. The S0 step can be broken down into the following four sub-actions: First sub-action: Speech recognition or text input reception. This involves using a cloud-based or local automatic speech recognition engine to transcribe the user's spoken language into natural language text. Typically, an end-to-end automatic speech recognition model based on an attention mechanism is used, with a character error rate of no more than 0.05 in a typical home setting. Second sub-action: Acquiring the robot's current field of view image. A high-resolution color image is acquired by the forward-facing camera in a single-frame capture manner. Third sub-action: Stitching the natural language navigation commands with the robot's current field of view image to form a multimodal input for the visual language action model. Typically, this is formatted according to the input template used during the pre-training of the visual language action model, with a fixed prompt word preceding the text, for example... "Please plan the next action intent based on the user's natural language navigation instructions and the current field of view image, and output 2 to 5 expected visual anchor descriptors; each descriptor must include the anchor object name, relative orientation, relative distance and significant attributes, and describe the feature objects that are expected to be visible in the field of view during the execution of the action intent; please output in the following structured format: Action intent: ...; Visual anchor descriptor 1: ...; ...; Visual anchor descriptor N: ...; The fourth sub-action, the visual language action big model inference outputs the action intent and visual anchor descriptor sequence. The typical output is structured text separated by specific delimiters. First, the target pose value of the action intent is output, and then several visual anchor descriptors separated by newline characters are output."

[0053] S1 Step General Implementation Details. The specific execution of S1 step can be broken down into the following three sub-actions: First sub-action: After receiving the motion intention, the underlying motion controller converts the target pose or desired trajectory into continuous control quantities in the joint space or chassis space, typically using model predictive control or state feedback-based control law to calculate speed and acceleration commands; Second sub-action: The underlying motion controller executes the control law at a high frequency of 50 to 200 times per second, without the need for the large visual language motion model to intervene; Third sub-action: The visual sensor periodically acquires the current real-time image at a preset visual acquisition frame rate, running asynchronously with the underlying motion controller, and the acquired real-time image is sent to the input queue of the visual language literal matching module. The upper limit of the preset visual acquisition frame rate is limited by the single inference latency of the visual language matcher. Under typical circumstances, the single inference latency of the visual language matcher is 12 to 28 milliseconds. Therefore, the preset visual acquisition frame rate should not exceed 30 frames per second. The lower limit is limited by the execution duration of the action intent. If the preset visual acquisition frame rate is too low, a sufficient number of visual language matching scores cannot be collected during the execution of a single action intent. Under typical circumstances, it should not be lower than 5 frames per second.

[0054] General implementation details of step S2. Step S2 can be broken down into the following five sub-actions: First sub-action: retrieve a current real-world image from the input queue of the visual language literal matching module; Second sub-action: input the current real-world image into the image encoder of the visual language matcher, and output a fixed-dimensional image embedding vector, typically 512 or 768 dimensions; Third sub-action: input each visual anchor descriptor in the visual anchor descriptor sequence into the text encoder of the visual language matcher, and output a text embedding vector with the same dimension as the image embedding vector; Fourth sub-action: calculate the cosine similarity for each pair of image embedding vectors and text embedding vectors, with the cosine similarity value ranging from [-1, 1]; Fifth sub-action: linearly map the cosine similarity to a closed interval between 0 and 1 to obtain the visual language matching score, and all visual language matching scores constitute the anchor matching score set. A typical form of linear mapping is to add 1 to the cosine similarity and then divide by 2 to obtain a visual language matching score ranging from 0 to 1. Alternatively, a nonlinear mapping can be performed using a Sigmoid function with a temperature coefficient adjusted. These two mapping methods can be used interchangeably in engineering implementation.

[0055] S3 Step General Implementation Details. The S3 step can be broken down into the following four sub-actions: First sub-action: Maintain a circular buffer with a preset sliding window length. Each element in the buffer records whether the most recent visual language matching failed. Failure is determined by the visual language matching score being lower than a preset low-level anchor point matching threshold. Second sub-action: For cases where any visual language matching score in the anchor point matching score set is lower than the preset low-level anchor point matching threshold, or where the proportion of failures in the circular buffer to the total number of verifications is not less than a preset failure proportion threshold, the case is classified as a hard drift, with an anchor point drift rating of 2. Third sub-action: If, without belonging to the hard drift category, at least one visual language matching score in the anchor point matching score set falls within the range of (preset low-level anchor point matching threshold, preset high-level anchor point matching threshold), the case is classified as a soft drift, with an anchor point drift rating of 1. Fourth sub-action: The remaining cases are classified as no drift, with an anchor point drift rating of 0. The order of these three categories is of great significance: certain boundary cases may simultaneously satisfy the literal conditions of the hard drift category and the soft drift category. For example, the anchor matching score set may contain both extremely low scores below the preset low anchor matching threshold and medium scores within the open interval. The order of these scores determines the drift as hard rather than soft, ensuring that the most stringent drift classification response is strictly adopted.

[0056] S4 Step General Implementation Details. The specific execution of S4 step is divided into three branches based on the anchor point drift grading value. Branch zero corresponds to an anchor point drift grading value of 0. The underlying motion controller continues to execute the current action intent without performing any additional actions. This branch accounts for 70% to 90% of the total time in a typical long-duration task. Branch one corresponds to an anchor point drift grading value of 1, triggering visual odometry fusion correction: local feature point descriptors are extracted from the current real-world image, and nearest neighbor matching is performed with the local feature point descriptors of the previous keyframe. The inter-frame relative pose increment is estimated according to epipolar geometry constraints. This inter-frame relative pose increment replaces the accumulated odometry pose increment within the corresponding time interval, resulting in the corrected pose. The underlying motion controller continues to execute the current action intent according to the corrected pose. This branch accounts for 8% to 25% of the total time in a typical long-duration task. Branch 2 corresponds to an anchor point drift level of 2. It calls back the visual language action model, using the current real-world image, the user's natural language navigation instructions, and the summary of executed historical actions as input, to regenerate the next action intent and the next visual anchor point descriptor sequence. This new sequence replaces the original intent and descriptor sequence, and then execution returns to step S1. This branch accounts for 2% to 8% of the total time for a typical long-duration task. The statistical analysis of the proportions of the three branches verifies the "on-demand compensation" characteristic of the method described in this invention. The robot executes smoothly according to the original plan most of the time, corrects minor visual deviations using local visual odometry fusion for a small amount of time, and requires a callback to the visual language action model for replanning in very rare cases, significantly saving high-level inference resources.

[0057] The adjustable parameter ranges for the general implementation steps are as follows: The visual anchor descriptor sequence length can be arbitrarily selected between 2 and 5, with a typical value of 3. The preset visual acquisition frame rate can be arbitrarily selected between 5 frames per second and 30 frames per second, with a typical value of 10 frames per second. The visual-language matcher backbone structure can be selected from one of three substructures: ViT-B / 32 backbone structure, ViT-L / 14 backbone structure, or a pre-trained language-image matching model with Sigmoid loss. The preset high-level anchor matching threshold can be arbitrarily selected within the closed interval [0.6, 0.85], and the preset low-level anchor matching threshold can be arbitrarily selected within the closed interval [0.25, 0.55]. The preset sliding window length can be arbitrarily selected between 5 and 30, and the preset failure rate threshold can be arbitrarily selected between 0.2 and 0.5. The local feature point descriptor for visual odometry fusion correction can be selected from one of three sub-descriptors: directional gradient histogram feature descriptor, acceleration segment test corner feature descriptor, and binary robust independent primitive feature descriptor. The pose increment estimation method can be selected from one of three sub-descriptors: eight-point method, five-point method, and PnP method.

[0058] Experimental verification and performance comparison. For example... Figure 9As shown, a comparison of the success rates of the method described in this invention and the original version of the baseline scheme NaVILA-8B in a 30-meter long-range navigation task shows that the success rate of the baseline scheme is 62.4%, while the success rate of the method described in this invention is 87.1%, an absolute improvement of 24.7 percentage points. Figure 10 As shown, with a two-dimensional parameter combination of preset high-level anchor point matching threshold values ​​of 0.6 to 0.85 and preset low-level anchor point matching threshold values ​​of 0.25 to 0.55, the optimal combination for success rate falls near the preset high-level anchor point matching threshold of 0.7 and the preset low-level anchor point matching threshold of 0.4, corresponding to a success rate of approximately 87.1%. When the parameter values ​​deviate from the optimal combination, the success rate decreases but is still significantly better than the baseline scheme. Figure 11 As shown, the single-inference latency of the visual language matcher described in this invention is 12 milliseconds under the ViT-B / 32 backbone architecture, 28 milliseconds under the ViT-L / 14 backbone architecture, and 22 milliseconds under the Sigmoid loss language image pre-trained matching model. These latencys are all significantly lower than the single-cycle response time of the underlying motion controller (10 to 20 milliseconds), thus allowing parallel execution with the underlying motion controller without causing blocking. The overall computational resource consumption of the method described in this invention on the NVIDIA Jetson AGX Orin edge platform is approximately 38% GPU utilization, 22% CPU utilization, and 1.8 gigabytes of memory usage. Compared to the baseline solution NaVILA-8B original version, this only increases GPU utilization by approximately 6%, CPU utilization by approximately 4%, and memory usage by approximately 0.4 gigabytes, meeting the real-time operational constraints of the embedded intelligent robot platform.

[0059] Boundary Cases and Anomaly Handling. Boundary Case 1: Minimum case where the visual anchor descriptor sequence length is 2. In this case, the anchor matching score set contains only 2 scores. The decision tree for the three-level anchor drift criterion remains unaffected and is still judged in the order of hard drift, soft drift, and no drift. Boundary Case 2: Abnormal score output by the visual language matcher. In this case, if the visual language matcher outputs a value exceeding the closed interval [0, 1] due to extremely poor image quality or special characters in the text description, the method described in this invention uses an over-boundary truncation method to map back to the closed interval, i.e., values ​​less than 0 are classified as 0, and values ​​greater than 1 are classified as 1. Boundary Case 3: Visual language action model callback timeout. In this case, if the visual language action model fails to return the next action intention within the preset timeout threshold due to network fluctuations or hardware failure, the method described in this invention automatically uses the pose after visual odometry fusion correction as the output, and the underlying motion controller hovers at zero speed and waits until the visual language action model recovers its response. Boundary Case 4: Contradictory semantics of visual anchor descriptors. In this case, if the visual language action model occasionally outputs two contradictory visual anchor descriptors (e.g., simultaneously containing "a white door should be seen ahead" and "a black iron door should be seen ahead"), the method described in this invention independently determines the accuracy of the anchor drift classification based on the visual language matching scores of each descriptor. The contradiction itself does not affect the accuracy of the anchor drift classification.

[0060] Detailed experimental data is supplemented below. The system-level test data of the method described in this invention under four typical deployment scenarios are summarized as follows, with each scenario having a sample size of no less than 100 complete task executions. For the home service robot scenario, there were 120 test samples, with an average path length of 12.6 meters, an average execution time of 18.4 seconds, a task success rate of 87.5%, an average of 4 calls to the visual-language-action large model, an average of 6.8 triggers for visual odometry fusion correction, and an average of 0.3 triggers for hard drift. For the warehouse inspection robot scenario, there were 105 test samples, with an average path length of 47 meters, an average execution time of 52.3 seconds, a task success rate of 86.7%, an average of 13 calls to the visual-language-action large model, an average of 19.2 triggers for visual odometry fusion correction, and an average of 2.1 triggers for hard drift. The unmanned delivery vehicle scenario involved 132 test samples, with an average path length of 230 meters, an average execution time of 4 minutes and 22 seconds, and a task success rate of 84.8%. The visual-language-action large model was invoked an average of 6 times, visual odometry fusion correction was triggered an average of 31 times, and hard drift was triggered an average of 1.4 times. The triage robot scenario involved 110 test samples, with an average path length of 38 meters, an average execution time of 38.6 seconds, and a task success rate of 89.1%. The visual-language-action large model was invoked an average of 5 times, visual odometry fusion correction was triggered an average of 11.6 times, and hard drift was triggered an average of 1.7 times. The weighted average overall task success rate for the four scenarios was 87.1%, significantly better than the baseline solution's 62.4% success rate.

[0061] Parameter ablation experiment results. To analyze the parameter sensitivity of the method described in this invention, a two-dimensional grid search was performed on the two core parameters: the preset high-level anchor matching threshold and the preset low-level anchor matching threshold. The search ranges were 0.6 to 0.85 with a step size of 0.05 for the preset high-level anchor matching threshold and 0.25 to 0.55 with a step size of 0.05 for the preset low-level anchor matching threshold, resulting in 35 parameter combinations. Each combination was executed 30 times in a home service robot scenario. The experimental results show that the optimal combination for success rate is a preset high-level anchor matching threshold of 0.7 and a preset low-level anchor matching threshold of 0.4, corresponding to a success rate of 87.1%. The success rate monotonically decreases when the parameter combination deviates from the optimal value, but the success rate is not lower than 84% when the preset high-level anchor matching threshold is in the range of 0.65 to 0.75 and the preset low-level anchor matching threshold is in the range of 0.35 to 0.45, indicating that the method described in this invention has a fairly wide robust range to parameter values. The effect of the preset sliding window length on the success rate is relatively mild when the value is between 5 and 30. If the value is too small, it will be too sensitive to occasional matching failures, and if the value is too large, the hard drift trigger will be delayed. The preset failure rate threshold between 0.2 and 0.5 has an effect of about 5 percentage points on the success rate, with a typical optimal value of 0.3.

[0062] Engineering trade-offs regarding the length of the visual anchor descriptor sequence. The length of the visual anchor descriptor sequence can be arbitrarily chosen between 2 and 5, with a typical value of 3. The lower limit of the visual anchor descriptor sequence length is set to 2 because a single visual anchor descriptor is easily fooled in visually similar scenes. For example, with only the descriptor "a white door should be visible ahead," the robot may misjudge between two visually similar white doors. The upper limit of the visual anchor descriptor sequence length is set to 5 because the single inference latency of the visual language matcher for each visual anchor descriptor is 12 to 28 milliseconds. If the visual anchor descriptor sequence length exceeds 5, the total matching latency exceeds 100 milliseconds, which cannot meet the real-time tracking requirements of the underlying motion controller. In actual deployment, a visual anchor descriptor sequence length of 3 is the optimal engineering compromise, ensuring sufficient visual redundancy while keeping the total matching latency within 50 milliseconds, providing several times the time margin compared to the single-cycle response time of 10 to 20 milliseconds of the underlying motion controller.

[0063] Engineering implementation details of synchronized image acquisition by visual sensors. In multi-visual-sensor configurations, image acquisition between visual sensors requires hardware-level or software-level synchronization. Hardware-level synchronization uses an external trigger signal to simultaneously trigger the shutters of multiple visual sensors, typically employing a global exposure shutter camera paired with a high-precision synchronization signal generator, achieving microsecond-level synchronization accuracy. Software-level synchronization uses timestamp alignment; each visual sensor acquires images in free-running mode, and the acquired images carry local clock timestamps. When retrieving real-world images, the visual language literal matching module matches images based on their nearest timestamp, with a typical matching window of 20 milliseconds. The method described in this invention is compatible with both synchronization methods; in actual deployment, one method is selected based on the visual sensor hardware capabilities and system latency requirements.

[0064] Edge-side optimization path for the visual language matcher. When running on an embedded intelligent robot platform, the visual language matcher can employ the following edge-side optimization paths to further reduce single-inference latency: First, model quantization: quantizing the floating-point parameters of the visual language matcher from 32-bit floating-point to 8-bit integer or 16-bit floating-point can reduce model storage space by approximately 75% and single-inference latency by approximately 30%. Second, text encoder caching: caching the text embedding vectors within the same visual anchor descriptor sequence after the first calculation, so that subsequent frames only recalculate the image embedding vectors for visual language matching, can reduce single-frame matching latency by approximately 40%. Third, image resolution reduction: reducing the image input resolution of the visual language matcher from the original 224p×224p to 112p×112p can reduce single-inference latency by approximately 50%, but simultaneously reduces the accuracy of the visual language matching score by approximately 0.05.

[0065] System, medium, and computer program product implementation description. The system of this invention includes five modules: instruction receiving and initial planning module, low-level execution and image acquisition module, visual language literal matching module, anchor point drift three-level hierarchical module, and hierarchical response execution module. Each module can be handled by an independent processing unit or by multiple parallel threads or processes of the same processing unit. Data exchange between modules is achieved through one of three communication methods: shared memory, message queue, and remote procedure call. A typical implementation uses shared memory for high-frequency image transmission and message queue for low-frequency visual language action large model callback. The computer-readable storage medium of this invention stores a computer program. When the computer program is executed by a processor, it implements the steps of the method of this invention. The specific form of the computer program includes both object code and source code. The specific form of the computer-readable storage medium includes magnetic media, optical media, and semiconductor media. The method of this invention can also be implemented as a computer program product. The computer program product includes the computer program and the running instructions document of the computer program. The computer program has the same function and interface as the computer program stored on the computer-readable storage medium.

[0066] The specific embodiments described in this specification are only used to explain the present invention and are not intended to limit the present invention. Any adjustments, combinations, or substitutions made by those skilled in the art to the corresponding parameters, module combination methods, visual sensor configurations, etc., based on the technical solutions described in this invention and actual application scenarios, that do not depart from the inventive concept of this invention, shall fall within the protection scope of this invention.

Claims

1. A visual autonomous positioning and navigation method for a visual language action large model, characterized in that, The method is operated according to the following steps: S0 receives the user's natural language navigation instructions and the current field of view image, and calls the visual language action big model to generate a sequence of action intentions and visual anchor point descriptors; The visual anchor descriptor sequence contains 2 to 5 visual anchor descriptors, each of which is a natural language literal description of a feature object in the expected field of view. S1, the underlying motion controller executes the action intention, during which it captures the current real-time image at a preset visual acquisition frame rate; S2, for each current real-shot image and each visual anchor descriptor, call the visual language matcher to output a visual language matching score with a value in [0, 1], and all visual language matching scores constitute the anchor matching score set; S3, classify the anchor point matching score set in the following order, with the anchor point drift grade set to 2, 1, and 0 respectively: The first type of hard drift occurs when any of the visual language matching scores is lower than the preset low-position anchor point matching threshold, or the proportion of times the score is lower than the threshold within the most recent preset sliding window length is not less than the preset failure rate threshold. The second type of soft drift is not in the first type and at least one of the visual language matching scores is in the range of (preset low anchor matching threshold, preset high anchor matching threshold). The third category has no drift, and the remaining cases; S4, according to the anchor point drift graded response: When the anchor point drift level is 0, the intended action will continue to be executed; When the anchor point drift level is set to 1, the odometry accumulation pose is fused and corrected using the visual features of the current real-shot image before execution continues. When the anchor point drift level is 2, the visual language action model is called back. The next action intention and the next visual anchor point descriptor sequence are regenerated with the current real-shot image, the user's natural language navigation command, and the historical action summary as input. After replacement, return to S1. The visual language matcher is a contrastive language image pre-trained matching model, which is selected from one of the following: a contrastive language image pre-trained matching model with a ViT-B / 32 backbone structure, a contrastive language image pre-trained matching model with a ViT-L / 14 backbone structure, or a language image pre-trained matching model with Sigmoid loss. The visual language matching score is obtained by linearly mapping the cosine similarity between the image embedding vector and the text embedding vector output by the visual language matcher to a closed interval between 0 and 1.

2. The method of claim 1, wherein, Each visual anchor descriptor is a short natural language sentence containing four structured fields: anchor object name field, relative orientation field, relative distance field, and significant attribute field. The anchor object name field is a literal word of the target object in the object semantic space of the user's natural language navigation command. The relative orientation field is one of six options: front, left front, right front, left side, right side, and rear. The relative distance field is a non-negative integer in a preset calibration unit length. The significant attribute field is at least one of three visual attributes: color, shape, and texture.

3. The method of claim 1, wherein, The preset high-level anchor point matching threshold is within a closed interval between 0.6 and 0.85, and the preset low-level anchor point matching threshold is within a closed interval between 0.25 and 0.

55. Furthermore, the preset high-level anchor point matching threshold is strictly greater than the preset low-level anchor point matching threshold.

4. The method of claim 1, wherein, The preset sliding window length is a visual language matching number of no less than 5 times and no more than 30 times. The preset failure rate threshold is within a closed interval between 0.2 and 0.

5.

5. The method of claim 1, wherein, The visual odometry fusion correction specifically includes the following sub-steps: extracting local feature point descriptors from the current real-shot image, wherein the local feature point descriptors are selected from one of three categories: directional gradient histogram feature descriptors, acceleration segment test corner feature descriptors, and binary robust independent primitive feature descriptors; The local feature point descriptor is matched with the local feature point descriptor of the previous keyframe to obtain a set of matching point pairs; Estimate the inter-frame relative pose increment for the set of matching point pairs according to epipolar geometry constraints; The odometry cumulative pose increment within the corresponding time interval is replaced with the inter-frame relative pose increment to obtain the corrected pose.

6. The method of claim 1, wherein, The historical action summary consists of the following three fields: the executed action sequence number field, which takes the value of the accumulated action intent number since the method was started; The cumulative distance traveled field takes the value of a non-negative real number in a preset calibration unit length. The pose field triggered by the most recent hard drift is the pose coordinates of the underlying motion controller when the anchor point drift level is 2.

7. The method of claim 1, wherein, The current field-of-view image and the current real-shot image are simultaneously acquired by no less than one and no more than four visual sensors. When the number of visual sensors is no less than two, the visual language matching score is the maximum value among the visual language matching scores output by all visual sensors for the same visual anchor descriptor at the same time.

8. A visual autonomous positioning and navigation system for visual language action large model, applying the method steps of claim 1, characterized in that, include: The instruction receiving and initial planning module is used to receive user natural language navigation instructions and the robot's current field of view image, call the visual language action big model to generate action intentions and visual anchor descriptor sequences. The visual anchor descriptor sequence contains 2 to 5 visual anchor descriptors, and each visual anchor descriptor is a natural language literal description of the expected field of view feature object. The underlying execution and image acquisition module is used to drive the underlying motion controller to execute the action intention, and to acquire the current real-shot image at a preset visual acquisition frame rate during execution; The visual language literal matching module is used to call the visual language matcher to calculate the visual language matching score for each of the current real-shot images and each of the visual anchor point descriptors, and obtain the anchor point matching score set. The visual language matching score takes a value in a closed interval between 0 and 1. Anchor point drift three-level classification module is used to classify the anchor point matching score set according to three categories: hard drift, soft drift, and no drift, to obtain the anchor point drift classification. The values ​​of the anchor point drift classification are 2, 1, and 0 respectively. The graded response execution module is used to execute the corresponding response according to the graded values ​​of the anchor point drift. When the value is 0, the action intention continues to be executed. When the value is 1, the visual odometry fusion correction is called and the execution continues. When the value is 2, the visual language action big model is called back to regenerate the next action intention and the next visual anchor point descriptor sequence.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Layered action conversion method for zero-sample visual language navigation

    CN122384815A

  • Augmented reality-based system and method for providing navigation using a route based spatial reference

    US20260194360A1