VLN reasoning optimization method and device based on fast-slow risk self-correction

Abstract

The application discloses a VLN reasoning optimization method and device based on fast and slow risk self-correction, comprising: constructing a navigation state representation and generating a candidate execution hypothesis, fusing model cognitive uncertainty, trajectory inconsistency and visual missing risk to obtain a risk estimation value; according to the risk estimation value, the visual word units are historically conditioned and cut; the navigation state is predicted through a fast execution twin network, and the cut visual word units are reconstructed through a slow verification twin network adopting a mask self-encoding structure; according to the reconstruction error, the twin deviation and the risk estimation value, the word unit reservation mask is calibrated, and the direct execution, memory correction, slow verification, re-planning, rollback or clarification inquiry mode is determined; the corresponding action is executed, and the navigation memory is updated by using actual observation. The application can adaptively allocate visual computing amount and reasoning resources according to navigation risk, and give consideration to navigation real-time performance and reliability.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence, specifically relating to a VLN inference optimization method and apparatus based on fast and slow risk self-correction. Background Technology

[0002] Vision-and-Language Navigation (VLN) refers to the autonomous path selection and destination arrival of a navigation agent in indoor, warehousing, or logistics environments, based on natural language navigation instructions and visual observation data collected by cameras. With the development of visual language models and embodied intelligence technology, VLN is increasingly being applied to scenarios such as logistics sorting, cargo handling, warehouse inspection, and mobile robot operations.

[0003] Existing visual-language navigation methods typically employ fixed inference strategies. One type of method directly outputs actions using a lightweight navigation strategy, exhibiting low inference latency. However, it is prone to continuously executing erroneous actions when faced with obstructed shelves, similar aisles, missing target landmarks, or deviations from historical trajectories. Another type of method invokes a large visual-language model or language inference model for complex inference at each navigation moment. While this can improve decision-making capabilities in some complex scenarios, it involves significant computational demands, making it difficult to meet the real-time and power consumption requirements of logistics robots.

[0004] Furthermore, existing visual word segmentation methods typically remove some visual words based on visual saliency or linguistic relevance, and the segmentation process lacks linkage with navigation risks and historical trajectories. Whether the segmented words contain key landmark information that affects route judgment is usually impossible to verify during the reasoning process. When visual words corresponding to key signs, passage entrances, or obstacles are incorrectly segmented, subsequent navigation models may make decisions based on incomplete visual information.

[0005] Existing navigation memory is usually used to save historical locations or retrieve similar scenes. It does not uniformly write the results of visual word segmentation, the difference between fast and slow model predictions, the actual action results, and the risk estimation error into the navigation memory. Therefore, it is difficult to use the actual execution results to continuously correct subsequent risk judgments and inference resource allocation. Summary of the Invention

[0006] To address the problems existing in the prior art, this application provides a VLN inference optimization method and apparatus based on fast and slow risk self-correction.

[0007] According to a first aspect of the embodiments of this application, a VLN inference optimization method based on fast and slow risk self-correction is provided, including: Acquire current visual observation data, historical navigation trajectory, and natural language navigation instructions; determine the current sub-instruction execution progress based on the historical navigation trajectory; and construct a navigation status representation. Based on the navigation state representation, a set of candidate navigation actions is generated through the navigation strategy, and the risk estimate of the current navigation state is calculated; Based on the risk estimate, historical navigation trajectory, and current sub-instruction, historical conditional visual lexical pruning is performed on the current visual observation data to obtain a risk-matching visual representation. Based on the aforementioned risk matching visual representation, a fast and slow risk self-correcting Siamese network is constructed. The initial lexical retention mask is self-supervised and calibrated, and the inference control mode is determined based on the calibration results. The navigation action determined by the inference control mode is executed, and the actual visual observation data and twin inference feedback after execution are written into the navigation memory for subsequent risk estimation, visual lexical pruning and fast and slow twin inference control.

[0008] According to a second aspect of the application embodiments, a VLN inference optimization apparatus based on fast and slow risk self-correction is provided, comprising: The state construction module is used to acquire current visual observation data, historical navigation trajectory and natural language navigation instructions, determine the current sub-instruction execution progress based on the historical navigation trajectory, and construct a navigation state representation; The risk estimation module is used to generate a set of candidate navigation actions based on the navigation state representation and the navigation strategy, and to calculate the risk estimate value of the current navigation state. The visual lexical trimming module is used to perform historical conditional visual lexical trimming on the current visual observation data based on the risk estimate, historical navigation trajectory and current sub-instruction to obtain a risk matching visual representation; The twin self-calibration module is used to construct a fast and slow risk self-calibration twin network based on the risk matching visual representation, perform self-supervised calibration on the initial lexical retention mask, and determine the inference control mode based on the calibration results. The execution feedback module is used to execute the navigation actions determined by the inference control mode, and write the actual visual observation data and twin inference feedback after execution into the navigation memory for subsequent risk estimation, visual word pruning and fast and slow twin inference control.

[0009] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0010] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the method described in the first aspect.

[0011] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects: First, a self-supervised calibration process is established, encompassing initial pruning, mask reconstruction, twin verification, and lexical recovery, enabling the pruned visual information to be verified before action execution. Second, the lexical retention mask is jointly calibrated using reconstruction error and twin bias, directly linking visual computing resource allocation to navigation decision risk and reducing the probability of key landmark lexicals being incorrectly pruned. Third, fewer visual lexicals are retained and a fast strategy is adopted in low-risk states, while key lexicals are recovered and slow verification, backtracking, or querying is invoked in high-risk states, thus balancing inference speed and navigation success rate. Fourth, the navigation state predicted by the fast network is used as a condition for slow mask reconstruction, enabling the fast and slow networks to mutually verify each other around the same candidate action, avoiding the simple parallel combination of two independent models. Fifth, actual observations, risk labels, lexical mask changes, and twin bias are written into navigation memory, providing closed-loop feedback for subsequent risk estimation, lexical importance calculation, and inference mode selection, reducing the recurrence of similar navigation errors. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a VLN inference optimization method based on fast and slow risk self-correction, according to an exemplary embodiment.

[0013] Figure 2 This is a block diagram illustrating a VLN inference optimization apparatus based on fast and slow risk self-correction, according to an exemplary embodiment.

[0014] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0016] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0017] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0018] Figure 1 This is a flowchart illustrating a VLN inference self-correction method based on future visual imagination verification, according to an exemplary embodiment. In this example, a warehouse logistics packaging scenario is used as an example. The logistics navigation agent is equipped with an RGB-D camera, an odometer, and a mapping module to complete package packaging in the warehouse according to natural language instructions. Of course, it is not limited to this; it can also be an intelligent robot or intelligent vehicle equipped with a VLN model, which can all be processed and implemented by this invention.

[0019] refer to Figure 1 The present invention provides a VLN inference optimization method based on fast and slow risk self-correction, which may include the following steps: S1: Acquire current visual observation data, historical navigation trajectory, and natural language navigation instructions; determine the current sub-instruction execution progress based on the historical navigation trajectory; and construct candidate execution hypotheses. This step may include the following sub-steps: S11: Obtain current visual observation data, historical navigation trajectory, and natural language navigation instructions, and construct navigation memory based on the historical navigation trajectory to obtain a representation of the current navigation state; Specifically, the navigation agent acquires the current visual observation data at the current time t. Historical navigation trajectory And natural language navigation instructions I. Among them, current visual observation data... This includes the current RGB image, depth information, object detection results, landmark recognition results, and passable area detection results; historical navigation trajectory. This includes visited locations, actions performed, observed landmarks, obstacle-avoided areas, and rollback nodes.

[0020] The navigation agent constructs navigation memory based on historical navigation trajectories and current visual observation data. :

[0021] in, This indicates the function for constructing navigation memory.

[0022] Navigation memory This includes the set of visited nodes, the set of observed landmarks, the set of completed sub-instructions, the set of repeatedly visited areas, and the set of nodes that can be reverted. Subsequently, the current visual observation data, historical navigation trajectory, natural language navigation instructions, and navigation memory are encoded into a current navigation state representation. :

[0023] in, This indicates the current navigation status; This represents the state coding function.

[0024] S12: Based on the natural language navigation instructions and the navigation memory, determine the current sub-instruction to be executed; Specifically, use To represent the j-th sub-instruction, the navigation agent first divides the natural language navigation instruction I into multiple ordered sub-instructions:

[0025] Where m represents the number of sub-instructions.

[0026] For example, the above logistics repackaging instructions can be divided into... Enter the blue channel from the receiving point; : Passed through cold chain shelves; Turn right after passing the cold chain shelves; Send small packages to the B2 sorting area; If you see a yellow return box, walk around it from the left.

[0027] Subsequently, the navigation agent, based on navigation memory Determine if each sub-instruction has been completed:

[0028] in, Indicates the completion status of the j-th subinstruction; when When the sub-instruction has been completed, it indicates that the sub-instruction has been completed. This indicates that the sub-instruction has not yet been completed; This indicates the function for detecting the completion status of a subinstruction.

[0029] The currently pending subinstruction is determined to be the first subinstruction in the sequence that has not yet been completed:

[0030] in, Indicates the sub-instruction currently to be executed; This function selects the first subinstruction from the set of uncompleted subinstructions.

[0031] S13: Based on the current navigation state representation and the current sub-instruction to be executed, a set of candidate navigation actions is generated through the navigation strategy, and further expanded to obtain candidate execution hypotheses.

[0032] Specifically, the navigation agent represents the current navigation state. and currently pending sub-instructions Input the navigation strategy to obtain the probability distribution of each candidate action:

[0033] in, This indicates that the candidate navigation action will be executed under the current navigation state and the current sub-instruction to be executed. The probability of; Indicates the navigation strategy.

[0034] Subsequently, the top K highest-scoring actions are selected to generate a candidate navigation action set. :

[0035] in, Represents the set of candidate navigation actions; This indicates that the top K candidate actions with the highest probability are selected; K represents the number of candidate actions.

[0036] For the candidate navigation action set For each candidate action, the navigation agent bases its actions on the current navigation state. Navigation memory and maximum unfolding step size The action is expanded to obtain candidate execution hypotheses. :

[0037] in, Indicates the nth candidate execution hypothesis; This represents the nth candidate navigation action; This indicates the function expansion.

[0038] Furthermore, candidate execution hypotheses It can be represented as:

[0039] in, This indicates that the nth candidate performs the l-th action in the hypothesis; l represents the number of steps to be expanded.

[0040] Through the above steps, the system first determines which sub-instruction it is currently in, and then generates candidate actions using a fast navigation strategy, instead of calling slow, large-model inference at every step. This maintains real-time navigation and provides clear input for subsequent risk estimation and switching between fast and slow inference.

[0041] S2: Based on the candidate execution assumptions, generate a set of candidate navigation actions using a navigation strategy, and calculate the risk estimate of the current navigation state; this step may include the following sub-steps: S21: Calculate the cognitive uncertainty of the model based on the action probability distribution output by the navigation strategy; Specifically, the navigation agent reads the navigation strategy. Perform hypothesis on candidate Candidate navigation actions The probability distribution of the action is used to calculate the cognitive uncertainty of the model. :

[0042] in, This indicates cognitive uncertainty in the model; This indicates a candidate navigation action.

[0043] If the navigation strategy has a significantly higher probability for a certain action, then The probability is low; if the probabilities of multiple actions such as continuing forward, turning right, and detouring to the left are similar, then... A higher value indicates that the fast strategy itself cannot clearly determine which action should be performed at the current time.

[0044] S22: The calculated trajectory is inconsistent based on the historical navigation trajectory, navigation memory, and the current sub-instruction to be executed; The navigation agent compares historical navigation trajectories. Navigation memory and currently pending sub-instructions To determine whether the current actual path contradicts the instruction progress and historical memory, trajectory inconsistency can be represented as:

[0045] in, This indicates inconsistency in the trajectories; Indicates the degree of sequence error, used to determine whether to execute subsequent actions without completing the preceding sub-instruction; This indicates the degree of repeated access, used to determine whether an already visited area has been entered repeatedly; Indicates the degree of deviation from the target direction; Indicates the degree of conflict observed in history; to This indicates the weight of each item.

[0046] The degree of sequence error can be determined based on the current sub-instruction to be executed and the historically completed sub-instructions. For example, if the navigation memory has not yet confirmed "passing the cold chain shelf," but the historical actions show that "turning right" has already been executed, then... Increase. The degree of repeated visits can be calculated based on the degree of overlap between the current node and the set of visited nodes; the degree of deviation from the target direction can be judged based on the change in distance from the current topology node to the target area; the degree of historical observation conflict can be judged based on whether the currently observed landmarks contradict the area labels already confirmed in the past.

[0047] S23: Calculate the visual loss risk based on the appearance of key landmarks and target objects in the currently pending sub-instruction within a preset time window; Specifically, the navigation agent starts from the currently pending sub-instruction. Extract key landmarks and target object sets:

[0048] in, This represents the set of key landmarks or target objects in the currently pending sub-instruction; This represents the key landmark extraction function.

[0049] Subsequently, the presence of each key landmark was determined from the visual observation data sequence within the most recent preset time window:

[0050] in, This represents the visual observation data window from time tw to the current time t; w represents the window length.

[0051] For each key landmark Calculate the degree of its missing information:

[0052] in, Indicates key landmarks The degree of absence; Indicates visual observation data Key landmarks detected in China The confidence level.

[0053] The risk of visual loss is represented as :

[0054] in, Indicates the risk of visual impairment; Indicates key landmarks Importance weights.

[0055] S24: By integrating the model's cognitive uncertainty, trajectory inconsistency, and visual loss risk, a risk estimate of the current navigation state is obtained.

[0056] Specifically, the navigation agent will recognize uncertainty in the model. Inconsistent trajectories and the risk of visual loss By performing fusion, a risk estimate of the current navigation state can be obtained. :

[0057] in, This represents the risk estimate of the current navigation status; to This indicates the weight of the corresponding risk item.

[0058] Further Normalize the values ​​to ensure they fall within the preset range of 0-1. A higher risk estimate indicates that relying solely on navigation strategies for direct action is less suitable; a lower risk estimate suggests that a lighter visual representation and faster execution mode can be employed. If the fast strategy action distribution is clear, the historical trajectory matches the current sub-instruction, and key landmarks have already appeared within the nearest window, then... Lower; if the rapid strategy is uncertain, the path progress contradicts the order of instructions, or key landmarks fail to appear for an extended period, then... Relatively high.

[0059] Through the steps described above, the navigation agent transforms the process of determining whether slow inference or additional verification is required from a fixed rule into a computable risk estimation process. This risk estimation considers not only the inherent uncertainties of the navigation strategy itself, but also whether the current path contradicts the instruction progress and whether key landmarks are missing for extended periods. As a result, the system can identify potential failure states earlier and provide a unified risk basis for subsequent steps such as determining the visual lexical retention ratio in S3, twin MAE verification in S4, and the selection of inference control modes in S5.

[0060] S3: Based on the risk estimate, historical navigation trajectory, and current sub-instruction, perform historical conditional visual lexical pruning on the current visual observation data to obtain a risk-matching visual representation; this step may include the following sub-steps: S31: Encode the visual observation data into a set of visual terms, calculate the historical conditional importance of each visual term based on the risk estimate, historical navigation trajectory and current sub-instruction, and determine the retention ratio of visual terms; Specifically, the navigation agent will use current visual observation data Input the visual encoder to obtain the visual lexicon set. :

[0061] in, Represents the current set of visual lexical units; Represents the visual lexical encoding function; represents the i-th visual word; N represents the number of words obtained after encoding the current visual observation data.

[0062] Subsequently, the navigation agent based on the currently pending sub-instruction Historical navigation trajectory Navigation memory Risk of visual loss and candidate execution hypotheses Calculate the historical conditional importance of each visual lexical unit. :

[0063] in, Indicates the candidate execution hypothesis Next The historical contextual importance of visual lexical elements; Visual lexical elements With the currently pending subinstruction semantic relevance; Visual lexical elements The degree of correlation with historical navigation trajectories and navigation memories; Visual lexical elements Risks related to navigation status; Visual lexical units With candidate execution hypothesis The correlation between the consequences of actions; to This indicates the weight of each item.

[0064] The final output is a set of visual lexical terms. and the set of lexical importance corresponding to each candidate execution hypothesis. :

[0065] in, Indicate candidate execution hypothesis The set of visual lexical importance.

[0066] Furthermore, the risk estimate obtained by the navigation agent based on S2 The retention ratio of visual words is dynamically determined. Lower risk indicates the current state is more suitable for rapid inference, allowing for the pruning of more visual words; higher risk indicates potential landmark missingness, path contradictions, or model uncertainty, necessitating the retention of more visual words. The visual word retention ratio can be expressed as... :

[0067] in, Indicates the current percentage of visual lexical units retained; This indicates the minimum percentage of lexical units retained under low-risk conditions. This indicates the maximum percentage of lexical units retained under high-risk conditions; This represents the risk estimate for the current navigation status.

[0068] When the importance of a few words is significantly higher than that of others, it indicates that there are a few key regions in the current observation. In this case, the retention ratio should be appropriately increased to avoid key landmarks or obstacle words being pruned.

[0069] S32: Based on the historical conditional importance of each visual word and the retention ratio of the visual word, generate an initial word retention mask, and obtain the initial retained visual word and the initial pruned visual word; The lexical importance set obtained by the navigation agent and retention ratio Select the visual lexical units with the highest importance ranking to generate an initial lexical unit preservation mask. :

[0070] in, Indicate candidate execution hypothesis The corresponding initial lexical preservation mask; This represents a function that generates a mask based on lexical importance and retention ratio.

[0071] Based on the initial lexical retention mask, the visual lexical set is divided into initially retained visual lexical units and initially pruned visual lexical units:

[0072]

[0073] in, Indicate candidate execution hypothesis The corresponding initial retained visual lexical units; Indicate candidate execution hypothesis The corresponding initial cropped visual lexical units; This indicates an element-by-element selection operation.

[0074] In the logistics sorting scenario of this case, for the candidate route "turn right into the aisle", the mask will tend to retain the words near the right aisle entrance, sorting port sign, relative position of cold chain shelf and yellow box; for the candidate route "continue forward", the mask will tend to retain the words at the end of the cold chain shelf ahead and the direction of the main aisle.

[0075] S33: Based on the initially retained visual lexical units, construct a visual representation for risk matching; Specifically, the navigation agent will initially retain visual lexical units. Input a lightweight visual representation encoder to construct a risk-matching visual representation. :

[0076] in, Indicate candidate execution hypothesis The corresponding initial risk matching visual representation; Represents the visual representation encoding function; Indicates the initial retention of visual lexical units; Indicates the sub-instruction currently to be executed; This represents the candidate execution hypothesis.

[0077] This risk-matching visual representation is used in the subsequent fast execution Siamese network in S4 to predict the navigation state after executing candidate actions. Meanwhile, S3 retains... The cut and initial lexical preservation mask The slow check twin network that shares the input to S4 is used to perform MAE reconstruction and self-supervised lexical mask calibration.

[0078] Through the above steps, the navigation agent can dynamically adjust the computational load of visual lexical units according to the current risk level: in a low-risk state, only a small number of high-value lexical units are retained to improve real-time performance; in a high-risk state, more lexical units related to the current sub-instruction, historical trajectory contradictions, missing key landmarks, and consequences of candidate actions are retained to reduce the risk of mis-pruning, providing reconstructable, verifiable, and feedback-updated inputs for the twin MAE self-supervised calibration in S4, thereby forming a small self-supervised closed loop of S3-S4-S3.

[0079] S4: Construct a fast and slow risk self-calibrating Siamese network based on the aforementioned risk matching visual representation, perform self-supervised calibration on the initial lexical retention mask, and determine the inference control mode based on the calibration results; this step may include the following sub-steps: S41: Input the risk matching visual representation, candidate execution hypothesis and current sub-instruction to be executed into the fast execution Siamese network to obtain the fast prediction state corresponding to each candidate navigation action; Specifically, the navigation agent will match the risk visual representation obtained by S3. Candidate execution hypothesis and currently pending sub-instructions Input the fast execution Siamese network to obtain the fast predicted state corresponding to each candidate navigation action.

[0080]

[0081] in, This indicates that the fast execution twin network is used to perform the candidate execution hypothesis. The predicted state is obtained quickly; This indicates the rapid execution of twin networks; This represents the initial visual representation of risk matching. Indicate the candidate execution hypothesis; This indicates the sub-instruction currently pending execution.

[0082] The rapid prediction This can include information such as predicted location, predicted orientation, confidence level of candidate actions, progress of sub-instructions, and whether it is possible to enter an abnormal region. Because this step uses S3-pruned lexical units for fast prediction, the computational cost is low, making it suitable for real-time navigation.

[0083] S42: Using the fast prediction state as the action consequence condition, the initial truncated visual lexical is reconstructed by a slow-verification twin network with a masked autoencoder structure to obtain the reconstructed visual lexical and reconstruction error. Specifically, the navigation agent will initially retain visual lexical units. Historical navigation trajectory Navigation memory and currently pending sub-instructions The input is a slow, verified Siamese network with a masked autoencoder structure, which is used to reconstruct the pruned visual lexical units in S3:

[0084] in, This represents the reconstruction result of the slow-verification twin network on the pruned visual words; This represents the mask autoencoder structure; Indicates the initial retention of visual lexical units; Indicates historical navigation trajectory; Indicates navigation memory; This indicates the sub-instruction currently pending execution.

[0085] Subsequently, the reconstruction error between the reconstructed result and the initial cropped visual lexical units was calculated. :

[0086] in, Indicate candidate execution hypothesis The corresponding error in reconstructing the clipped lexical units; Represents the distance metric function; Indicates the initial cropped visual word.

[0087] like A smaller value indicates that the pruned word can be well inferred from the preserved word, historical trajectory, and current sub-instruction, making the pruning of the current word relatively safe; if A large value indicates that the clipped lexical units may contain key landmarks, obstacles, or path clues, requiring reverse calibration of the S3 lexical retention mask.

[0088] S43: Based on the initial retained visual lexical units and the reconstructed visual lexical units, the slow verification state is obtained through the slow verification Siamese network, and the Siamese deviation between the slow verification state and the fast prediction state is calculated. The navigation agent will initially retain visual lexical units. Reconstructed cropped visual lexical units Historical navigation trajectory Navigation memory Currently pending sub-instruction and candidate execution hypotheses Input a slow verification twin and predict the verification state corresponding to the same candidate execution hypothesis.

[0089]

[0090] in, This indicates the verification status obtained from the slow-speed verification twin prediction. This indicates a slow-speed verification twin.

[0091] Subsequently, the fast prediction state for fast twin prediction is calculated. Verification status predicted by slow verification twin Twin bias between them:

[0092] in, Indicate candidate execution hypothesis Corresponding twin bias; This represents a state difference measurement function.

[0093] like A smaller value indicates that the fast-execution twin and the slow-verification twin make consistent judgments on the consequences of candidate actions; if A large value indicates that the fast path may make incorrect judgments due to word segmentation, insufficient historical memory, or insufficient local observation, requiring the triggering of a more conservative reasoning control mode.

[0094] S44: Based on the reconstruction error, twin bias and risk estimate, calibrate the initial lexical retention mask to obtain the calibrated visual representation, and determine the inference control mode based on the calibrated visual representation; Specifically, the navigation agent reconstructs the error based on the MAE. Twin bias and risk estimates Preserve the mask for the initial tokens generated by S3. Perform self-supervised calibration to obtain the lexical preservation mask after calibration. :

[0095] in, This indicates the lexical preservation mask after calibration; This represents the word mask update function; This indicates the initial lexical preservative mask.

[0096] when When the value is large, it indicates that there is information in the pruned lexical units that is difficult to reconstruct, and the system increases the lexical unit retention weight of the corresponding spatial or semantic region; when When the value is large, it indicates a significant discrepancy between fast prediction and slow verification, and the system increases the retention rate of terms related to the current sub-instruction, historical missing landmarks, or candidate actions.

[0097] Furthermore, the calibrated visual representation is reconstructed based on the calibrated lexical-preserving mask. :

[0098]

[0099] in, This indicates that visual lexical units are retained after calibration; Represents the current set of visual lexical units; Indicates calibration of visual representation; This represents the visual representation encoding function.

[0100] Finally, the navigation agent determines the inference control mode based on the risk estimate, reconstruction error, and twin bias:

[0101] in, Indicates the reasoning control mode; This represents the inference control function.

[0102] The reasoning control modes include direct execution, memory retrieval correction, slow verification, replanning, rollback, or clarification query. Specifically, when , and When both are low, choose to execute directly; when When the value is high, prioritize increasing lexical retention or triggering slow verification; when When the value is high, it triggers memory retrieval correction, replanning, or rollback; when neither historical memory nor visual observation data is sufficient to eliminate ambiguity, it triggers clarification inquiry.

[0103] This step preserves the calibrated lexical mask. and calibrating visual representations The feedback is then sent to the subsequent navigation control process, thus forming a small self-supervised closed loop of S3-S4-S3.

[0104] Through the above steps, the navigation agent does not simply switch between a fast policy and a slow inference engine based on a risk threshold. Instead, it utilizes a Siamese MAE self-supervised mechanism to check whether the pruned visual lexical units in S3 contain key navigation information, and judges the reliability of fast inference by the Siamese bias between the fast execution Siamese and the slow verification Siamese. This step can maintain real-time performance in low-risk states while automatically adding visual information, triggering slow verification, replanning, backtracking, or clarifying queries in high-risk or unreliable pruning states, thereby improving navigation success rate and reducing unnecessary large model calls.

[0105] S5: Execute the navigation action determined by the inference control mode, and write the actual visual observation data and twin inference feedback after execution into the navigation memory for subsequent risk estimation, visual lexical pruning, and fast / slow twin inference control; this step may include the following sub-steps: S51: Based on the inference control mode, determine the final output action from the candidate navigation action set and execute it, and obtain the actual visual observation data after execution; Specifically, by applying candidate execution hypotheses Make a selection:

[0106] in, This represents the execution utility of the nth candidate route; This represents the task benefit calculated based on the calibrated visual representation; , These are the corresponding weights.

[0107] Select the candidate route with the highest execution efficiency. :

[0108] Based on the inference control model, from the set of candidate navigation actions Inference control mode generates actions :

[0109] in, Indicates the reasoning control mode; Indicates the target candidate route; This indicates the current navigation memory.

[0110] Execute the final output action The actual visual observation data after execution is obtained. :

[0111] in This represents the action function executed by the navigation agent, which collects current visual observation data.

[0112] S52: Include risk estimates, initial lexical retention mask, calibrated lexical retention mask, reconstruction error, twin bias, inference control mode, and visual observation data. and output action The navigation memory is written into the system for subsequent risk estimation, visual lexical pruning, and fast / slow twin reasoning control.

[0113] Specifically, based on actual visual observation data Extract target landmarks, spatial relationships, and regional identifiers to calculate the completion progress of the current sub-instruction. :

[0114] in, Indicates the current execution progress of the subinstruction; Indicates the visual matching degree of the target landmark; Indicates the degree of correspondence between the actual location and the spatial relationship; It indicates the degree to which the actual action conforms to the requirements of the sub-instruction action; Indicates the level of support provided by historical navigation memory; This indicates the fusion weight.

[0115] Then construct and execute the output action. Subsequent navigation status representation and navigation fact record :

[0116]

[0117] Risk estimate Initial lexical preservation mask , calibrated lexical preservation mask Reconstruction error Twin bias Inference control mode and navigation fact record Write the navigation memory and update the obtained navigation memory. :

[0118] S5 establishes a closed-loop connection between the inference control results of S4 and the robot's actual execution results. Actual visual observation data is not only used to update position and sub-instruction progress, but is also converted into risk supervision labels and lexical recovery labels, enabling the system to identify whether previous risk estimates were accurate and which visual lexical units should not be pruned. This forms a continuous self-correcting process of risk estimation, lexical pruning, twin calibration, action execution, and actual result feedback, reducing the probability of the logistics navigation agent repeatedly entering the wrong channel, missing target signs, or executing invalid backtracking actions.

[0119] Corresponding to the aforementioned embodiments of the VLN inference optimization method based on fast and slow risk self-correction, this application also provides embodiments of a VLN inference optimization apparatus based on fast and slow risk self-correction.

[0120] Figure 2 This is a block diagram illustrating a VLN inference optimization apparatus based on fast and slow risk self-correction according to an exemplary embodiment. The apparatus includes: The state construction module 1 is used to acquire current visual observation data, historical navigation trajectory and natural language navigation instructions, determine the current sub-instruction execution progress based on the historical navigation trajectory, and construct a navigation state representation; Risk estimation module 2 is used to generate a set of candidate navigation actions based on the navigation state representation and the navigation strategy, and to calculate the risk estimate value of the current navigation state; The visual lexical trimming module 3 is used to perform historical conditional visual lexical trimming on the current visual observation data based on the risk estimate, historical navigation trajectory and current sub-instruction to obtain a risk matching visual representation; The twin self-calibration module 4 is used to construct a fast and slow risk self-calibration twin network based on the risk matching visual representation, perform self-supervised calibration on the initial lexical retention mask, and determine the inference control mode based on the calibration results. The execution feedback module 5 is used to execute the navigation actions determined by the inference control mode, and write the actual visual observation data and twin inference feedback after execution into the navigation memory for subsequent risk estimation, visual word pruning and fast and slow twin inference control.

[0121] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement a VLN inference optimization method based on fast and slow risk self-correction as described above. Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device, including a VLN inference optimization device based on fast and slow risk self-correction provided in an embodiment of the present invention. (Except for...) Figure 3 In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0122] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the VLN inference optimization method based on fast / slow risk self-correction as described above. The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0123] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0124] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A VLN inference optimization method based on fast and slow risk self-correction, characterized in that, include: Acquire current visual observation data, historical navigation trajectory, and natural language navigation instructions; determine the current sub-instruction execution progress based on the historical navigation trajectory; and construct a navigation status representation. Based on the navigation state representation, a set of candidate navigation actions is generated through the navigation strategy, and the risk estimate of the current navigation state is calculated; Based on the risk estimate, historical navigation trajectory, and current sub-instruction, historical conditional visual lexical pruning is performed on the current visual observation data to obtain a risk-matching visual representation. Based on the aforementioned risk matching visual representation, a fast and slow risk self-correcting Siamese network is constructed. The initial lexical retention mask is self-supervised and calibrated, and the inference control mode is determined based on the calibration results. The navigation action determined by the inference control mode is executed, and the actual visual observation data and twin inference feedback after execution are written into the navigation memory for subsequent risk estimation, visual lexical pruning and fast and slow twin inference control.

2. The VLN inference optimization method based on fast and slow risk self-correction according to claim 1, characterized in that, Acquire current visual observation data, historical navigation trajectories, and natural language navigation commands; determine the current sub-command execution progress based on the historical navigation trajectory; and construct a navigation state representation, including: Acquire current visual observation data, historical navigation trajectory, and natural language navigation instructions, and construct navigation memory based on the historical navigation trajectory to obtain a representation of the current navigation state; Based on the natural language navigation instructions and the navigation memory, the current sub-instruction to be executed is determined; Based on the current navigation state representation and the current sub-instruction to be executed, a set of candidate navigation actions is generated using a navigation strategy; Expand the candidate navigation actions in the candidate navigation action set to obtain candidate execution hypotheses.

3. The VLN inference optimization method based on fast and slow risk self-correction according to claim 1, characterized in that, Based on the navigation state representation, a set of candidate navigation actions is generated through a navigation strategy, and a risk estimate of the current navigation state is calculated, including: Based on the action probability distribution output by the navigation strategy, the cognitive uncertainty of the model is calculated. The calculated trajectory is inconsistent based on the historical navigation trajectory, navigation memory, and the current sub-instruction to be executed. Based on the appearance of key landmarks and target objects in the currently pending sub-instruction within a preset time window, calculate the risk of visual loss. By integrating the model's cognitive uncertainty, trajectory inconsistency, and visual loss risk, a risk estimate for the current navigation state is obtained.

4. The VLN inference optimization method based on fast and slow risk self-correction according to claim 1, characterized in that, Based on the risk estimate, historical navigation trajectory, and current sub-instruction, historical conditional visual lexical pruning is performed on the current visual observation data to obtain a risk-matching visual representation, including: Visual observation data is encoded into a set of visual terms. Based on risk estimates, historical navigation trajectories, and current sub-instructions, the historical conditional importance of each visual term is calculated, and the retention ratio of visual terms is determined. Based on the historical conditional importance of each visual lexical and the retention ratio of the visual lexical, an initial lexical retention mask is generated, and the initial retained visual lexical and the initial clipped visual lexical are obtained. Based on the initial retained visual lexical units, a risk matching visual representation is constructed.

5. The VLN inference optimization method based on fast and slow risk self-correction according to claim 1, characterized in that, Based on the aforementioned risk-matching visual representation, a fast-slow risk self-calibrating Siamese network is constructed. The initial lexical-preserving mask is self-supervised for calibration, and the inference control mode is determined based on the calibration results, including: The risk matching visual representation, candidate execution hypotheses, and currently pending sub-instructions are input into the fast execution Siamese network to obtain the fast prediction state corresponding to each candidate navigation action. Using the fast prediction state as the action consequence condition, the initially pruned visual lexical is reconstructed by a slow-verification twin network with a masked autoencoder structure, resulting in the reconstructed visual lexical and reconstruction error. Based on the initial retained visual lexical units and the reconstructed visual lexical units, the slow verification state is obtained through the slow verification Siamese network, and the Siamese deviation between the slow verification state and the fast prediction state is calculated. The initial lexical retention mask is calibrated based on the reconstruction error, twin bias, and risk estimate to obtain a calibrated visual representation, and the inference control mode is determined based on the calibrated visual representation.

6. The VLN inference optimization method based on fast and slow risk self-correction according to claim 1, characterized in that, Execute the navigation actions determined by the inference control mode, and write the actual visual observation data and twin inference feedback after execution into the navigation memory for subsequent risk estimation, visual lexical pruning, and fast / slow twin inference control, including: Based on the inference control mode, the final output action is determined from the candidate navigation action set and executed, and the actual visual observation data after execution is obtained; The risk estimate, initial lexical retention mask, calibrated lexical retention mask, reconstruction error, twin bias, inference control mode, visual observation data, and output action are written into the navigation memory for subsequent risk estimation, visual lexical pruning, and fast and slow twin inference control.

7. A VLN inference optimization device based on fast and slow risk self-correction, characterized in that, include: The state construction module is used to acquire current visual observation data, historical navigation trajectory and natural language navigation instructions, determine the current sub-instruction execution progress based on the historical navigation trajectory, and construct a navigation state representation; The risk estimation module is used to generate a set of candidate navigation actions based on the navigation state representation and the navigation strategy, and to calculate the risk estimate value of the current navigation state. The visual lexical trimming module is used to perform historical conditional visual lexical trimming on the current visual observation data based on the risk estimate, historical navigation trajectory and current sub-instruction to obtain a risk matching visual representation; The twin self-calibration module is used to construct a fast and slow risk self-calibration twin network based on the risk matching visual representation, perform self-supervised calibration on the initial lexical retention mask, and determine the inference control mode based on the calibration results. The execution feedback module is used to execute the navigation actions determined by the inference control mode, and write the actual visual observation data and twin inference feedback after execution into the navigation memory for subsequent risk estimation, visual word pruning and fast and slow twin inference control.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction performs the method as described in any one of claims 1-6.

Images