Vla control method and system fusing world model and uncertainty quantification decision

Abstract

The application provides a VLA control method and system fusing a world model and uncertainty quantification decision, wherein the method comprises: acquiring visual input, robot body state and language instruction to generate fast action and prudent action; acquiring the state at the current moment and the to-be-verified action, and inputting the state at the current moment and the to-be-verified action into a predictive world model to obtain the predicted state at the next moment; obtaining strategy inconsistency according to the difference between the fast action and the prudent action; obtaining world model deviation according to the deviation between the predicted state and the state at the current moment; weighting and fusing the strategy inconsistency and the world model deviation to obtain an uncertainty score; and controlling the corresponding decision result according to the uncertainty score; thereby, dynamic balance between speed and deep thinking can be achieved, and the robustness, safety and intelligent level of task execution in an open and unknown environment are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and intelligent manufacturing technology, in particular to a VLA control method fusing a world model and uncertainty quantization decision, a VLA control system fusing a world model and uncertainty quantization decision, a computer readable storage medium and a computer device. BACKGROUND

[0002] At present, the visual-language-action (VLA) model in the field of intelligence mainly develops along two technical routes, wherein, (1) End-to-End VLA model: such a model (such as OpenVLA) unifies the prediction of vision, language and action sequence into a single, huge neural network; the advantage is that the structure is simple, and the coherent action can be learned through behavior cloning. The defects are: a) the model is huge, the inference speed is slow, and it is difficult to meet the real-time requirement of robot action in the physical world; b) as a "black box", its decision-making process is difficult to explain and intervene, and there is a safety hazard in critical tasks; c) it completely depends on demonstration data, and its behavior is unpredictable and poor in robustness for rare or unknown situations not covered by the data. (2) Hierarchical VLA model: such a model (such as the concept of NVIDIA GR00T) separates task decomposition (handled by a large VLM) and action execution (handled by a lightweight policy model). The advantage is to balance complex planning and real-time control. The defects are: a) "planning" and "execution" are decoupled, which may cause the planned sub-tasks to exceed the actual capability range of the executor, resulting in the problem of "high eyes and low hands"; b) the system design and training process is more complex; c) there is no quick feedback and correction mechanism for dynamic changes in the environment and execution errors of itself.

[0003] In summary, the existing technologies either sacrifice real-time and safety, or sacrifice the coordination of the system and the adaptability to dynamic environment. SUMMARY

[0004] The present application aims to at least solve one of the above technical problems in the technical field. To this end, one object of the present application is to propose a VLA control method fusing a world model and uncertainty quantization decision, so that the intelligent agent can achieve a dynamic balance between speed and deep thinking, and significantly improve the robustness, safety and intelligent level of task execution in an open and unknown environment.

[0005] A second object of the present application is to propose a computer readable storage medium.

[0006] A third object of the present application is to propose a computer device.

[0007] A fourth object of the present application is to propose a VLA control system that fuses a world model and uncertainty quantification decision-making.

[0008] To achieve the above object, the first aspect of the present application proposes a VLA control method that fuses a world model and uncertainty quantification decision-making, comprising the following steps: obtaining visual input, robot body state and language instruction; generating a fast action according to the visual input and the robot body state; generating a prudent action according to the visual input, the robot body state and the language instruction; obtaining the state at the current time and the to-be-verified action, and inputting the state at the current time and the to-be-verified action into a predictive world model to obtain a predicted state at the next time; obtaining a policy inconsistency according to the difference between the fast action and the prudent action; obtaining a world model deviation according to the deviation between the predicted state and the state at the current time; weighting and fusing the policy inconsistency and the world model deviation to obtain an uncertainty score; and controlling a corresponding decision-making result according to the uncertainty score, wherein the decision-making result includes executing the fast action, executing the prudent action or actively exploring; thereby enabling the intelligent agent to achieve a dynamic balance between speed and depth of thinking, significantly improving the robustness, safety and intelligent level of task execution in an open and unknown environment.

[0009] In addition, the VLA control method that fuses a world model and uncertainty quantification decision-making according to the above embodiments of the present application can have the following additional technical features:

[0010] Optionally, generating a fast action according to the visual input and the robot body state comprises: inputting the visual input and the robot body state into a lightweight feature extraction backbone to extract features from the visual input, and splicing the features with a vector representation of the robot body state to obtain a splicing result with time sequence information, and inputting the splicing result with time sequence information into an action head to obtain the fast action.

[0011] Optionally, generating a prudent action according to the visual input, the robot body state and the language instruction comprises: inputting the visual input, the robot body state and the language instruction into a pre-trained visual language model for internal reasoning to decompose the language instruction into executable steps, and generating a prudent action using an action decoder.

[0012] Optionally, the decision result corresponding to the uncertainty score is controlled, including: obtaining a preset first threshold and a second threshold, wherein the first threshold is less than the second threshold; if the uncertainty score is less than the first threshold, then the rapid action is executed; if the uncertainty score is greater than or equal to the first threshold and less than the second threshold, then the prudent action is executed; if the uncertainty score is greater than or equal to the second threshold, then active exploration is performed.

[0013] Optionally, active exploration can be carried out, including: performing uncertainty source analysis based on the uncertainty score to obtain different analysis results; generating corresponding exploration actions based on the different analysis results; executing the exploration actions and acquiring new visual inputs, robot body states, and language commands to re-explore and make decisions.

[0014] Optionally, generating corresponding exploration actions based on the different analysis results includes: if the analysis result is that the strategy inconsistency is too high, then physical exploration and interactive exploration are performed to generate corresponding exploration actions; if the analysis result is that the world model bias is too high, then interactive exploration is performed to generate corresponding exploration actions.

[0015] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing a VLA control program that integrates a world model and uncertainty quantification decision-making. When executed by a processor, the VLA control program that integrates a world model and uncertainty quantification decision-making implements the VLA control method that integrates a world model and uncertainty quantification decision-making as described above.

[0016] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the VLA control method described above, which integrates a world model and uncertainty quantification decision-making.

[0017] To achieve the above objectives, a fourth aspect of the present invention proposes a VLA control system integrating a world model and uncertainty quantification decision-making, comprising: a perception input module for acquiring visual input, robot body state, and language commands; a fast path module for generating fast actions based on the visual input and the robot body state; a deliberate path module for generating deliberate actions based on the visual input, the robot body state, and the language commands; a prediction module for acquiring the current state and the action to be verified, and inputting the current state and the action to be verified into a predictive world model to obtain a predicted state for the next moment; and an uncertainty quantification and decision-making module for obtaining policy inconsistency based on the difference between the fast action and the deliberate action; obtaining world model deviation based on the deviation between the predicted state and the current state; performing weighted fusion of the policy inconsistency and the world model deviation to obtain an uncertainty score; and controlling the corresponding decision result based on the uncertainty score, wherein the decision result includes executing the fast action, executing the deliberate action, or conducting active exploration. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the VLA control method that integrates a world model and uncertainty quantification decision-making according to an embodiment of the present invention.

[0019] Figure 2 This is a flowchart illustrating the fast-track strategy according to an embodiment of the present invention;

[0020] Figure 3 This is a flowchart illustrating a prudent channel planner according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the prediction training process of a predictive world model according to an embodiment of the present invention;

[0022] Figure 5 This is a flowchart illustrating the uncertainty quantification and decision-making process according to an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the active exploration process according to an embodiment of the present invention;

[0024] Figure 7 This is a diagram illustrating the overall architecture of a VLA control system that integrates a world model and uncertainty quantification decision-making according to an embodiment of the present invention. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0026] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0027] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0028] refer to Figure 1 As shown, the VLA control method integrating world model and uncertainty quantification decision-making in this embodiment of the invention includes the following steps:

[0029] S101 acquires visual input, robot body status, and language commands.

[0030] As an example, visual input can be image information, depth information, etc. collected by a visual sensor (camera), robot body state can be kinematic / dynamic parameters such as joint angles, position coordinates, and velocity, and language commands can be task-oriented commands or interactive questions.

[0031] S102 generates rapid actions based on visual input and the robot's body state.

[0032] As one embodiment, generating rapid actions based on visual input and robot body state includes: inputting the visual input and robot body state into a lightweight feature extraction backbone to extract features from the visual input, concatenating the features with the vector representation of the robot body state to obtain a concatenated result with temporal information, and inputting the concatenated result with temporal information into an action head to obtain rapid actions.

[0033] As a specific example, such as Figure 2As shown, a speed-optimized, lightweight end-to-end reactive policy model is used to obtain fast actions. This typically uses a convolutional neural network (CNN, such as EfficientNet) or a visual Transformer (ViT) as the visual backbone to extract features from the visual input. These features are concatenated with vector representations of the robot's body state (such as joint angles) and then fed into a small recurrent network or Transformer (i.e., the "lightweight feature extraction backbone" in the figure) to capture temporal information. Finally, a motion head composed of a multilayer perceptron (MLP) directly regresses the robot's control command, i.e., the fast action Af. Thus, in simple scenarios requiring rapid response (such as tracking a moving object), it provides low-latency motion output; it is equivalent to human "intuitive reaction."

[0034] S103 generates deliberate actions based on visual input, robot body state, and language commands.

[0035] As one embodiment, generating deliberate actions based on visual input, robot body state, and language instructions includes: inputting visual input, robot body state, and language instructions into a pre-trained visual-language model for internal reasoning to decompose the language instructions into executable steps, and using an action decoder to generate deliberate actions.

[0036] As a specific example, such as Figure 3 As shown, it is built upon a large-scale pre-trained visual language model (such as LLaVA, GPT-4V, etc.). It receives language commands, visual input, and robot body state as multimodal inputs. Within the VLM core, the model utilizes its powerful world knowledge and language understanding capabilities to perform "chain-of-thought" internal reasoning, breaking down complex commands into a series of executable steps. Finally, through an action decoder (usually a Transformer Decoder), a more robust and deliberate action Ad is generated that conforms to the planned intent. Thus, it can handle complex language commands that require multi-step reasoning (such as "put the apple on the table into the refrigerator"), providing a well-thought-out sequence of actions; it is equivalent to human "rational thinking."

[0037] S104: Obtain the current state and the action to be verified, and input the current state and the action to be verified into the predictive world model to obtain the predicted state for the next time step.

[0038] As a specific example, such as Figure 4 As shown, a predictive world model is a recurrent neural network or Transformer model Mw, which functions to simulate changes in the physical world "in the brain".

[0039] Input: The current system state S_t and an action A_v to be verified from the decision module.

[0040] It should be noted that the current system state is derived from the system's real-time perception data of the environment and itself, and the actions to be verified are randomly selected from fast actions Af and cautious actions Ad.

[0041] Processing flow: The model Mw takes (S_t, A_v) as input and outputs a predicted state for the next time step S_pred(t+1).

[0042] Mathematical expression: S_pred(t+1) = Mw(S_t, A_v)

[0043] Training process: The model can be trained offline from the robot's historical experience data, and the model parameters are optimized by minimizing the prediction error (e.g., mean squared error, MSE).

[0044] Loss function: Loss = MSE(S_pred(t+1), S_actual(t+1))

[0045] This provides the system with a predictive ability to ask "What will happen if I do this?", which is one of the key bases for uncertainty quantification.

[0046] S105, policy inconsistency is derived from the difference between quick actions and prudent actions.

[0047] S106, the world model bias is obtained based on the deviation between the predicted state and the current state.

[0048] S107 uses a weighted fusion of policy inconsistency and world model bias to obtain an uncertainty score.

[0049] S108, control the corresponding decision outcome based on the uncertainty score, where the decision outcome includes taking a quick action, taking a prudent action, or conducting an active exploration.

[0050] As one embodiment, the decision result is controlled according to the uncertainty score, including: obtaining a preset first threshold and a second threshold, wherein the first threshold is less than the second threshold; if the uncertainty score is less than the first threshold, a rapid action is performed; if the uncertainty score is greater than or equal to the first threshold and less than the second threshold, a prudent action is performed; if the uncertainty score is greater than or equal to the second threshold, an active exploration is performed.

[0051] As a specific example, such as Figure 5 As shown:

[0052] Input: Fast action Af, deliberate action Ad, and the predicted state S_pred and the current true perception state S_t obtained after the world model rehearses one of the actions (e.g., Ad).

[0053] Implementation methods and steps:

[0054] Calculate the policy inconsistency (U_policy): Calculate the difference between Af and Ad. The greater the difference, the greater the conflict between the "intuition" and "deliberation" results of the system.

[0055] Pseudocode: U_policy = Distance(Af, Ad)

[0056] Calculate the world model deviation (U_world): Calculate the deviation between the prediction result S_pred of the world model and the current true state S_t. The greater the deviation, the worse the world model's understanding of the current situation or the more unstable the environmental state.

[0057] Pseudocode: U_world = Discrepancy(S_pred, S_t)

[0058] Calculate the total uncertainty (U_total): Weight and fuse the above two uncertainties.

[0059] Pseudocode: U_total = w1 * U_policy + w2 * U_world

[0060] Execute the decision logic: Based on two preset thresholds T1 and T2 (T1 < T2), make the final decision.

[0061] Pseudocode:

[0062] if U_total<T1:

[0063] # Highly confident, select the fast action

[0064] return SELECT_FAST_ACTION(Af)

[0065] elif T1<= U_total<T2:

[0066] # Moderately uncertain, select the deliberate action

[0067] return SELECT_DELIBERATE_ACTION(Ad)

[0068] else: # U_total>= T2

[0069] #High uncertainty, triggering active exploration

[0070] return TRIGGER_ACTIVE_EXPLORATION()

[0071] Output: A decision signal that is sent to the action gating to execute the final selected action or trigger an active exploration behavior.

[0072] As an example, active exploration includes: performing uncertainty source analysis based on uncertainty scores to obtain different analysis results; generating corresponding exploration actions based on different analysis results; executing exploration actions and acquiring new visual inputs, robot body states, and language commands to re-explore and make decisions.

[0073] It's important to note that proactive exploration isn't directly aimed at completing the user's primary task (like "picking up the cup"), but rather a system-initiated auxiliary behavior designed to reduce its own decision-making uncertainty. When the system assesses the information and deems it insufficient to support a reliable decision, it pauses the original task and performs a physical or interactive action that acquires new information. This signifies the agent's shift from a passive executor of instructions to a proactive information seeker.

[0074] As an example, corresponding exploration actions are generated based on different analysis results, including: if the analysis result is that the strategy inconsistency is too high, then physical exploration and interactive exploration are performed to generate corresponding exploration actions; if the analysis result is that the world model bias is too high, then interactive exploration is performed to generate corresponding exploration actions.

[0075] As a specific example, such as Figure 6 As shown, the steps of the active exploration mechanism include:

[0076] Step 1: Uncertainty Source Analysis

[0077] The system first analyzes the main reasons why the total uncertainty U_total exceeds the threshold T2:

[0078] High policy inconsistency (U_policy) usually indicates a significant conflict between the intuitive reaction of the "fast track" and the deliberate result of the "prudent track." This often occurs in scenarios with ambiguous perception, such as when the target object is partially occluded, or when two similar objects cannot be distinguished.

[0079] A high world model bias (U_world) means that the world model's prediction of actions differs significantly from the real world. This typically occurs when physical interactions with the environment do not meet expectations, such as trying to push a box that is thought to be light but is actually very heavy.

[0080] Step 2: Exploration Policy Generation

[0081] Based on the source of uncertainty, the system generates a specific exploration action. These actions can be divided into two categories:

[0082] 1. Physical Probing: Acquiring new sensory information through physical interaction with the environment.

[0083] Example 1 (for U_policy being too high): If a decision cannot be made because a cup is obscured, the system may generate a move_camera_to_new_angle action, that is, adjust the robot's viewpoint to observe from the side to obtain the complete outline of the object.

[0084] Example 2 (for U_world being too high): If the system is unsure whether a door is to be pushed or pulled, it may generate a nudge_door action, which is to gently push the door and determine its opening method by observing the initial reaction of the door.

[0085] 2. Interactive Probing: Obtaining higher-level semantic information through interaction with human users.

[0086] Example (for U_policy too high): When there are two similar red cups on the table, and the instruction is simply "pick up the red cup", the system can generate a verbal action that asks the user a question, such as outputting via voice or text: "Do you mean the cup on the left or the cup on the right?".

[0087] Step 3: Construct a feedback loop

[0088] The purpose of performing proactive exploration is to acquire new information, which must then be fed back into the system to form a complete closed loop of information acquisition and decision optimization. This is one of the key designs of this invention, including:

[0089] 1. Information feedback: After an exploration action is performed, new "sensory input" will be generated.

[0090] Physical detection (such as moving the viewpoint) generates new visual input and robot body state.

[0091] Interactive exploration (such as asking questions) will receive language instructions from the user in response.

[0092] 2. Closed-loop process: This new, more informative data will flow back through the entire VLA control system (fast track, prudent track, world model, etc.) as input for the next round of decision-making. Due to the acquisition of crucial new information, the system's total uncertainty U_total is expected to decrease significantly in the next evaluation.

[0093] 3. Task continuation: When the uncertainty decreases to below the threshold T2, the system will exit the active exploration mode and, based on the new evaluation results, confidently choose to execute either a "quick action" or a "prudent action" to continue completing the user's original task.

[0094] Through this closed loop of "assessment -> exploration -> perception -> reassessment", a dynamic and intelligent decision-making process is achieved, which greatly improves its robustness and success rate in complex and uncertain environments.

[0095] In summary, the VLA control method integrating world model and uncertainty quantification decision-making according to the embodiments of the present invention has the following technical effects: (1) Balancing real-time performance and performance: Through a hybrid "fast track" and "prudent track" architecture, the system can achieve extremely fast reaction speed in normal and deterministic scenarios, while in complex scenarios requiring planning, it can call upon the powerful VLM for deep thinking, dynamically balancing speed and intelligence. (2) Enhancing security and robustness: A predictive world model is introduced for "action rehearsal," and an uncertainty quantification module is designed. Before taking action, the system will assess the degree of conflict between different decisions and the reliability of the consequences of the action. For decisions with high uncertainty, it will choose a more stable solution or refuse to execute, greatly enhancing security in unknown environments. (3) Achieving proactive exploration capability: When the system perceives that its own knowledge or current information is insufficient to make a reliable decision (i.e., the uncertainty is extremely high), it can autonomously trigger "proactive exploration" behavior, such as adjusting the perspective, gently touching objects, or asking questions to the user, realizing a leap from passive execution to proactive learning. (4) Modularity and interpretability: The architecture of this application decouples different functions into clear modules, which not only facilitates independent optimization and upgrades, but also makes the decision-making process of the entire system (e.g., why choose slow decision, why trigger exploration) more transparent and interpretable.

[0096] In addition, the present invention also proposes a computer-readable storage medium storing a VLA control program that integrates a world model and uncertainty quantification decision-making. When the VLA control program that integrates a world model and uncertainty quantification decision-making is executed by a processor, it implements the VLA control method that integrates a world model and uncertainty quantification decision-making as described above.

[0097] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the VLA control method described above, which integrates a world model and uncertainty quantification decision-making.

[0098] Figure 7 This is a block diagram of a VLA control system that integrates a world model and uncertainty quantification decision-making according to an embodiment of the present invention. Figure 7 As shown, the VLA control system includes: a sensing input module, a fast channel module, a prudent channel module, a prediction module, and an uncertainty quantification and decision-making module.

[0099] The system comprises the following modules: a perception input module for acquiring visual input, robot body state, and language commands; a fast path module for generating fast actions based on visual input and robot body state; a deliberate path module for generating deliberate actions based on visual input, robot body state, and language commands; a prediction module for acquiring the current state and the action to be verified, and inputting these into a predictive world model to obtain the predicted state for the next moment; an uncertainty quantification and decision-making module for obtaining policy inconsistency based on the difference between fast and deliberate actions; obtaining world model bias based on the deviation between the predicted state and the current state; weighted fusion of policy inconsistency and world model bias to obtain an uncertainty score; and control of the corresponding decision result based on the uncertainty score, where the decision result includes executing a fast action, executing a deliberate action, or conducting active exploration.

[0100] In other words, this application discloses a Visual-Language-Action (VLA) control system that integrates world model prediction and uncertainty, aiming to resolve the core contradictions of existing intelligent models in terms of real-time performance, decision safety, and environmental adaptability. It includes a lightweight "fast track" strategy for rapid response and a large "prudent track" planner for complex semantic planning. The system introduces a predictive world model to "pre-enact" the consequences of candidate actions and designs an uncertainty quantification module. This module quantifies the uncertainty of the overall decision by calculating the "strategy inconsistency" between the fast and prudent tracks, and the "world model bias" between world model prediction and actual perception. Based on this uncertainty score, the system adopts a three-stage decision-making mechanism: executing fast actions to ensure real-time performance under low uncertainty; adopting prudent actions to ensure planning accuracy under moderate uncertainty; and autonomously triggering "active exploration" behavior under high uncertainty, actively acquiring new information through physical or interactive probing to reduce ambiguity, thus forming a decision-making closed loop. This application enables the agent to achieve a dynamic balance between speed and deep thinking, significantly improving robustness, safety, and intelligence in task execution in open and unknown environments.

[0101] Specifically, such as Figure 7 As shown, the two channels in the VLA engine generate candidate actions in parallel. The innovation module then evaluates, verifies, and makes decisions on the candidate actions, and finally outputs the most suitable action and behavior. If an exploration behavior is triggered, new perceptual input needs to be acquired again, and the system returns to the main VLA loop (see the blue dashed arrow). If a selection action is triggered, the corresponding action is executed directly.

[0102] It should be noted that the explanations and descriptions of the VLA control method that integrates the world model and uncertainty quantification decision-making described above also apply to the VLA control system that integrates the world model and uncertainty quantification decision-making in this embodiment, and will not be repeated here.

[0103] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0108] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0109] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0110] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0111] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0112] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0113] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0114] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A VLA control method integrating world model and uncertainty quantification decision-making, characterized in that, Includes the following steps: Acquire visual input, robot body status, and language commands; Generate rapid movements based on the visual input and the robot's body state; Generate deliberate actions based on the visual input, the robot's physical state, and the language instructions; Obtain the current state and the action to be verified, and input the current state and the action to be verified into the predictive world model to obtain the predicted state for the next moment; The strategy inconsistency is derived from the difference between the rapid action and the deliberate action; The world model bias is obtained based on the deviation between the predicted state and the current state. The inconsistency of the strategy and the bias of the world model are weighted and fused to obtain an uncertainty score; The decision result is controlled based on the uncertainty score, wherein the decision result includes performing the rapid action, performing the prudent action, or conducting proactive exploration.

2. The VLA control method integrating world model and uncertainty quantification decision-making as described in claim 1, characterized in that, Generate rapid actions based on the visual input and the robot's body state, including: The visual input and the robot's body state are input into a lightweight feature extraction backbone to extract features from the visual input. The features are then concatenated with the vector representation of the robot's body state to obtain a concatenated result with temporal information. The concatenated result with temporal information is then input into the motion head to obtain the rapid motion.

3. The VLA control method integrating world model and uncertainty quantification decision-making as described in claim 1, characterized in that, Generate deliberate actions based on the visual input, the robot's body state, and the language instructions, including: The visual input, the robot's body state, and the language instructions are input into a pre-trained visual language model for internal reasoning to decompose the language instructions into executable steps, and an action decoder is used to generate deliberate actions.

4. The VLA control method integrating world model and uncertainty quantification decision-making as described in claim 1, characterized in that, The decision results corresponding to the uncertainty score control include: Obtain a preset first threshold and a second threshold, wherein the first threshold is less than the second threshold; If the uncertainty score is less than the first threshold, then the fast action is performed; If the uncertainty score is greater than or equal to the first threshold and less than the second threshold, then the prudent action is performed; If the uncertainty score is greater than or equal to the second threshold, then active exploration is performed.

5. The VLA control method integrating world model and uncertainty quantification decision-making as described in claim 4, characterized in that, Active exploration, including: Uncertainty source tracing analysis is performed based on the aforementioned uncertainty score to obtain different analysis results; Generate corresponding exploration actions based on the different analysis results; The exploration action is performed, and new visual inputs, robot body states, and language commands are acquired to re-evaluate the exploration and decision-making process.

6. The VLA control method integrating world model and uncertainty quantification decision-making as described in claim 5, characterized in that, Based on the different analysis results, corresponding exploration actions are generated, including: If the analysis result indicates that the strategy inconsistency is too high, then physical exploration and interactive exploration will be performed to generate corresponding exploration actions. If the analysis result indicates that the world model bias is too high, then interactive exploration will be performed to generate corresponding exploration actions.

7. A computer-readable storage medium, characterized in that, It stores a VLA control program that integrates a world model and uncertainty quantification decision-making. When the processor executes the VLA control program that integrates a world model and uncertainty quantification decision-making, it implements the VLA control method that integrates a world model and uncertainty quantification decision-making as described in any one of claims 1-6.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the VLA control method that integrates the world model and uncertainty quantification decision-making as described in any one of claims 1-6.

9. A VLA control system integrating world model and uncertainty quantification decision-making, characterized in that, include: The perception input module is used to acquire visual input, robot body state, and language commands; The fast track module is used to generate rapid actions based on the visual input and the robot's body state; The deliberation channel module is used to generate deliberate actions based on the visual input, the robot's body state, and the language instructions. The prediction module is used to obtain the current state and the action to be verified, and input the current state and the action to be verified into the predictive world model to obtain the predicted state for the next moment. An uncertainty quantification and decision-making module is used to obtain policy inconsistency based on the difference between the rapid action and the prudent action; obtain world model deviation based on the deviation between the predicted state and the current state; perform weighted fusion of the policy inconsistency and the world model deviation to obtain an uncertainty score; and control the corresponding decision result based on the uncertainty score, wherein the decision result includes executing the rapid action, executing the prudent action, or conducting active exploration.

Images