A world model-enhanced vla robot control method and system based on prediction credibility

Abstract

The application discloses a world model-enhanced VLA robot control method and system based on prediction credibility, and belongs to the field of robot control. The method comprises the following steps: acquiring first observation data and task instructions; generating a prediction state sequence based on the first observation data, wherein the prediction state sequence indicates the state of the robot at multiple future time steps; determining a credibility parameter of the prediction state sequence; fusing a first control instruction and a second control instruction to generate target action data according to the credibility parameter, wherein the first control instruction is determined based on the prediction state sequence, and the second control instruction is determined based on the first observation data; and outputting a control signal corresponding to the target action data, wherein the control signal is used to drive a servo driver of the robot. The application realizes a deterministic coupling of a full-stack signal driving, so that the bottom controller can directly consume the prediction quality and make decisions accordingly.

Description

Technical Field

[0001] This invention relates to the field of robot control, and in particular to a world model-enhanced VLA method and system based on predictive credibility dynamic coupling. Background Technology

[0002] With the development of fundamental robot models such as Visual-Language-Motion (VLA) models and world models, robots are now able to generate control actions based on natural language tasks and multimodal environmental observations. However, existing solutions typically use the long-horizon prediction results output by the world model directly as the basis for planning, or combine predictive control with reactive control based on current observations according to fixed weights, fixed periods, or preset patterns. There is a lack of a unified technical measure to determine whether future predictions are trustworthy.

[0003] First, world models measure prediction quality using image reconstruction error, latent state prediction error, or training loss, making it difficult to directly identify physically infeasible situations such as predicted joint configurations exceeding mechanical limits or spatial interference between the robot's envelope and obstacles. Furthermore, even if the predicted trajectory satisfies kinematic and collision constraints, it may gradually deviate from the task objective defined by the verbal instructions. Moreover, existing control architectures typically employ fixed open-loop horizons and fixed replanning frequencies, leading to frequent calls to high-computation models even when predictions are reliable, increasing inference latency and computational resource consumption; conversely, when predictions are distorted, outdated instructions cannot be promptly cleared and high-frequency closed-loop control restored. Second, direct Euclidean weighting or hard switching between anticipatory and reactive actions easily causes rotational pose distortion, abrupt command changes, and mechanical shocks. Furthermore, uncertainties at the cognitive layer are not further transmitted to the underlying stiffness and damping control; erroneous instructions can penetrate upper-level judgments, allowing the robot to still engage in resistance against the environment with high stiffness. Summary of the Invention

[0004] One of the objectives of this invention is to provide a world model-enhanced VLA robot control method based on prediction reliability, in order to solve the problems in the prior art, such as the lack of a unified evaluation of the physical feasibility and task consistency of future predictions, the fixed coupling relationship between predictive control and reactive control, and the possibility that the robot may still perform unreliable actions with high stiffness when the prediction is distorted.

[0005] This invention is achieved through the following technical solution: a world model-enhanced VLA robot control method based on prediction confidence, comprising the following steps: acquiring first observation data and task instructions, wherein the first observation data indicates the state of the robot body and the working environment, and the task instructions indicate the task objective; generating a predicted state sequence based on the first observation data, wherein the predicted state sequence indicates the state of the robot at multiple future time steps; determining a confidence parameter of the predicted state sequence, wherein the confidence parameter is obtained by fusing a first confidence component and a second confidence component, wherein the first confidence component indicates the feasibility of the predicted state sequence relative to physical constraints, and the second confidence component indicates the consistency of the predicted state sequence with respect to the task instructions; fusing a first control instruction and a second control instruction according to the confidence parameter to generate target motion data, wherein the first control instruction is determined based on the predicted state sequence, and the second control instruction is determined based on the first observation data; and outputting a control signal corresponding to the target motion data, wherein the control signal is used to drive the robot's servo driver.

[0006] Further, acquiring the first observation data includes: acquiring the first observation data through at least one of the following methods: collecting the first observation data by sensors of the robot body; receiving the first observation data pushed by an external observation device via a communication link; generating and synthesizing the first observation data by a simulation engine; wherein the first observation data includes at least one of lidar point cloud data, visual image data, and force and tactile observation data.

[0007] Furthermore, the first control instruction is temporarily stored in a dynamic circular queue structure. The step of protecting the mainline control instruction with an independent storage area includes: freezing the write pointer of the dynamic circular queue structure during the injection of the probe action data to seal the mainline control instruction; opening an independent storage area isolated from the dynamic circular queue structure, pushing the probe action data into the independent storage area, and switching the instruction source of the servo driver to the independent storage area; after terminating the injection of the probe action data, releasing the independent storage area, unfreezing the write pointer, and switching the instruction source of the servo driver back to the dynamic circular queue structure.

[0008] Furthermore, after acquiring the first observation data, the robot control method further includes: performing spatiotemporal synchronization processing on the first observation data to make the first observation data consistent in terms of time reference and spatial coordinate frame; wherein, the spatiotemporal synchronization processing includes: using the sampling time of one observation in the first observation data as the alignment target, performing interpolation on the observation values ​​of the other observations between two adjacent samplings to reconstruct each observation at the same time, or using a hardware trigger signal to enable multiple observations to be acquired simultaneously; uniformly transforming each observation to a spatial coordinate frame with the base coordinate system of the robot body as the origin; and filtering out non-physical jump data that exceeds a preset amplitude range from the unified observation.

[0009] Further, the determination of the confidence parameter of the predicted state sequence includes: determining the first confidence component through a first evaluation unit; wherein, the first evaluation unit is used to determine the first confidence component based on the degree of boundary violation of the predicted state sequence relative to the physical constraints, the physical constraints including the kinematic limit constraints and spatial occupancy constraints of the robot, the degree of boundary violation being characterized by a kinematic boundary violation metric and a spatial interference metric, the kinematic boundary violation metric indicating the degree of boundary violation of the predicted joint configuration in the predicted state sequence relative to the kinematic limit constraints, and the spatial interference metric indicating the degree of interference of the predicted position in the predicted state sequence relative to the spatial occupancy constraints.

[0010] Further, the first evaluation unit is used to: calculate the sum of the kinematic boundary violation metric and the spatial interference metric step by step to obtain the comprehensive boundary violation metric for that time step; perform positive truncation and amplification on the comprehensive boundary violation metric and accumulate it over the next few time steps to obtain the cumulative boundary violation metric; determine the first confidence component that monotonically decreases with the cumulative boundary violation metric based on the cumulative boundary violation metric, so that the first confidence component takes the maximum value when both the predicted joint configuration and the predicted position are within the safe region.

[0011] Furthermore, when the predicted state sequence satisfies that the sum of the kinematic boundary metric and the spatial interference metric is not greater than zero at multiple future time steps, the first confidence component is determined to be the maximum value; when the predicted state sequence does not satisfy that the sum of the kinematic boundary metric and the spatial interference metric is not greater than zero at at least one time step, the first confidence component is determined to be the minimum value.

[0012] Furthermore, the determination of the confidence parameter of the predicted state sequence further includes: determining the second confidence component through a second evaluation unit; wherein the second evaluation unit is used to: map each time step of the predicted state sequence to a visual representation in an aligned feature space through a visual mapping operator, map the task instruction to a linguistic representation in the aligned feature space through a linguistic mapping operator, determine the directional consistency between the visual representation and the linguistic representation in the aligned feature space step by step, and determine the second confidence component based on the average of the directional consistency over the next multiple time steps; the second evaluation unit is also used to perform non-negative truncation on the directional consistency, or recalibrate the directional consistency to a preset interval, so that the second confidence component falls within a preset value range.

[0013] Further, the credibility parameter is obtained by fusing the first credibility component and the second credibility component, including: using the first credibility component as a multiplicative gating factor to weight the second credibility component after nonlinear shaping to obtain a first fused component, wherein the nonlinear shaping amplifies the contribution of high consistency and suppresses the contribution of medium and low consistency after the second credibility component undergoes a strictly monotone convex mapping; when the second credibility component exceeds a preset excitation threshold, a second fused component is obtained based on the portion of the second credibility component that exceeds the preset excitation threshold; and the credibility parameter is determined based on the sum of the first fused component and the second fused component.

[0014] Furthermore, the credibility parameter is obtained by fusing the first credibility component and the second credibility component, and further includes: using the first credibility component as a multiplicative gating factor to weight the second credibility component to obtain a first fused component; when the second credibility component exceeds a preset excitation threshold, obtaining a second fused component based on the portion of the second credibility component that exceeds the preset excitation threshold; and determining the credibility parameter based on the sum of the first fused component and the second fused component.

[0015] Further, determining the confidence parameter based on the sum of the first fused component and the second fused component includes: ensuring that the weighting coefficients of the first fused component and the second fused component satisfy a preset normalization constraint so that the confidence parameter falls within a preset value range; or, performing saturation limiting on the sum of the first fused component and the second fused component so that the confidence parameter falls within a preset value range.

[0016] Furthermore, the robot control method further includes: determining a horizon length parameter and a replanning frequency parameter based on the confidence parameter; wherein the horizon length parameter increases with the confidence parameter in a power function relationship, and the exponent of the power function relationship is less than 1, so that the growth rate of the horizon length parameter in the low value range of the confidence parameter is greater than the linear growth rate; the replanning frequency parameter switches between two values ​​based on the comparison result of the confidence parameter and the frequency switching threshold; the horizon length parameter indicates the upper limit of the number of the first control instructions that can be executed sequentially in an open loop in the instruction buffer, and the replanning frequency parameter indicates the replanning frequency of the second control instructions.

[0017] Furthermore, a high confidence threshold is preset, which is greater than the frequency switching threshold; the robot control method further includes: when the confidence parameter is greater than the high confidence threshold, using a forward append strategy to sequentially write the newly generated first control instructions into the tail of the instruction buffer to increase the field of view length parameter, and executing the first control instructions in an open loop according to the first pointer order of the instruction buffer; when the confidence parameter is not greater than the high confidence threshold and not less than the frequency switching threshold, stopping the forward append strategy and sequentially consuming the existing instructions in the instruction buffer; when the confidence parameter is less than the frequency switching threshold, clearing the first control instructions that have not yet been executed in the instruction buffer to reduce the field of view length parameter to a preset minimum value, and performing closed-loop control according to the updated second control instructions; wherein, the proportion of the effective depth of the instruction buffer occupied by the first pointer of the instruction buffer each time it advances is inversely proportional to the field of view length parameter.

[0018] Further, fusing the first control command and the second control command according to the confidence parameter includes: applying a second-order low-pass filter to the time series of the confidence parameter to obtain a smooth confidence parameter; the second-order low-pass filter recursively obtains the smooth confidence parameter of the current control cycle in each control cycle based on the confidence parameter of the current control cycle and the smooth confidence parameters of the previous two control cycles; and using the smooth confidence parameter as an interpolation coefficient to fuse the first control command and the second control command.

[0019] Furthermore, the second-order low-pass filter satisfies the following: when the confidence parameter remains constant, the smoothing confidence parameter converges to the constant value in steady state; the output of the second-order low-pass filter under a step input is monotonically convergent and without overshoot; the smoothing confidence parameter and its first-order and second-order rates of change change continuously and boundedly with the control cycle; wherein, the time scale parameter of the second-order low-pass filter is determined based on the electrical response time constant of the robot's motor.

[0020] Further, the first control command and the second control command are respectively represented by rigid body pose transformation. The step of fusing the first control command and the second control command with the smoothness confidence parameter as interpolation coefficient includes: in the rigid body pose space, using the smoothness confidence parameter as interpolation coefficient, performing pose interpolation along the geodesic connecting the two poses represented by the first control command and the second control command to obtain the target motion data; wherein, the pose interpolation is scaled by the interpolation coefficient in the tangent space where the velocity screw representation of the rigid body pose space is located, and the scaling result is mapped back to the rigid body pose space to maintain the orthogonality of the rotational components of the rigid body pose transformation throughout the interpolation process.

[0021] Further, the pose interpolation includes: determining the relative pose between the pose represented by the first control command and the pose represented by the second control command; extracting the relative pose into a velocity screw representation in the tangent space through a logarithmic mapping; scaling the velocity screw representation with the interpolation coefficient; mapping the scaled velocity screw representation back to the rigid body pose space through an exponential mapping, and combining it with the pose represented by the second control command to obtain the target motion data; such that the target motion data converges to the pose represented by the second control command when the interpolation coefficient is at its minimum value, and converges to the pose represented by the first control command when the interpolation coefficient is at its maximum value.

[0022] Furthermore, the pose interpolation further includes: extracting the pose represented by the first control command and the pose represented by the second control command into two velocity screw representations in the tangent space through logarithmic mapping; performing linear interpolation on the two velocity screw representations in the tangent space according to the interpolation coefficient to obtain a fused velocity screw representation; and mapping the fused velocity screw representation back to the rigid body pose space through exponential mapping to obtain the target motion data.

[0023] Further, outputting a control signal corresponding to the target motion data includes: constructing an impedance control equation comprising mass parameters, damping parameters, and stiffness parameters, wherein the impedance control equation defines the dynamic response characteristics of the robot's pose deviation when subjected to external contact forces; establishing a positive correlation mapping between the stiffness parameter and the reliability parameter, and updating the stiffness parameter in the impedance control equation in real time according to the reliability parameter via the positive correlation mapping; determining the pose deviation based on the target motion data and the robot's measured current pose, and converting the pose deviation into the control signal according to the updated impedance control equation, and outputting it to the servo driver; the positive correlation mapping makes the stiffness parameter a high-order convex polynomial of the reliability parameter, wherein the power of the high-order convex polynomial is greater than 1, so that the stiffness parameter decreases at a rate greater than linear as the reliability parameter decreases in the high-value range of the reliability parameter; wherein the positive correlation mapping further includes applying different adjustment ratios to the stiffness parameter in different task space directions.

[0024] Further, the step of converting the pose deviation into the control signal according to the updated impedance control equation includes: determining the damping parameter based on the real-time stiffness parameter and the real-time task space inertia of the robot, so that the impedance control equation maintains a critical damping state; applying the stiffness parameter to the pose deviation and the damping parameter to the rate of change of the pose deviation, projecting the result into joint torque through the transpose of the robot's Jacobian matrix, and superimposing a gravity compensation term and a Coriolis compensation term to obtain the control signal; wherein, the mass parameter is taken as the task space inertia of the robot.

[0025] Furthermore, the robot control method further includes: monitoring the push delay time of the predicted state sequence; when the push delay time exceeds a preset real-time communication period, blocking the determination of the confidence parameter and the generation of the target motion data through a hardware timer that runs independently of the processor software, resetting the stiffness parameter in the impedance control equation to zero, and making the control signal retain only the gravity compensation term, so that the robot enters a hovering state; after the push delay time recovers to within the preset real-time communication period, gradually restoring the stiffness parameter and the control signal, and releasing the blockage.

[0026] Furthermore, the robot control method also includes: when the robot is detected to be in a deadlock state, generating trial action data and injecting it into the servo driver for execution, and protecting the main control instructions in an independent storage area during the execution of the trial action data; wherein, the deadlock state is a state in which the statistical value of the confidence parameter in the sliding window is lower than a low confidence threshold and the movement speed of the robot end effector is lower than a preset speed threshold; during the execution of the trial action data, reacquiring the first observation data and redetermining the confidence parameter, and after the confidence parameter rises above the low confidence threshold, terminating the injection of the trial action data and restoring the main control instructions.

[0027] Further, generating trial action data includes: acquiring uncertainty characterization data output by the prediction module used to generate the predicted state sequence, the uncertainty characterization data indicating the distribution of cognitive uncertainty of the predicted state sequence in each spatial direction; performing feature decomposition on the uncertainty characterization data to determine the maximum uncertainty direction and the maximum uncertainty intensity of the cognitive uncertainty; generating trial action data along the maximum uncertainty direction and repeating over time, the amplitude of the trial action data increasing with the increase of the maximum uncertainty intensity and not exceeding a preset safety displacement limit; wherein, the uncertainty characterization data is obtained based on the sample covariance between the prediction outputs of multiple members of the prediction module, or based on the output dispersion of the prediction module after multiple random discard forward samples.

[0028] Furthermore, the generation of trial action data further includes: generating a random disturbance action in the neighborhood of the current position of the robot end effector with an absolute displacement value less than a preset safe displacement upper limit, as the trial action data; or generating a contact action in which the torque increment of the robot end effector is less than a preset torque upper limit, as the trial action data.

[0029] Another aspect of the present invention provides a world model-enhanced VLA robot control system based on prediction confidence, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the world model-enhanced VLA robot control methods based on prediction confidence as described above.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] 1. This invention evaluates the predicted state from two independent dimensions: the degree of compliance with physical constraints and the degree of compliance with task objectives. The evaluation results are integrated into a unified prediction confidence level, which improves the completeness of the prediction quality judgment. Furthermore, the same prediction confidence level data is used to simultaneously adjust the influence of predictive and reactive actions, the control horizon of the instruction execution buffer, and the replanning frequency of reactive control. When the prediction confidence level decreases, the open-loop horizon can be shrunk in a timely manner, unexecuted old instructions can be cleared, and high-frequency closed-loop control can be restored, thereby achieving a dynamic balance between computational efficiency and control safety.

[0032] 2. When the predicted data delay exceeds the limit, the present invention cuts off the failure control link and enters a safe hovering state through an independent hardware timing circuit; when the confidence level is low and the robot is stuck, it actively generates new observations through exploration actions constrained by the safety range, and uses the isolated storage area to protect the main instructions, thereby improving the system's recoverability and operational continuity under inference blocking, unknown scenarios and control deadlock conditions.

[0033] 3. This invention performs pose fusion along geodesics in rigid body pose space, strictly maintaining the orthogonality of the rotation matrix throughout the process, and using second-order low-pass filtering to ensure the continuity of output pose, velocity, and acceleration, thereby eliminating gimbal lock-up and mechanical shock during switching. When a low-confidence lock-up is detected, a safety-constrained trial micro-action is injected along the direction with the greatest cognitive uncertainty, exchanging the maximum information gain for the minimum physical cost to quickly restore confidence. The main command is protected by an isolated quiet zone, solving the pain point of embodied intelligence stopping and reporting errors when it encounters the unknown. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0035] Figure 1 This is a flowchart of the method provided in Embodiment 1 of the present invention.

[0036] Figure 2 The nonlinear penalty function curve of the credibility of physical common sense provided in Embodiment 1 of the present invention.

[0037] Figure 3 This is a schematic diagram illustrating the adjustment of the exponential decay coefficient to the severity of physical veto provided in Embodiment 1 of the present invention.

[0038] Figure 4 The time-series evolution curve of cosine similarity provided in Embodiment 1 of the present invention.

[0039] Figure 5 This is a three-dimensional surface diagram of the nonlinear plasticity characteristics provided in Embodiment 1 of the present invention.

[0040] Figure 6 This is a comparison diagram of the feedforward time step and the linear mapping provided in Embodiment 1 of the present invention.

[0041] Figure 7 This is a schematic diagram of the threshold binary step of the reprogramming frequency for reaction control provided in Embodiment 1 of the present invention.

[0042] Figure 8 This is a schematic diagram of hysteresis characteristics and hysteresis-free flutter provided in Embodiment 1 of the present invention.

[0043] Figure 9 This is a comparison chart of the smoothing effect provided in Embodiment 1 of the present invention.

[0044] Figure 10 This is a deviation attenuation response diagram provided in Embodiment 1 of the present invention.

[0045] Figure 11 This is a torque timing comparison diagram of the hardware watchdog stiffness hovering protection mechanism provided in Embodiment 1 of the present invention.

[0046] Figure 12 The simulation diagram of the overall timing response of the credibility-driven system provided in Embodiment 1 of the present invention is shown. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0048] It should be noted that, in this application, the environmental state data stream may be derived from one or more of the following combinations: visual sensors (e.g., RGB cameras, RGB-D depth cameras, binocular stereo cameras, LiDAR, or event cameras), body perception sensors (e.g., absolute encoders, incremental encoders, and inertial measurement units for each joint), and force and tactile sensors (e.g., wrist six-dimensional force / torque sensors, joint torque sensors, or fingertip tactile arrays) installed on the robot body or in the working environment; the environmental state data stream includes at least: joint state observations that can be used to characterize the robot body configuration, and visual observations that can be used to characterize the relative geometric relationship between the end effector and the working environment; optionally, the environmental state data stream may also include contact force observations, target object pose estimates, or a scene semantic representation layer calculated from multiple basic observations.

[0049] In this application, a robot refers to a multi-degree-of-freedom actuator with continuous pose control capability, which can be a serial robotic arm or parallel mechanism with a fixed base, or a dual-arm collaborative robot, a wheeled or legged mobile operation composite robot, or a humanoid robot; the forward dynamics inference module (i.e., world model) refers to a learnable state transition model that takes the current environmental state and candidate action sequences as input and the prediction of the environmental state at multiple future time steps as output; the vision-language-action (VLA) model refers to an end-to-end multimodal large model that takes visual observation and natural language commands as input and directly outputs robot action commands.

[0050] Example 1

[0051] This embodiment discloses a world model-enhanced VLA robot control method based on prediction credibility. This embodiment considers that VLA large models and world models deployed in real physical environments typically exhibit the following characteristics: prediction results exist in the form of high-dimensional tensors and cannot be directly executed by the underlying controller; the time consumed in a single inference is much longer than the servo control cycle; prediction quality fluctuates drastically with scene dynamics; hallucination commands cannot be intercepted upstream; and there is a lack of autonomous escape mechanisms during prolonged low-credibility stagnation. Furthermore, traditional control architectures call large models at a fixed frequency, which easily leads to a waste of significant GPU memory bandwidth and computing power in high-credibility scenarios, while failing to tighten the control loop in a timely manner in low-credibility scenarios. This embodiment addresses these issues by employing deterministic two-dimensional credibility quantification, macro-scale asymmetric resource scheduling, and meso-scale... The synergistic effect of manifold geodesic fusion and credibility smoothing filtering, microscale credibility-stiffness inverse mapping, and active perception micro-perturbation injection based on information gain maximization improves the safety, action continuity, computing power utilization efficiency, and self-recovery capability of robots when executing language command tasks in open dynamic environments. Figure 1 The overall method flowchart of this embodiment is shown. As can be seen from the figure, this embodiment includes the following steps:

[0052] Step 1: Obtain the current environmental state data stream collected by the robot's sensors, as well as the language instruction string containing the task objective.

[0053] Among them, the current environmental state data stream refers to the multimodal observation data sequence organized by timestamps, which is collected by the aforementioned sensors at the current control moment and its adjacent historical moments. In the spatial dimension, it describes the instantaneous geometric and physical state of the robot body configuration and the working environment, and in the temporal dimension, it retains the short-term history of state evolution, providing initial conditions for forward dynamics deduction.

[0054] A language instruction string refers to a text sequence issued by a human operator in natural language to describe a task objective; the task objective refers to the desired final state of the environment or the desired behavioral pattern pointed to by the text sequence, which does not explicitly contain any underlying motion parameters and needs to be jointly interpreted into an executable action by the VLA model and the world model.

[0055] In this embodiment, the environmental state data stream can be acquired in real time by the sensor suite built into the robot body, or it can be pushed by external observation devices (e.g., global top-down cameras) deployed in the working environment via a communication link, or it can be a synthetic observation stream generated by the simulation engine for offline verification of the scenario.

[0056] In this embodiment, to ensure that multi-source observations from different sensors with different sampling frequencies and transmission delays are strictly consistent in time axis and spatial coordinate system before entering the forward dynamics deduction, thereby avoiding false state abrupt changes introduced by time misalignment or coordinate system mismatch from being misjudged as real physical anomalies in subsequent credibility calculations, after obtaining the current environmental state data stream, it may further include:

[0057] Using the sampling clock of the joint encoder as a time reference, timestamp interpolation soft synchronization is performed for visual observation and force-tactile observation, or hardware trigger signals are used to achieve simultaneous exposure acquisition of multiple sensors.

[0058] Based on the pre-completed hand-eye calibration and sensor extrinsic parameter calibration results, the observations of each sensor are uniformly transformed into a global coordinate frame with the robot base coordinate system as the origin;

[0059] The unified observations are subjected to outlier removal and amplitude limiting verification to filter out non-physical jump data caused by sensor momentary failures.

[0060] Among them, timestamp interpolation soft synchronization refers to a method that uses the sampling time of a certain sensor as the alignment target and performs linear or spline interpolation on the observations of other sensors between two adjacent samplings, thereby reconstructing all modal observations at the same virtual time. Hand-eye calibration refers to the calibration process of solving the fixed rigid body transformation relationship between the vision sensor coordinate system and the robot end effector (or base) coordinate system. The result can usually be expressed as a... Homogeneous transformation matrix.

[0061] It should be noted that this spatiotemporal synchronization step provides a strict spatiotemporal consistency prerequisite for all subsequent calculations based on state prediction and state comparison. If this step is ignored, the initial state received by the world model itself contains cross-modal misalignment noise, and the predicted state sequence derived from it will systematically deviate from the actual evolution, causing the credibility scalar calculated in the next step 2 to be falsely suppressed, thereby causing the entire dynamic coupling system to mistakenly enter a conservative mode in a scenario where it could have been operating normally.

[0062] Step 2: Input the current environmental state data stream into the forward dynamics extrapolation module to generate the future... The predicted state sequence at each time step is evaluated deterministically from two independent dimensions: physical common sense and semantic intent. The fusion generation range is limited to [specific domain]. The predictive confidence scalar within.

[0063] Among them, the predicted state sequence refers to the future state sequence generated by the forward dynamics deduction module step by step, using the current environmental state as the initial condition and the candidate action sequence as the condition input. To the A set of environmental state estimates for each time step. To predict the total number of time steps within the horizon; each element in the predicted state sequence contains both a high-dimensional continuous latent state vector and interpretable physical quantities reconstructed by the decoder, such as future joint configurations. Future terminal Cartesian coordinates and future scene visual state .

[0064] Spatial extreme boundary refers to the extreme range of configuration envelope and position envelope touched by the predicted state sequence in joint space and Cartesian space. It is used to compare with the kinematic limits of the robot body and the spatial occupancy range of environmental obstacles, and is a direct basis for determining whether the prediction is physically feasible.

[0065] The prediction confidence scalar refers to the single scalar signal obtained by reducing and compressing a high-dimensional and unobservable prediction tensor. This value simultaneously reflects the physical rationality and semantic consistency of the prediction: the closer the value is to 1, the more trustworthy the prediction is and the more likely it is to be executed in a long-scope open-loop manner; the closer the value is to 0, the less reliable the prediction is, and the system should be reduced to conservative reactive closed-loop control. In this embodiment, the prediction quality is compressed into a scalar because the prediction results of the VLA large model and the world model usually exist in the form of high-dimensional tensors, which cannot be directly consumed by the underlying controller, nor can the core decision of whether the prediction is trustworthy be obtained directly.

[0066] In this embodiment, a prediction confidence scalar can be generated by separately constructing a physical common sense verification channel and a semantic intent alignment channel, and then nonlinearly fusing the evaluation results of the two channels into a single scalar. Specifically, the extraction and verification of the extreme boundary of the predicted state sequence space can be represented by the following sub-step 1, the calculation of the cosine similarity of the included angle in the alignment vector space can be represented by the following sub-step 2, and the deterministic algebraic operation fusion can be represented by the following sub-step 3.

[0067] Sub-step 1: Construct a physical common sense verifier and calculate the credibility of physical common sense. .

[0068] Among them, the physical common sense checker refers to a deterministic computational unit that performs physical feasibility checks on the predicted state sequence step by step, based on the rigid body kinematic limits and three-dimensional space occupancy constraints. The constraints checked by the common sense checker are: joints cannot exceed mechanical limits and rigid bodies cannot penetrate each other. These are a priori laws that are always true in the physical world without relying on any learning process.

[0069] Joint kinematic barrier function This refers to a scalar function that takes the predicted joint configuration as input. It takes a negative value when the joint configuration is within the absolute safety region, and turns from negative to positive after the configuration enters the soft limit margin zone, and increases sharply as it approaches the mechanical limit. For example, it can be constructed as the maximum overrun of each joint angle relative to its upper and lower soft limit margins.

[0070] Collision obstacle function This refers to a scalar function that takes the predicted position of the end effector and the link envelope as input. For example, it can be constructed as the inverse of the minimum sign distance between the robot envelope and the space occupied by environmental obstacles. When there is no interference, the sign distance is positive and the function value is negative. When three-dimensional spatial volume interference occurs, the function value is positive and increases with the penetration depth.

[0071] In this embodiment, to overcome the technical shortcomings of existing large models that only focus on pixel reconstruction loss and completely ignore the extreme values ​​of rigid body motion and spatial occupancy constraints in the physical world, a nonlinear penalty function is used to construct the credibility of physical common sense. When the predicted joint configuration approaches the mechanical limit, or when the predicted end trajectory interferes with the environment in three-dimensional volume, the penalty should increase sharply; while when the prediction is within the absolute safety region, the penalty should disappear completely. The specific calculation logic may include:

[0072] Analyzing the Future The predicted joint configuration is extracted step by step from the predicted state sequence at each time step. With the predicted terminal Cartesian coordinates ;

[0073] The sum of the joint kinematic obstacle function and the collision obstacle function is calculated step by step and used as a comprehensive measure of the degree of boundary violation at that time step.

[0074] Apply positive truncation and square amplification to the overall degree of boundary violation, in all The summation over each time step is then mapped using an exponential function to a confidence value that monotonically decays with the degree of violation.

[0075] For example, in this embodiment, the credibility of physical common sense can be calculated by the following formula:

[0076]

[0077] in, The credibility of physical common sense; It is a natural exponential function; This is the exponential decay coefficient, used to control the rate at which the accumulated penalty decays into confidence. The larger the value, the more drastic the decay of the credibility of physical common sense under the same degree of violation, and the more severe the physical veto. To predict the total number of time steps within the horizon, To predict a single time step within the horizon; A composite operator for positive truncation and square amplification is used to ensure that negative values ​​within the safe region do not incur any penalty, while out-of-bounds values ​​are nonlinearly amplified in square form. and The world model in the future The predicted joint configuration and distal Cartesian coordinates; For joint kinematic obstacle functions, The collision obstacle function increases sharply when the predicted state approaches the joint limit or volume interference occurs, and takes a negative value when in the absolute safe region. Figure 2 The nonlinear penalty function curve of the physical common sense credibility in this embodiment is shown. It can be seen that compared with the conventional binary gating which jumps from 1 to 0 at the moment of hard limit crossing, the physical common sense credibility in this embodiment begins to gradually decay within the soft limit margin band, which reserves a lead time for downstream horizon contraction and stiffness adjustment. Figure 3 This diagram illustrates the adjustment of the exponential decay coefficient to the severity of physical veto in this embodiment.

[0078] Understandably, the above formula constructs an asymmetric penalty function structure with zero penalty within the safe region and severe penalty after exceeding the boundary. The characteristics of this function structure are: when the predicted state is within the absolutely safe region, the sum of the two barrier functions is always negative, the positive truncation operator is completely zero, and the independent variable of the exponent term is zero. Precision equals 1, meaning the physical dimension is given complete trust; however, if any time step goes out of bounds, the out-of-bounds amount is first amplified by a squaring operation, then accumulated through summation across time steps, and finally enters the exponentially decaying term, making... As the degree of violation deepens, it rapidly approaches zero, thus negating physically infeasible predictions at the source.

[0079] It should be noted that the continuous physical reliability given in the above formula also covers the implementation of binary gating in engineering: when only a hard determination of whether the limit is exceeded is required, the above formula can be degenerated into a threshold decision form. If the sum of the obstacle functions is not greater than zero throughout the predicted state sequence (i.e., the boundary of the continuous potential state vector does not exceed the kinematic extreme value of the robot body, nor does it cause three-dimensional volume interference), then the output is... Otherwise output The advantage of this continuous implementation compared to the binary implementation in this embodiment is that the barrier function changes from negative to positive within the soft limit margin band, making... It can provide a gradual warning attenuation before the hard limit is actually exceeded, allowing for advance of downstream horizon contraction and stiffness adjustment.

[0080] Sub-step 2: Perform semantic intent alignment and calculate semantic intent credibility. .

[0081] Here, the alignment vector space refers to the same high-dimensional feature space that can be pre-established through cross-modal representation learning, into which visual observations and textual language can be mapped and whose directional consistency can be directly compared; visual feature extraction operator Text language mapping operator These refer to the encoding functions that map the predicted visual state and natural language instructions to the alignment vector space; both can be implemented using a cross-modal dual-tower encoder pre-trained through large-scale image-text contrast learning.

[0082] Cosine similarity refers to the cosine of the angle between two vectors in their inner product space. It depends only on the vector directions and is unaffected by their magnitudes. The theoretical range of its values ​​is... The closer the value is to 1, the more consistent the directions of the two vectors are.

[0083] In this embodiment, to identify predictions that are physically feasible but have deviated from the task intent—predictions that, while not triggering any physical constraints, lead to a future scenario that no longer evolves towards the goal described by human language instructions—these predictions should not be trusted. The specific operational logic includes:

[0084] Transform the predicted state sequence into a future visual representation step by step;

[0085] The future visual representation and the language instruction string are respectively transmitted via and Map to the alignment vector space;

[0086] The normalized inner product (i.e., cosine similarity) of the two feature vectors is calculated step by step over time, and then averaged over the entire prediction horizon.

[0087] For example, in this embodiment, the semantic intent credibility can be calculated using the following formula:

[0088]

[0089] in, For semantic intent credibility; For visual feature extraction operators, The future predicted by the world model The scene state of the step; For text language mapping operators, The natural language instruction strings given by humans; the visual feature extraction operator and the text language mapping operator both map heterogeneous modalities to the same high-dimensional aligned vector space. The inner product in this space; the norm product in the denominator is used for normalization, so that the metric depends only on the feature direction and is not affected by the feature magnitude. Figure 4 The figure shows the temporal evolution curve of the cosine similarity of the semantic intent credibility in this embodiment. As can be seen from the figure, the temporal stepwise averaging design in this embodiment can smooth the noise in a single frame and capture the semantic consistency trend of the entire prediction.

[0090] Understandably, this formula assesses the directional question of whether the predicted future is still evolving towards the command target, rather than the absolute strength of the visual feature response; calculating and averaging over the entire prediction horizon smooths out the occasional representation noise at a single time step, resulting in... It reflects the semantic consistency of the entire prediction, not a momentary coincidence of a particular frame.

[0091] It should be noted that, considering the theoretical range of cosine similarity values, To ensure the semantics of the input for subsequent fusion operations are clear, in this embodiment, the output of the above equation can be further truncated to non-negativity, i.e., the value is taken as... Or, the linear recalibration of the stepwise similarity may be performed. After averaging the intervals, the result is as follows: , where 0 represents that the predicted direction is completely unrelated to or deviates from the task intention, and 1 represents that it is completely consistent.

[0092] Sub-step 3: Perform deterministic nonlinear algebraic fusion on the physical common sense credibility and semantic intent credibility to generate the final prediction credibility scalar. In this embodiment, the key design consideration for fusion computation is:

[0093] First, physical credibility should be used as a veto-based multiplicative gating mechanism; predictions that are physically infeasible should not be trusted, no matter how semantically consistent they may be. Second, semantic credibility should be nonlinearly amplified through exponential shaping, making the system more sensitive to high semantic consistency and suppressing semantically mediocre predictions. Third, by superimposing a linear excitation term with a threshold, additional bonuses should only be given when semantic consistency exceeds a preset threshold.

[0094] For example, in this embodiment, the final prediction confidence scalar It can be calculated using the following formula:

[0095]

[0096] in, , These are the preset system weighting coefficients for the two fused components; It is a natural constant. It is an exponential shaping factor; The basic semantic excitation threshold constant, Permanent stability ensures that additional incentives are generated only when semantic consistency exceeds this threshold; multiplicative structure To make physical credibility act as a gate on semantic components, as long as When the value approaches zero, regardless of the semantic component, the first term is completely erased. Figure 5 A three-dimensional surface plot of the nonlinear shaping characteristics of the prediction confidence scalar in this embodiment is shown.

[0097] Understandably, the exponential shaping term It is a general Mapped to A strictly monotonically convex function: its... Take 0 at the location, in The exact value is set to 1, and due to convexity, low-to-medium level semantic consistency is suppressed after mapping, while the marginal contribution of high-level semantic consistency is amplified, thus filtering out semantically mediocre predictions. In particular, when the shaping factor... When, according to limit calculations, this term strictly degenerates into At this point, the entire fusion degenerates into a linear product form. In other words, linear fusion can be viewed as the fusion formula in... This is a special case under the limit; this fusion formula is a strict generalization of linear fusion.

[0098] It should be noted that, in order to ensure that the fusion output strictly adheres to... Within the range, the weighting coefficients can satisfy the normalization constraint. Alternatively, a saturation limiting operation can be added to the fused output, resulting in a scalar of prediction confidence. This scalar becomes a unified signal source that runs through all subsequent control levels. It provides a unique decision input for horizon reconstruction, frequency scheduling and manifold interpolation in step 3, and stiffness adjustment in step 4, thereby realizing a signal-driven, full-stack deterministic coupled architecture.

[0099] Step 3: Based on the prediction confidence scalar calculated in Step 2, dynamically reconstruct the horizon length parameter of the instruction execution buffer and the VLA response control reprogramming frequency on a macroscopic time / frequency scale, and then apply this to the robot's three-dimensional special Euclidean group. Within the manifold space, using the smoothed and filtered confidence level as the interpolation coefficient, dynamic Lie algebra geodesic interpolation is performed on the predictive control command matrix output by the forward dynamics deduction module and the reactive control command matrix triggered by the current state to generate the target motion trajectory matrix.

[0100] The instruction execution buffer refers to a first-in-first-out storage structure used to temporarily store the future multi-step predictive control instructions output by the forward dynamics deduction module; the horizon length parameter (also known as the feedforward time step) The buffer is the maximum number of instructions that can be executed sequentially in an open-loop manner, which directly determines how far the system can proceed without consulting the larger model again.

[0101] The predictive control command matrix refers to the action commands planned by the world model based on the deduction of future states. In this embodiment, it is represented by the homogeneous transformation matrix of the end pose. Characterization; the reactive control command matrix refers to the instantaneous intuitive action commands directly generated by the VLA model based on the latest observations, also represented by a homogeneous transformation matrix. Characterization; Open-loop execution refers to the execution mode of executing existing instructions sequentially according to the buffer pointer without waiting for new sensing feedback; Closed-loop execution refers to the execution mode of regenerating or correcting instructions based on the latest observations for each control cycle.

[0102] In this embodiment, this step may specifically include the following sub-steps:

[0103] Sub-step 1: Asymmetric resource scheduling at the macroscopic time / frequency scale.

[0104] In this embodiment, to eliminate the timing blockage caused by the fixed asynchronicity between the high-computation inference module and the high-frequency response module, the traditional architecture always calls the large model at a fixed rhythm regardless of the prediction quality. This wastes a lot of GPU memory bandwidth and computing power in high-confidence scenarios and fails to tighten the control loop in time in low-confidence scenarios. Therefore, an asymmetric response logic is constructed, the specific operation logic of which includes:

[0105] The feedforward open-loop horizon is designed as a power function of the confidence level with an exponent of less than 1. This allows the horizon to be aggressively stretched as soon as the confidence level increases slightly, thus fully realizing the computing power benefits brought by high-confidence prediction.

[0106] The VLA reprogramming frequency is designed as a threshold-based binary switching. Once the confidence level falls below the threshold, the full-frequency closed loop is immediately and forcibly restored, ensuring that the safe response has no gradual delay.

[0107] For example, in this embodiment, the feedforward time step can be calculated using the following formula:

[0108]

[0109] The reaction control reprogramming frequency of the VLA model can be expressed as follows:

[0110]

[0111] in, This is the feedforward time step; , These are the preset shortest and longest open-loop horizons, respectively; This is a floor operator that ensures the time step is a discrete integer. This is the curvature adjustment constant (e.g., it can be 0.5); For the response control reprogramming frequency of the VLA model; The reliability threshold for frequency switching; This is an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise. As a frequency reduction factor, when the confidence level is not lower than the threshold At that time, the replanning frequency of the VLA model was reduced to the baseline frequency. times. Figure 6 A comparison diagram of the feedforward time step and the linear mapping in this embodiment is shown; Figure 7 A schematic diagram of the threshold binary step of the reaction control reprogramming frequency in this embodiment is shown.

[0112] Understandably, the horizon formula and the frequency formula together constitute an asymmetric response logic that is asymmetrically elongated and instantaneously tightened. Regarding the horizon, since... power function It is an upwardly convex function, and its slope in the low confidence interval is significantly greater than that of a linear mapping: when the confidence level... When it rises slightly, This results in growth that exceeds linear proportions, and the system consequently radically extends its open-loop horizon. To release the computing resources occupied by the high-frequency closed loop as soon as possible and fully realize the computing power dividend brought by high-reliability prediction; in terms of frequency, the judgment conditions of the two indicator functions are mutually exclusive and complete, ensuring that only one frequency mode is effective at any given time; and the switch is a binary switch without a gradual process, once the reliability falls below the threshold The reprogrammed frequency was immediately forced to revert to full frequency. This reduces the gradual delay of security response; in other words, computing power savings are realized gradually along a smooth curve, while security tightening is completed instantaneously along a step path. This asymmetry is the mathematical expression of the principle of efficiency giving way to security design at the resource scheduling level, which completely eliminates the timing blockage caused by the fixed asynchrony between the high computing power inference module and the high frequency response module.

[0113] In this embodiment, the specific operation of the above-mentioned horizon reconstruction logic at the instruction execution buffer level includes:

[0114] Define the length of the instruction execution buffer as the feedforward time step. ;

[0115] Set a high confidence threshold Frequency switching threshold and low confidence threshold All three conditions are met. The order relationship;

[0116] when At that time, a forward append strategy is adopted to expand. The numerical value is determined by the forward dynamics derivation module, which appends newly generated predictive control commands sequentially to the end of the buffer after each round of derivation, thus increasing the effective depth of the buffer towards the formula. The indicated target value increases; simultaneously, the sampling frequency for generating the reactive control command matrix decreases to the fundamental frequency according to the aforementioned frequency formula. The system executes the predictive control instruction matrix in an open-loop manner based on the first pointer of the buffer.

[0117] when When this happens, stop adding forward, maintain the existing buffer depth and consume the queue stock in sequence, and maintain the replanning frequency at [a certain level]. ;

[0118] when At this time, the unexecuted subsequent queue in the instruction execution buffer is forcibly cleared, so that... Immediately reduce to the set minimum extreme value and with full frequency Closed-loop execution is based on the latest observed and updated reactive control command matrix. Figure 8 The diagram illustrates the hysteresis characteristics of instruction execution buffer scheduling in this embodiment, along with a schematic diagram of hysteresis-free chatter.

[0119] It should be noted that the expansion threshold and the clearing threshold are designed as follows: The separate layout of the buffer creates a natural hysteresis zone. Aggressive expansion of the buffer only occurs after the confidence level exceeds a higher threshold, and forced clearing of the buffer only occurs after the confidence level falls below a lower threshold. The intermediate zone between the two simply maintains the status quo. If this hysteresis zone is not set, and expansion and clearing are triggered simultaneously by a single threshold, the system will oscillate frequently between the two modes when the confidence level fluctuates slightly around the threshold, resulting in computational waste caused by repeated filling and invalidation of the buffer. In addition, forced clearing is designed as an unconditional hard action. Its purpose is that: a confidence level falling below the threshold means that the entire previously deduced future is no longer reliable. If the remaining outdated predictive instructions in the buffer continue to be consumed in an open loop, it is equivalent to making the robotic arm execute a trajectory that has been determined to be unreliable. Therefore, it must be immediately and completely invalidated. This is a direct manifestation of the safety-first principle at the data structure operation level. It should also be noted that the minimum threshold... It does not participate in the horizon scheduling of this sub-step. It is specifically used for the determination of low-confidence deadlock state in subsequent step 5. It is predefined here to maintain the unity and self-consistency of the whole text threshold system.

[0120] Sub-step 2: Apply a second-order low-pass difference equation filter to the time series of the prediction confidence scalar to generate smooth interpolation coefficients.

[0121] Among them, second-order low-pass differential equation filtering refers to discretizing the characteristics of second-order low-pass filtering in the continuous-time domain into a recursive difference equation that depends only on the current input and the historical outputs of the previous two cycles. This allows for digital filtering operations that can be directly executed recursively step-by-step within the discrete control cycle; control cycle. This refers to the fixed time interval between two consecutive instruction updates from the underlying controller; the hardware time constant. This refers to the filter timescale parameters, which are tuned based on the electrical response time constant of the robotic arm motor. These parameters determine how quickly the filter output responds to changes in the input.

[0122] In this embodiment, to eliminate the problem of high-frequency oscillations in the prediction confidence scalar caused by scene transients, sensor noise, or model output jitter, which are then directly transmitted to the action command end via interpolation weights, thereby triggering mechanical resonance and mechanical impact, the time series is processed before proceeding to the manifold interpolation operation in sub-step 3. The value is filtered using a second-order low-pass differential equation to smooth it into a slowly varying signal suitable for use as geometric interpolation weights. The specific operational logic includes:

[0123] The second-order low-pass transfer function in the continuous domain backward difference transformation Discretize into digital difference equations;

[0124] In each discrete control phase Internally, it only relies on the current original credibility input. The smoothing reliability of the current beat is recursively calculated using the historical smoothed outputs of the previous two beats. ;

[0125] by Replace the original scalar , which are used as smoothed interpolation coefficients in all subsequent interpolation operations.

[0126] For example, in this embodiment, the second-order low-pass differential filter can be represented by the following equation:

[0127]

[0128]

[0129] in, The smoothness and reliability of the filtered output; For the first The original credibility input for each control shot; To control the cycle, The hardware time constant and the two together precisely define all the coefficient values ​​of the difference equation, referring to the general formula. In this embodiment, the hardware matching filter coefficients are as follows: , , . Figure 9 The diagram shows a comparison of the smoothing effect of the second-order low-pass differential equation filter on reliability jitter in this embodiment. The three sub-plots in the diagram share the same X-axis and respectively show the influence of the time constants τ / Ts=5, τ / Ts=10, and τ / Ts=20 on the smoothing degree. As can be seen from the diagram, the second-order low-pass differential equation can effectively filter out high-frequency reliability jitter, and the unity DC gain characteristic ensures that the filter does not change the steady-state semantics of reliability. The larger τ is, the more thorough the smoothing. The matching tuning strategy with the hardware time constant is intuitively visible.

[0130] It is understandable that this difference equation is the transfer function of a continuous second-order system. The backward differential discretization is implemented as follows: the two-phase recursive terms of the historical output act as low-pass memory, continuing the existing smooth trend forward; the weights of the input terms... This determines the rate at which new information is injected. relatively The larger, The smaller the value, the smoother the filtering and the slower the response; conversely, the larger the value, the more sensitive the filtering, thus filtering out high-frequency jitter in reliability at the instruction level.

[0131] It should be noted that the above coefficient set possesses three mathematical properties indispensable for interpolation coefficient filters. First, unity DC gain: the sum of the three coefficients satisfies... Therefore, the constant confidence input is transmitted completely without decay in steady state (i.e., The filtering process removes high-frequency jitter without altering the steady-state semantics of reliability. This is a rigid constraint that the interpolation coefficient filter must satisfy. If the DC gain is not 1, the constant high-reliability input will be systematically suppressed, causing the fusion result to be permanently biased towards reactive commands, rendering the predictive channel ineffective. Secondly, unconditional stability and critical damping: the filter characteristic equation. It can be organized into Its double real pole For any positive and The filter is unconditionally stable because it is always located within the unit circle; moreover, the double real poles mean that the output is monotonically convergent under a step input, with no overshoot and no ringing, and the interpolation coefficients will not oscillate around the target value. Thirdly, output continuity: due to... Both its first and second order differences evolve smoothly and boundedly over time. The fused pose sequence generated by these interpolation coefficients remains continuous in terms of velocity and acceleration, meaning that the physical curves of the velocity and acceleration ultimately output to the robotic arm's end effector remain consistent. Continuity is maintained to avoid Jerk shocks caused by sudden shifts in credibility. Furthermore, the time constant... Based on the electrical response time constant tuning of the robotic arm motor (e.g., taking a multiple of the equivalent time constant of the current loop), so that... The maximum rate of change is always within the frequency band that the servo system can track smoothly, while the high-frequency jitter band that is filtered out is precisely the frequency band that the actuator cannot track effectively, but is enough to excite mechanical resonance. Therefore, the above coefficients are called hardware matching filter coefficients.

[0132] Sub-step 3: In Dynamic Lie algebraic interpolation of predictive and reactive control commands is performed along the geodesic on the manifold to generate the target motion trajectory matrix.

[0133] Among them, the three-dimensional special Euclidean group This refers to a rigid body pose transformation group consisting of both three-dimensional rotation and three-dimensional translation, whose elements are... Homogeneous transformation matrix, top left corner The sub-block is a rotation matrix that satisfies orthogonality constraints and has a determinant of 1. (Top right corner) The sub-block is a translation vector; the end effector pose of the robotic arm is composed of both rotation and translation, naturally existing on this nonlinear manifold; Lie algebra It means, The tangent space of the group at the identity element, whose elements are six-dimensional velocity twisters containing three-dimensional angular velocity components and three-dimensional linear velocity components; logarithmic mapping This means to Group element mapping as Operations on spinor coordinates; exponential mapping This refers to its inverse operation, remapping the spinor back to the pose manifold. A geodesic is a local shortest path connecting two points on a manifold. The above corresponds to a smooth spiral motion trajectory driven by constant spin.

[0134] In this embodiment, to avoid the problems of traditional Euclidean scalar linear weighting (e.g., linear addition of positions, addition of Euler angle components) disrupting the orthogonality of rotation matrices, causing gimbal lock (i.e., the degradation of Euler angle representation by losing a rotational degree of freedom in a specific attitude), or unnatural rotation paths when switching between two sets of control commands, this embodiment abandons Euclidean interpolation and strictly performs action fusion in the Lie group Lie algebra space. Its specific computational logic includes:

[0135] Extract the homogeneous transformation matrix corresponding to the predictive control command respectively (i.e., in the claims) (The same below) Homogeneous transformation matrix corresponding to the reactive control command ;

[0136] Calculate the relative pose between the two The relative pose is extracted as a velocity spinor through Lie algebra-logarithmic mapping;

[0137] The smoothness confidence level output by sub-step 2 As a scalar scale, the spinor is scaled in the tangent space;

[0138] Remapping the scaled spinor back using exponential mapping group, and with The composite pose is obtained.

[0139] Exemplarily, in this embodiment, The manifold geodesic fusion can be calculated using the following formula:

[0140]

[0141] in, These are the homogeneous transformation matrices representing VLA intuitive actions (reactive control commands) and world model-derived actions (predictive control commands), respectively. From observe The resulting relative pose; for The Lie algebraic logarithmic mapping is used to extract spinors containing angular velocity and linear velocity components; for The exponential mapping of the scaled spinor remaps the spinor back to the pose manifold. As the interpolation ratio, when it is 0, the fusion result degrades precisely to... When its value is 1, the fusion result converges precisely to The intermediate value corresponds to the intermediate pose on the geodesic line connecting the two poses. This is obtained from... This is the target homogeneous transformation pose matrix that is the only legal target within the current control cycle and maintains the spatial manifold characteristics, which is also the target motion trajectory matrix in the step statement.

[0142] Understandably, the core of this formula lies in the fact that the scaling operation occurs in the tangent space (Lie algebra) rather than on the manifold itself: the relative spinor is first stretched until it is proportionally truncated in the linear tangent space, and then bent back to the manifold through exponential mapping. Therefore, the fusion path is naturally the shortest geodesic between two poses, which physically corresponds to a smooth spiral motion around a fixed spiral axis. The orthogonality of the rotation matrix and the determinant constraint are strictly maintained by the mathematical structure itself throughout the process, thus eliminating the problem of orthogonality destruction in Euclidean interpolation from the root.

[0143] In another alternative implementation, logarithmic mapping can be performed on the two absolute poses separately to obtain the velocity spinor. and Then, deterministic linear interpolation is performed in the Lie algebra space. Then through exponential mapping Remapping back group.

[0144] It is understood that this optional implementation, like the aforementioned relative pose implementation, belongs to the Lie algebraic interpolation paradigm of tangent space interpolation-exponential back-projection, and both strictly maintain the orthogonality of the rotation matrices. When the spinors of the two poses satisfy the commutative condition (e.g., rotation or pure translation about the same axis), the fused poses given by the two implementations are strictly consistent. In general, the aforementioned relative pose implementation gives the exact shortest geodesic path, while the latter is its equivalent first-order implementation in tangent space coordinates; both fall within the scope of this application. The scope of dynamic Lie algebra interpolation computation that performs logarithmic and exponential mapping interpolation within the Lie algebra space.

[0145] It should be noted that the geodesic interpolation in this sub-step and the second-order smoothing filter in sub-step 2 form a complementary design: the filter ensures the smoothness of the interpolation coefficients in the time dimension, and the geodesic interpolation ensures the manifold validity of the interpolation path in the spatial dimension. The two work together to ensure that the pose, velocity and acceleration of the robotic arm are continuous throughout the entire process when the control is transferred between the VLA intuitive action and the world model deduced action. This completely solves the problem of spatial pose distortion of the robotic arm caused by traditional Euclidean space interpolation and the Jerk impact problem at the moment of switching.

[0146] Step 4: Construct the underlying robot variable impedance control equations that include mass parameters, damping parameters, and stiffness parameters. Establish a positive correlation mapping function between stiffness parameters and prediction confidence scalars. Update the variable impedance control equations in real time through the mapping function, and convert the target motion trajectory matrix into underlying joint torque control electrical signals for output to the servo driver.

[0147] Among them, the variable impedance control equation refers to the equation of the form... The second-order objective dynamic equation defines the behavior of the robotic arm end effector under contact forces from the external environment. Its trajectory position deviation during operation The dynamic response characteristics that should be presented; For the target quality matrix, Let be the target damping matrix. The stiffness matrix is ​​in Cartesian space. The first derivative of the trajectory position deviation. This is the second derivative of the trajectory position deviation. The trajectory position deviation refers to the target motion trajectory matrix output in step 3. The deviation between the indicated desired pose and the actual pose measured by the sensor is, in The above can be represented as a six-dimensional minimum displacement spinor through the logarithmic mapping of the relative poses of the two. Stiffness refers to the magnitude of the elastic restoring force corresponding to a unit pose deviation: the higher the stiffness, the more rigid and accurate the robot arm's tracking of the reference trajectory; the lower the stiffness, the softer the robot arm, and the easier it is to be pushed away from the reference trajectory by external forces without generating resistance.

[0148] In this embodiment, to address the extreme risk of a large model generating illusions and issuing erroneous commands that cannot be intercepted upstream, the purpose of this step is proactive compliant safety: rather than pursuing perfect interception at the command level, the physical properties of the robotic arm itself change with confidence level. When confidence level is high, the robotic arm maintains high stiffness to ensure trajectory tracking accuracy; once confidence level decreases, stiffness drops rapidly, making the robotic arm softer. At this point, even if an erroneous command is executed, the actuator will be directly bounced off by external physical forces due to its extremely low stiffness, rather than forcibly pushing through the environment. Thus, the cognitive uncertainty of the high-dimensional large model is directly reduced in dimension and transmitted to the lowest-level task space stiffness matrix. Its specific computational logic includes:

[0149] Construct a high-order convex polynomial positive correlation mapping of the stiffness matrix with respect to credibility, and achieve the asymmetric property of a slight decrease in credibility resulting in a cliff-like drop in stiffness.

[0150] The damping matrix is ​​dynamically recalculated based on the real-time stiffness and the real-time task space inertia, so that the system always maintains the critical damping state and prevents the robot arm from oscillating due to the variable stiffness process itself.

[0151] The stiffness and damping of the task space are applied to the pose deviation and its rate of change, projected back to the joint torque via the Jacobian transpose, and the gravity and Coriolis effect are compensated to form a complete impedance control law.

[0152] For example, in this embodiment, the nonlinear stiffness adjustment operator can be calculated using the following formula:

[0153]

[0154] in, , These are the pre-stored task space compliant minimum diagonal matrix and rigid maximum diagonal matrix, respectively; Adjustment coefficients in each direction The resulting diagonal matrix is ​​used to apply differentiated stiffness adjustment ratios to different task space directions (e.g., in contact operations, a smaller component is taken along the contact normal to obtain a more compliant normal response). For convex polynomial factors (e.g., can be taken as) This formula is a specific implementation of the positive correlation mapping function of the claim, corresponding to... :because about Strictly monotonically increasing with all adjustment coefficients positive, when the prediction confidence scalar During descent, the stiffness matrix The diagonal elements and even their traces decrease synchronously, forcing the physical representation of the robotic arm when faced with unreliable predictions to be converted into a highly compliant avoidance state.

[0155] Understandably, due to power function The slope is extremely large in the high confidence interval: For example, when When it slightly decreased from 1.0 to 0.9, It has dropped from 1.0 to approximately 0.729, with the stiffness increment term instantly losing nearly 30%; when When it drops to 0.7, The stiffness increment has dropped by nearly two-thirds; in other words, as long as the credibility... Even a slight decrease can cause a precipitous drop in the rigidity of the robotic arm. The design intent is to allow the robotic arm to enter a compliant state in advance when the large model is about to experience illusions but is not yet fully defined, thereby preventing motor overload or environmental damage caused by forcibly executing incorrect commands.

[0156] For example, in this embodiment, the critical damping automatic matching mechanism can be represented by the following formula:

[0157]

[0158] in, To determine the desired damping ratio, by taking... To maintain the system in a critically damped state; The inertial matrix of the manipulator in mission space varies with joint configuration. Real-time changes, in non-singular configurations, can be determined by the joint space inertia matrix. Calculated using Jacobi mapping The square root operation in the formula, at the implementation level, when... and When all elements are approximately diagonal, scalar root extraction can be performed on each diagonal element. In general, the symmetric positive definite square root of the product matrix can be taken. Figure 10 The diagram shows the deviation attenuation response of the critical damping automatic matching mechanism and the fixed damping step response in different scenarios in this embodiment. In the figure, (a) is the scenario of sudden increase in stiffness and (b) is the scenario of sudden decrease in stiffness.

[0159] Understandably, this equation represents the critical damping condition for a classical second-order system. Matrix generalization in the task space: Real-time changes in stiffness will synchronously alter the system's inherent oscillation characteristics (natural frequency varies with...). (drift), therefore damping It must be based on real-time stiffness With real-time inertia Dynamic recalculation, if the damping remains constant, will cause the system to slip into an underdamped state and generate damped oscillations when the stiffness drops sharply, and will enter an overdamped state and have a sluggish response when the stiffness rises sharply, thus preventing the mechanical arm from oscillating due to the variable stiffness process itself.

[0160] For example, in this embodiment, the mission space impedance control torque projection can be calculated using the following formula:

[0161]

[0162] in, This is the final command torque issued to each joint; For the transpose of the Jacobian matrix of the robotic arm, it is responsible for projecting the generalized forces in the task space back into the joint space; and Let be the trajectory position deviation in the task space and its first derivative, respectively. The target motion trajectory matrix output from step 3 Obtained by comparison with the measured current pose; This is a gravity compensation term; For the Coriolis and centrifugal force compensation term, here The Coriolis matrix in robot dynamics should be related to the aforementioned prediction confidence scalar. Strict distinctions must be made.

[0163] It should be noted that, in this embodiment, the target mass matrix in the variable impedance control equation... Directly obtain the robot's own task space manipulator inertia matrix This means that no artificial shaping of inertia is performed. Its engineering benefit lies in eliminating the need for real-time measurement of end-contact force, which is essential for inertial shaping. The target impedance response defined by the equation can be achieved solely through attitude deviation feedback. At this point, the stiffness and damping terms within the square brackets of the above equation are fully realized. The defined target dynamics; Subsequently, each joint servo driver converts the motor torque constant into a torque control current command, which is the underlying joint torque control electrical signal mentioned in the steps, and is executed by the driver current loop in a closed loop.

[0164] It should be noted that the stiffness and damping terms within the square brackets in the above equation are both confidence levels. The function implies that cognitive uncertainty at the large model level is transmitted via... and It is directly bound to the lowest-level physical execution torque, achieving end-to-end connectivity from cognitive confidence to physical compliance; this connectivity does not rely on any additional intermediate decision logic, so there is no possibility of interception failure. Even if an illusionary instruction penetrates all upstream software barriers and is actually executed, its physical consequences are naturally limited by the stiffness of the synchronous fall.

[0165] In addition, in this embodiment, in order to deal with extreme scenarios of memory overflow and high-frequency communication blockage, that is, the forward dynamics inference module is unable to push prediction data within hard real-time constraints due to GPU memory overflow, inference queue backlog or communication link congestion, a hardware decoupling protection mechanism based on a hardware watchdog timer is also included.

[0166] Among them, the hardware watchdog timer refers to a hardware timing circuit that runs independently of the main processor software stack. It needs to be periodically reset by the software, and if it is not reset within a timeout period, it will automatically trigger a preset protection action. Because its triggering does not depend on the software itself, which may be blocked, it has an independent fallback capability in the event of a software failure. The real-time interrupt bus cycle refers to the maximum cycle allowed for a servo communication bus (e.g., real-time industrial buses such as EtherCAT) to refresh one frame of control data under the real-time scheduling framework of the operating system.

[0167] In this embodiment, the specific execution logic of the hardware decoupling mechanism includes:

[0168] The delay time of the forward dynamics inference module pushing the predicted state sequence within the monitoring unit time window. For example, it can be calculated in real time by appending a sending timestamp to each frame output of the simulation module and having the control side compare the receiving time.

[0169] If the judgment If the real-time interrupt bus cycle is exceeded, the hardware watchdog timer is started, forcibly blocking the execution of steps 2 and 3. At this time, any confidence and fusion instructions calculated based on outdated or incomplete predictions have lost their timeliness, and continuing to execute will introduce uncontrollable risks. Figure 11 A torque timing comparison diagram of the hardware watchdog stiffness hovering protection mechanism in this embodiment is shown.

[0170] The last known point of the target motion trajectory matrix from the previous control cycle is used as the adsorption target. At the same time, the stiffness parameter in the variable impedance control equation is instantaneously reset to zero, so that only the gravity compensation term is retained in the torque output. With the Coriolis compensation, the actuator enters a pure hovering safety mode. Since the stiffness is zero, the robotic arm no longer generates any active restoring force toward the adsorbed target. It maintains the current configuration hovering solely by the gravity compensation algorithm. The end effector can be freely pushed by external forces but will never move actively.

[0171] Continuous monitoring of latency Once the current time bus cycle is restored, the stiffness and instruction tracking are gradually restored through the smoothing filtering and geodesic interpolation mechanism in step 3. Then the watchdog timer is released to avoid a momentary torque jump during recovery.

[0172] Understandably, the hardware decoupling property of this mechanism is reflected in the fact that the protection action is triggered by an independent hardware timer. Even if the main control software stack becomes completely unresponsive due to memory overflow, the watchdog can still autonomously push the system into a zero-stiffness hovering state, ensuring that the safety fallback does not depend on the faulty software itself. The instantaneous return of stiffness to zero can also be regarded as the stiffness mapping in step 4. exist The hardware-forced version of extreme behavior. Figure 12 The following is a simulation diagram of the overall timing response of the credibility-driven system in this embodiment. The diagram contains five horizontally aligned sub-graphs, with the X-axis uniformly aligned to the time axis (0~10s). From top to bottom, they are: Sub-graph ① represents the original C and... The timing diagram; sub-diagram ② shows the real-time changes in the feedforward horizon; sub-diagram ③ shows the step switching of the VLA replanning frequency; sub-diagram ④ shows... The real-time change graph; sub-graph ⑤ is a comparison graph of end position tracking error.

[0173] Step 5: When the system is detected to be in a low-confidence double-blind deadlock state, the active exploratory perception micro-action injection mechanism is triggered. The most valuable micro-probing actions are generated under the guidance of information theory to actively break the deadlock, and the main trajectory instructions are protected with an independent memory quiet zone during the exploration.

[0174] Among them, low-confidence double-blind deadlock refers to a mutually locked state in which the system is in a low-confidence state for a long time and the robotic arm is stuck in a state of stagnation because it dares not move: on the one hand, the robotic arm is stationary, which means that new observations cannot be updated in a substantial way and the confidence cannot be repaired by new observations; on the other hand, the persistently low confidence makes the robotic arm dare not resume operation. These two factors are mutually causal, and the system will stagnate indefinitely without external intervention.

[0175] Predicting cognitive covariance matrix This refers to the second-order statistical characterization of the cognitive uncertainty of the world model's prediction of the current state, which describes the distribution of the degree of unknowns in various spatial directions; for example, It can be obtained by estimating the sample covariance between the predicted outputs of each member in the model ensemble, or by estimating the output scatter through multiple forward samplings in Monte Carlo Dropout.

[0176] Eigenvalue decomposition refers to solving for eigenvalues ​​that satisfy... The matrix decomposition operation of all eigenvalues ​​and eigenvectors; since the covariance matrix is ​​symmetric and positive semi-definite, its largest eigenvalue is... The maximum intensity of cognitive uncertainty, and the corresponding principal feature vector. This represents the most uncertain spatial direction in the current world model. A tentative micro-motion parameter sequence refers to a sequence of tiny reciprocating motion commands whose amplitude is subject to safety constraints and can be terminated at any time.

[0177] In this embodiment, the mechanism unfolds in two steps: The first step is to accurately determine the deadlock state. This is designed because a low confidence level in a single shot might just be transient noise. Only when the confidence level remains consistently low within a sliding window, and the end-effector velocity simultaneously approaches zero (i.e., the robotic arm has indeed stopped), can the system be considered deadlocked, thus preventing trial actions from interfering with normal operation. The second step is to decide which direction to probe: Compared to blindly trying to explore with random perturbations, directional exploration using the principal component directions of the covariance matrix can achieve the greatest information gain with minimal physical intervention.

[0178] For example, in this embodiment, the deadlock trigger conditioner can be represented by the following formula:

[0179]

[0180] in, This serves as a trigger flag for the trial mode. The number of sampling steps within the sliding window, i.e., the number of consecutive claims. One control cycle, For the most recent Shoot a moving average of credibility; The threshold for low confidence level determination is the lowest threshold in the three-threshold system set in step 3. Let the norm of the end effector speed be denoted as . This is the speed threshold for determining stagnation. It should be noted that both conditions must be met simultaneously to trigger a trial: if only the confidence level is low while the endpoint is still moving, it indicates that the system is relying on reactive full-frequency closed-loop operation and should not be disturbed; if only the endpoint is stagnant while the confidence level is normal, it may just be a natural pause in the task; only when both pieces of evidence of persistently low confidence and actual stagnation are present can a deadlock be identified, thus ensuring that the normal operation process is not disturbed. Compared to single-shot determination, the moving average further filters out false triggers caused by instantaneous noise.

[0181] In this embodiment, the execution flow after the triggering condition is met specifically includes:

[0182] Pause the main task pointer of the reactive control instruction matrix, that is, freeze the instruction consumption and advancement of the main task queue;

[0183] Perform eigenvalue decomposition on the predicted covariance matrix output by the world model. Find the largest eigenvalue and its corresponding principal feature vector ;

[0184] A sequence of tentative micro-motion parameters is generated based on the following tentative vector formula and injected into the servo driver for execution;

[0185] While performing the exploratory micro-actions, step 1 is forced to resynchronize and sample the latest environmental state data stream and proceed to step 2 to recalculate the prediction confidence scalar. ;

[0186] treat When the sliding average rises above the threshold, the trial injection is terminated, the suspended mainline task pointer is restored, and the mainline control is seamlessly switched back via the frequency scheduling mechanism and geodesic fusion mechanism in step 3.

[0187] For example, in this embodiment, the trial vector based on maximizing information gain can be calculated using the following formula:

[0188]

[0189] in, The largest eigenvalue of the covariance matrix represents the maximum intensity of cognitive uncertainty; For the corresponding main feature vector, Its normalized direction vector; The ratio of the uncertainty intensity to the experimental amplitude; The preset safe displacement limit is the physical safety limit of the absolute value of the claimed displacement. outer layer The calculations ensure that the probe amplitude will not exceed the safety boundary, regardless of the level of cognitive uncertainty. To test the vibration frequency, The probing action manifests as symmetrical reciprocating micro-vibrations around the current position, with naturally continuous and bounded velocity and acceleration, and terminates at any full cycle time, returning to the original position, and can be safely canceled at any time.

[0190] It is understandable that conducting observational probes along the principal component with the greatest cognitive uncertainty can achieve the greatest information gain (i.e., the fastest decay of information entropy) with the least physical intervention cost, thereby increasing credibility. The system recovers as quickly as possible; in contrast, random directional perturbations have a limited probability of hitting directions with high uncertainty, and their expected information gain per unit displacement is far lower than that of main direction orientation exploration, often requiring multiple rounds of trial and error to be effective. Once credibility is restored, the system can seamlessly switch back to VLA master control via the aforementioned frequency scheduling model and geodesic fusion model, thereby solving the core pain point of traditional embodied intelligence crashing and reporting errors when encountering the unknown.

[0191] In simplified implementations where computational resources are limited or cognitive covariance estimation is unavailable, the tentative micro-motion parameter sequence can also be hard-coded as follows: the robotic arm end effector generates a displacement with an absolute value less than the physical safety limit within its local neighborhood in its current spherical coordinate system. The position is randomly perturbed; or configured with a torque increment of less than The distal end of the device makes a light touch to obtain tactile information that is not visually accessible through the force-tactile channel. The above simplification sacrifices the optimality of information gain, but retains the basic ability to break the double-blind deadlock, and thus falls within the scope of the insured concept of this application.

[0192] In this embodiment, the instruction execution buffer can be implemented using a dynamic circular queue data structure to support the protection of the mainline instructions during the exploration period.

[0193] Among them, the dynamic circular queue refers to a circular first-in-first-out (FIFO) structure in which the head and tail are logically connected, and the storage space is reused cyclically by modulo operations between the head pointer (read pointer) and the tail pointer (write pointer); its dynamic nature is reflected in the effective depth of the queue following the feedforward time step output in step 3. Real-time scaling and high-priority quiet zones refer to independent high-priority storage segments allocated in the memory stack that are physically isolated from the main circular queue.

[0194] In this embodiment, the operational logic of the data structure specifically includes:

[0195] The normalized offset of the first pointer within a single control beat (i.e., the ratio of the advance in a single beat to the effective depth of the queue, approximately...) (and the feedforward time step output in step 3) They are inversely proportional. The larger the queue size, the deeper the queue, the smaller the relative queue size consumed by single-cycle execution, and the stronger the system's open-loop continuous execution capability.

[0196] During the triggering of the proactive exploratory perception micro-action injection mechanism, the tail pointer write operation of the dynamic circular queue is frozen, so that the main line's predictive control instruction inventory is sealed as is.

[0197] A separate high-priority quiet zone is allocated in the memory stack to push the sequence of tentative micro-action parameters, and the source of servo command consumption is switched to this quiet zone during the exploration period.

[0198] After the exploration is complete, release the quiet zone, unfreeze the tail pointer write, and switch the servo command consumption source back to the mainline dynamic circular queue.

[0199] It should be noted that the significance of the quiet zone isolation design is to ensure that the memory space of the main trajectory instructions is not contaminated or overwritten by dirty data during physical exploration. If the trial instructions and the main instructions share the same queue, the write operations during the trial will overwrite the existing main instructions that have not yet been executed. After the exploration ends, the main task will not be able to resume in place due to the loss of instructions. The physically isolated quiet zone eliminates this pollution path at the data structure level, making the entire process of pause-explore-resume completely transparent to the main task.

[0200] In summary, the basic implementation scheme of this embodiment is as follows: A world model-enhanced VLA control system is constructed, using a prediction confidence scalar as the unified signal source. This scheme first establishes a strictly consistent spatiotemporal coordinate premise for multi-source heterogeneous observations through a spatiotemporal synchronization module. Then, the high-dimensional unobservable prediction tensor output by the world model is reduced and compressed into a range domain limited to a specific area by a multimodal confidence quantization channel. This is achieved through a two-dimensional evaluation aligned with physical common sense nonlinear penalty functions and semantic intent cosines, followed by nonlinear multiplicative gating fusion. The deterministic reliability scalar is used within the system; then, at the macro scale, asymmetric resource scheduling is constructed using power function horizon mapping and binary frequency switching, coupled with a three-threshold hysteresis buffer forward appending and forced clearing strategy to eliminate fixed asynchronous timing blocking between the high-computing power inference module and the high-frequency response module; at the meso scale, the reliability is first smoothed into slowly varying interpolation coefficients using a second-order low-pass differential filter with unity DC gain, unconditional stability, and critical damping characteristics, and then... The geodesic along the manifold performs Lie algebraic logarithmic-exponential mapping interpolation on predictive and reactive commands to ensure the pose legitimacy and velocity / acceleration continuity throughout the control transfer process. Furthermore, at the microscale, a high-order convex polynomial is used to positively correlate credibility to the task space stiffness matrix. The damping matrix is ​​dynamically recalculated using critical damping conditions, and then projected as joint torque electrical signals via Jacobi transpose, gravity, and Coriolis compensation, achieving end-to-end connectivity from cognitive confidence to physical compliance. Finally, a hardware watchdog zero-stiffness hovering protection is used to address extreme scenarios of inference latency and communication blockage, and a memory quiet zone protection mechanism breaks low-credibility double-blind deadlock, thus forming a complete closed-loop control system that balances computational efficiency, action continuity, physical security, and autonomous escape capability in an open dynamic environment.

[0201] Example 2

[0202] This embodiment discloses a world model-enhanced VLA robot control system based on prediction credibility.

[0203] Specifically, the world model-enhanced VLA robot control system based on prediction credibility can be integrated into an electronic device, which can be a terminal, server, or other such device. The terminal can be a robot body control cabinet, a real-time industrial control computer equipped with a graphics processing unit (GPU), an edge computing device, a workstation, a laptop, or a personal computer, etc. The server can be a single server, a server cluster composed of multiple servers, a deep learning computing node equipped with a GPU acceleration card, or a distributed computing cluster or cloud inference platform with GPU acceleration capabilities. When the electronic device is running, it can realize the world model-enhanced VLA robot control method based on prediction credibility of Embodiment 1 of this application.

[0204] In this embodiment, the system can also be integrated into multiple electronic devices. Considering the difference in computing power between the forward dynamics inference module and the VLA model and the hard real-time requirements of the underlying servo control, the multimodal reliability quantification module and the inference operations of the world model and VLA model can be deployed on a high-performance computing node equipped with a GPU. The dynamic coupling and manifold fusion module and the variable impedance physical execution module can be deployed on the real-time controller on the robot body side. The two work together via a real-time communication bus to realize the world model-enhanced VLA robot control method based on prediction reliability in Embodiment 1 of this application. At this time, the push delay monitoring and hardware watchdog mechanism in Embodiment 1 are applied to the cross-device communication link to provide physical fallback for communication anomalies in the distributed deployment mode.

[0205] In this embodiment, the server can also be implemented in the form of a terminal.

[0206] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A world model-enhanced VLA robot control method based on prediction reliability, characterized in that, The robot control method includes: Acquire first observation data and task instructions, wherein the first observation data indicates the state of the robot body and the working environment, and the task instructions indicate the task objective; Using the state indicated by the first observation data as the initial condition and the candidate action sequence as the condition input, a predicted state sequence is generated step by step through the forward dynamics deduction module. The predicted state sequence indicates the state of the robot in multiple future time steps. A confidence parameter is determined for the predicted state sequence. The confidence parameter is obtained by fusing a first confidence component and a second confidence component. The first confidence component indicates the feasibility of the predicted state sequence relative to the physical constraints, and the second confidence component indicates the consistency of the predicted state sequence with respect to the task instructions. The first control command and the second control command are fused according to the confidence parameter to generate target action data. The first control command is determined based on the predicted state sequence, and the second control command is determined based on the first observation data. Output a control signal corresponding to the target motion data, the control signal being used to drive the robot's servo driver.

2. The world model-enhanced VLA robot control method based on prediction reliability according to claim 1, characterized in that, After acquiring the first observation data, the robot control method further includes: Perform spatiotemporal synchronization processing on the first observation data to make the first observation data consistent in time reference and spatial coordinate frame; The spatiotemporal synchronization processing includes: using the sampling time of one of the first observation data as the alignment target, interpolating the observation values ​​of the remaining observations between two adjacent samplings to reconstruct each observation at the same time, or using a hardware trigger signal to enable simultaneous acquisition of multiple observations; uniformly transforming each observation to a spatial coordinate frame with the base coordinate system of the robot body as the origin; and filtering out non-physical jump data that exceeds the preset amplitude range from the unified observations.

3. The world model-enhanced VLA robot control method based on prediction reliability according to claim 1, characterized in that, The determination of the confidence parameter of the predicted state sequence includes: The first confidence component is determined by the first evaluation unit; The first evaluation unit is used to determine the first confidence component based on the degree of boundary violation of the predicted state sequence relative to the physical constraints. The physical constraints include the robot's kinematic limits and spatial occupancy constraints. The degree of boundary breach is characterized by a kinematic boundary breach metric and a spatial interference metric. The kinematic boundary metric indicates the degree to which the predicted joint configuration in the predicted state sequence exceeds the kinematic limit constraint. The spatial interference metric indicates the degree of interference between the predicted position in the predicted state sequence and the spatial occupancy constraint.

4. The world model-enhanced VLA robot control method based on prediction reliability according to claim 3, characterized in that, The determination of the confidence parameter of the predicted state sequence further includes: The second confidence component is determined by the second evaluation unit; The second evaluation unit is used for: The predicted state sequence is mapped to visual representations in the aligned feature space by a visual mapping operator, and the task instructions are mapped to linguistic representations in the aligned feature space by a linguistic mapping operator. The directional consistency between the visual representations and the linguistic representations in the aligned feature space is determined step by step, and the second confidence component is determined based on the average of the directional consistency of the next multiple time steps. The second evaluation unit is also used to perform non-negative truncation on the direction consistency, or to recalibrate the direction consistency to a preset interval so that the second confidence component falls into a preset value range; The credibility parameter is obtained by fusing the first credibility component and the second credibility component, including: Using the first confidence component as a multiplicative gate factor, the second confidence component after nonlinear shaping is weighted to obtain the first fusion component. The nonlinear shaping amplifies the contribution of high consistency and suppresses the contribution of medium and low consistency after the second confidence component is subjected to a strictly monotone convex mapping. When the second confidence component exceeds a preset excitation threshold, a second fusion component is obtained based on the portion of the second confidence component that exceeds the preset excitation threshold; The confidence parameter is determined based on the sum of the first fusion component and the second fusion component.

5. The world model-enhanced VLA robot control method based on prediction reliability according to claim 1, characterized in that, The robot control method further includes: The horizon length parameter and the replanning frequency parameter are determined based on the confidence parameter. Wherein, the horizon length parameter increases with the increase of the confidence parameter in a power function relationship, and the power exponent of the power function relationship is less than 1, so that the growth rate of the horizon length parameter in the low value range of the confidence parameter is greater than the linear growth rate. The replanning frequency parameter switches between two values ​​based on the comparison result between the confidence parameter and the frequency switching threshold; The horizon length parameter indicates the upper limit of the number of the first control instructions that the instruction buffer is allowed to execute sequentially in an open-loop manner, and the replanning frequency parameter indicates the replanning frequency of the second control instructions.

6. The world model-enhanced VLA robot control method based on prediction reliability according to claim 1, characterized in that, The step of fusing the first control command and the second control command according to the confidence parameter includes: A second-order low-pass filter is applied to the time series of the confidence parameter to obtain a smooth confidence parameter. The second-order low-pass filter recursively obtains the smooth confidence parameter of the current control period in each control period based on the confidence parameter of the current control period and the smooth confidence parameters of the two previous control periods. The smoothness confidence parameter is used as an interpolation coefficient to fuse the first control command and the second control command; The second-order low-pass filter satisfies: When the confidence parameter remains constant, the smooth confidence parameter converges to the constant value in steady state. The second-order low-pass filter has a monotonically convergent output with no overshoot under a step input. The smoothing reliability parameter and its first and second rates of change change change continuously and boundedly with respect to the control period. The time scale parameter of the second-order low-pass filter is determined based on the electrical response time constant of the robot's motor.

7. The world model-enhanced VLA robot control method based on prediction reliability according to claim 1, characterized in that, The output control signal corresponding to the target motion data includes: An impedance control equation is constructed that includes mass parameters, damping parameters, and stiffness parameters. The impedance control equation specifies the dynamic response characteristics of the robot's pose deviation when subjected to external contact forces. Establish a positive correlation mapping between the stiffness parameter and the confidence parameter, and update the stiffness parameter in the impedance control equation in real time according to the confidence parameter through the positive correlation mapping; The pose deviation is determined based on the target motion data and the robot's measured current pose, and the pose deviation is converted into the control signal based on the updated impedance control equation and output to the servo driver. The positive correlation mapping makes the stiffness parameter a high-order convex polynomial of the confidence parameter, and the power of the high-order convex polynomial is greater than 1, so that the stiffness parameter decreases at a rate greater than linear as the confidence parameter decreases in the high value range of the confidence parameter. The positive correlation mapping further includes applying different adjustment ratios to different task space directions of the stiffness parameter.

8. The world model-enhanced VLA robot control method based on prediction reliability according to claim 1, characterized in that, The robot control method further includes: When the robot is detected to be in a deadlock state, trial action data is generated and injected into the servo driver for execution, and the main control instructions are protected in an independent storage area during the execution of the trial action data; The deadlock state is defined as a state in which the statistical value of the confidence parameter within the sliding window is lower than the low confidence threshold and the movement speed of the robot end effector is lower than the preset speed threshold. During the execution of the probe action data, the first observation data is reacquired and the confidence parameter is redefined. After the confidence parameter rises above the low confidence threshold, the injection of the probe action data is terminated and the mainline control command is restored.

9. The world model-enhanced VLA robot control method based on prediction reliability according to claim 8, characterized in that, The generation of trial action data includes: Acquire uncertainty characterization data output by the prediction module used to generate the predicted state sequence, wherein the uncertainty characterization data indicates the distribution of cognitive uncertainty of the predicted state sequence in each spatial direction; Perform feature decomposition on the uncertainty characterization data to determine the maximum uncertainty direction and the maximum uncertainty intensity of the cognitive uncertainty; Generate trial action data that repeats over time along the direction of maximum uncertainty, wherein the amplitude of the trial action data increases with the increase of the intensity of the maximum uncertainty but does not exceed a preset safe displacement limit; The uncertainty characterization data is obtained based on the sample covariance between the prediction outputs of multiple members of the prediction module, or based on the output distribution of the prediction module after multiple random discard forward samples.

10. A world model-enhanced VLA robot control system based on prediction reliability, characterized in that, The robot control system includes: processor; A memory storing a computer program that, when executed by a processor, implements the world model-enhanced VLA robot control method based on prediction confidence as described in any one of claims 1 to 9.

Images