VLA multi-state early stop control method, system and device based on multi-modal perception

Abstract

The application provides a VLA multi-state early stop control method, system and device based on multi-modal perception, and relates to the technical field of industrial automation. The method comprises the following steps: collecting multi-modal perception data streams in an operating space environment in real time. The multi-modal perception data streams are input into a pre-feature extraction network of a VLA model, and representation alignment and modal fusion are performed to generate a dimension-reduced intermediate state feature manifold. The intermediate state feature manifold is input into a multi-state early stop classifier for dynamic evaluation. The classifier outputs a discrete state code representing a current fine-tuning state, and the discrete state code comprises a first state code, a second state code and a third state code. An upper-layer discretization control framework listens to the discrete state code in real time, and performs corresponding asynchronous routing control according to the discrete state code. While maintaining high-frequency control output, operation overshoot and deadlock stagnation are eliminated, and the control precision, computing power efficiency and deployment flexibility of an embodied large model in a precision industrial scene are improved.

Description

Technical Field

[0001] This invention relates to the field of industrial automation technology, and in particular to a VLA multi-state early stop control method, system and equipment based on multimodal sensing. Background Technology

[0002] With the evolution of embodied intelligence and large model technology, end-to-end vision-language-action (VLA) models have been widely studied and deployed in multi-task generalization operations of robots. However, in high-precision flexible physical interaction scenarios with extremely small tolerances (such as millimeter-level precise alignment of cloth), existing continuous end-to-end motion control paradigms still have the following significant technical bottlenecks: Peak error and accuracy degradation: Existing continuous control strategies lack explicit hard-stop constraints. When the pose error between the end effector and the target object reaches the critical minimum of the accuracy tolerance, the robot arm is unable to immediately terminate motion generation within the current control cycle due to the output inertia of the implicit timing characteristics of the model. This easily leads to small overshoots exceeding the accuracy tolerance, causing deviations in the converged pose state. In severe cases, the entire operation process needs to be re-executed.

[0003] It is prone to falling into the trap of ineffective oscillation: Under extreme observation conditions (such as high reflectivity and no texture), the model's reciprocating fine-tuning strategy is very likely to degenerate into high-frequency positive and negative alternating deadlock oscillation, that is, the end effector continues to move back and forth in a small amount in the local space without being able to converge to the target pose, causing the system to be unable to cross the physical alignment threshold for a long time, which seriously affects the efficiency and stability of operation.

[0004] The sequential logic and the upper-level scheduling framework are disconnected: existing VLA large models are usually highly coupled black boxes that continuously output continuous action flows. They lack standardized interaction interfaces with the upper-level logic framework, making it difficult to embed them into modern industrial-grade discrete task scheduling systems (such as behavior trees) in a lightweight manner, which limits their flexible deployment and collaborative control capabilities in complex processes.

[0005] Furthermore, the end-to-end inference latency of large VLA models is typically in the range of hundreds of milliseconds to seconds, making it difficult to complete the full inference of each frame within a high-speed servo control cycle of 500Hz. This results in the actual frequency of continuous control commands being issued being much lower than the update frequency of the servo driver, and there is a significant time scale mismatch between the decision delay on the inference side and the execution frequency on the control side. Summary of the Invention

[0006] The purpose of this invention is to provide a VLA multi-state early stop control method, system, and device based on multimodal perception. By using a bypass multi-state early stop classifier to directly perform three-state classification from the intermediate state feature manifold, and combining an asynchronous intelligent routing architecture, it achieves peak transient hard stop locking, local optimal oscillation active escape, and standard decoupling of continuous control flow and discrete scheduling framework. While maintaining high-frequency control output, it eliminates operational overshoot and deadlock, significantly improving the control accuracy, computing efficiency, and deployment flexibility of embodied large models in precision industrial scenarios.

[0007] In a first aspect, the present invention provides a VLA multi-state early stopping control method based on multimodal sensing, comprising the following steps: S1: Real-time acquisition of multimodal sensing data streams within the operating space environment; the multimodal sensing data streams include real-time physical state parameters of the robotic arm; S2: Input the multimodal sensing data stream into the pre-feature extraction network of the VLA model, perform representation alignment and modality fusion, and generate a dimension-reduced intermediate feature manifold; S3: Input the intermediate state feature manifold into a multi-state early stopping classifier for dynamic evaluation. The classifier outputs a discrete state code representing the current fine-tuning state. The discrete state code includes a first state code, a second state code, and a third state code. S4: The upper-layer discretized control framework monitors the discrete status codes in real time and executes corresponding asynchronous routing control based on the discrete status codes. When the first status code is received, the continuous action decoding and servo output of the VLA model are maintained; When the second status code is received, the VLA decoding thread is suspended and a hard stop command is sent to the lower-level servo controller to achieve zero speed maintenance; When the third status code is received, the current continuous control flow is cut off and the secondary fine-tuning strategy is activated.

[0008] Preferably, the intermediate state feature manifold is obtained as follows: The features output after the representation alignment and modality fusion are input into the VLA model backbone network for encoding, and the hidden state features of the penultimate layer of the backbone network are extracted as intermediate feature manifolds. A sliding time window is introduced to cascade the intermediate state feature manifolds of the current time t and its historical steps to construct a high-dimensional time-series feature manifold sequence, which serves as the input to the subsequent classifier.

[0009] Preferably, the multi-state early stopping classifier is constructed using a three-layer perceptron; the three-layer perceptron integrates a normalization layer, uses Gaussian error linear units as activation functions, and uses a normalized exponential function as the output layer.

[0010] Preferably, the upper-level discretization control framework includes a behavior tree engine.

[0011] Preferably, the first status code is used to indicate a release status; the second status code is used to indicate a peak cutoff status; and the third status code is used to indicate an abnormal escape status.

[0012] Preferably, the criteria for determining the peak cutoff state are: the position error between the end effector and the target object is less than or equal to the error threshold, the derivative sign of the position error is reversed within the timing window, and the torque change rate in the real-time physical state parameters exceeds the torque change rate threshold. The criteria for determining the abnormal escape state are: within a continuous time window of a preset duration, the motion direction command symbol of the end effector alternates between positive and negative at least N times, and the rate of change of the mean of the intermediate state feature manifold within the time window is less than the rate of change threshold.

[0013] Preferably, the multi-state early stopping classifier is optimized using a hybrid loss function, which is: Where L is the mixture loss function, Represents the multi-class cross-entropy loss used to supervise classification accuracy; One-hot encoding for the real label; The probability of the i-th class output by the classifier; To constrain the temporal smoothing regularization term for abrupt changes in adjacent prediction probabilities; Adjustment coefficient ; Let be the probability distribution at the current moment. This represents the probability distribution at the previous moment.

[0014] Preferably, the secondary fine-tuning strategy includes: At the input end, a local image region of a preset size is extracted from the original image, centered on the projection pixel coordinates of the current robotic arm end position in the image. The local image region is then upsampled and re-input into the VLA model. At the output end, the motion commands output by the VLA model are multiplied by a preset scaling factor for gain compression.

[0015] Secondly, this application provides a VLA multi-state early stop control system based on multimodal sensing, used to execute the method described in any one of the embodiments of this application, the system comprising: The multimodal data acquisition interface is used to connect to multi-view cameras and the physical state sensors of the robotic arm to acquire multimodal sensing data streams in the operating space environment in real time. Memory, which stores computer programs; A processor that, when executing the computer program, implements the steps of the method described above.

[0016] Preferably, the processor is configured with a multi-state early stopping classifier node, a VLA inference master node, and a secondary fine-tuning recovery strategy node, wherein the multi-state early stopping classifier node, the VLA inference master node, and the secondary fine-tuning recovery strategy node are each encapsulated as independent asynchronous ROS 2 nodes.

[0017] Thirdly, the present invention provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the embodiments of the present invention.

[0018] Compared with existing technologies, the beneficial effects of this invention include at least the following: This invention utilizes a bypass multi-state early-stop classifier. Without altering the continuous inference structure of the VLA master model, it extracts implicit multimodal feature manifolds and performs three-state classification evaluation. Hard stop is triggered the instant the pose error between the end effector and the target object reaches the accuracy tolerance threshold, achieving zero-velocity maintenance. This mechanism fundamentally eliminates pose deviation and overshoot caused by control inertia in end-to-end continuous inference models, locking the positioning accuracy at the optimal value. It is suitable for operational scenarios with stringent pose accuracy requirements, such as precision assembly and flexible object manipulation.

[0019] This invention bypasses the complete VLA action decoding stream and directly distinguishes between normal progression, peak arrival, and deadlock states from the intermediate feature space using a lightweight classifier (inference latency <1ms), significantly improving the system's computational efficiency and control frame rate. Compared to traditional schemes that run complete VLA inference in every frame, this invention significantly reduces the computational requirements while maintaining a high-frequency control output of 500Hz, enabling the method to be deployed on computationally limited embedded platforms.

[0020] By using a multi-state early stopping classifier to perceive abnormal escape states in real time, combined with a secondary fine-tuning adaptive strategy (such as dynamically adjusting the visual receptive field and action gain scale), the continuous inference model is endowed with the autonomous decision-making ability to break local optima. When unfavorable observation conditions such as strong reflection and low texture in the workspace cause the VLA model to get stuck in deadlock, the system can automatically detect the abnormal state and activate the escape branch, break the oscillating deadlock, and re-enter the normal operation loop. This significantly improves the system's self-recovery robustness in complex industrial environments and reduces the need for manual intervention.

[0021] A decoupled asynchronous intelligent routing architecture is constructed, successfully abstracting the underlying black-box action flow into control signals that can be directly monitored by industrial-grade discrete scheduling frameworks through a standardized status code topic interface. The VLA model can be seamlessly integrated into industrial-grade control frameworks such as behavior trees and finite state machines without internal modification, significantly reducing the deployment difficulty and integration cost of embodied intelligence models in real-world industrial scenarios, and has promising prospects for industrial applications.

[0022] By combining the operational accuracy limit (position error and torque jump) with deadlock oscillation characteristics, a rigorous three-state cutoff condition is defined, and a hybrid loss function with a temporal smoothing regularization term is proposed. This effectively suppresses the "glitch" oscillation of action boundary state switching, making the decision boundary of the classifier evolve smoothly in the time dimension. The output probability distribution will not change drastically due to small noise or feature fluctuations, making the decision process more stable and reliable. This avoids the cumulative impact damage to the robotic arm transmission mechanism caused by the high-frequency switching between control commands and emergency stop. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the VLA multi-state early stop control method based on multimodal sensing according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall process of the VLA multi-state early stop control method based on multimodal sensing according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the online asynchronous intelligent routing decision-making process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the hardware and software decoupling architecture according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the offline training and manifold extraction process of the multi-state early stopping classifier in an embodiment of the present invention. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0026] Example 1, refer to Appendix Figure 1 Appendix Figure 2 Appendix Figure 3 , Figure 1 This is a schematic diagram of the VLA multi-state early stop control method based on multimodal sensing according to the present invention. Figure 2 This is a schematic diagram of the overall process of the VLA multi-state early stop control method based on multimodal sensing of the present invention. Figure 3 This is a schematic diagram of the online asynchronous intelligent routing decision-making process of the present invention; the method of this embodiment will be described in detail below with reference to the accompanying drawings.

[0027] This embodiment provides a VLA multi-state early stopping control method based on multimodal sensing, including the following steps: S1: Real-time acquisition of multimodal perception data streams within the operating space environment; including visual data streams, task-driven natural language commands, and real-time physical state parameters of the robotic arm itself; S2: Input the multimodal perception data stream into the pre-feature extraction network of the VLA model (Vision-Language-Action Model) to perform representation alignment and modality fusion, and generate a dimension-reduced intermediate feature manifold; S3: Input the intermediate state feature manifold into a multi-state early stopping classifier for dynamic evaluation. The classifier outputs a discrete state code representing the current fine-tuning state. The discrete state code includes a first state code, a second state code, and a third state code. S4: The upper-layer discretized control framework monitors the discrete status codes in real time and executes corresponding asynchronous routing control based on the discrete status codes. When the first status code is received, the continuous action decoding and servo output of the VLA model are maintained; When the second status code is received, the VLA decoding thread is suspended and a hard stop command is sent to the lower-level servo controller to achieve zero speed maintenance; When the third status code is received, the current continuous control flow is cut off and the secondary fine-tuning strategy is activated.

[0028] In one possible implementation, the representation alignment and modality fusion includes: Features of each modality of data in the multimodal sensing data stream are extracted respectively; By aligning and fusing the extracted modal features through a cross-modal attention mechanism, a unified multimodal embedding representation is generated.

[0029] In one possible implementation, the intermediate state feature manifold is obtained as follows: The features output after the representation alignment and modality fusion are input into the VLA model backbone network for encoding, and the hidden state features of the penultimate layer of the backbone network are extracted as intermediate feature manifolds. A sliding time window of length τ is introduced, and the intermediate state feature manifolds of the current time t and its historical steps are concatenated to construct a high-dimensional time-series feature manifold sequence as the input of the subsequent classifier.

[0030] In one possible implementation, the multi-state early stopping classifier is constructed using a three-layer perceptron; the three-layer perceptron integrates a layer normalization layer, uses a Gaussian error linear unit as the activation function (GeLU), and uses a normalized exponential function (Softmax) as the output layer.

[0031] In other embodiments of the present invention, the three-layer perceptron can also be equivalently replaced by a lightweight Transformer encoder or a support vector machine (SVM) or other algorithm model that can achieve fast temporal classification. It can also achieve three-state classification output of intermediate feature manifold without affecting the realization of the core inventive concept of the present invention.

[0032] In one possible implementation, the first status code is used to indicate a release status, the second status code is used to indicate a peak cutoff status, and the third status code is used to indicate an abnormal escape status.

[0033] In one possible implementation, the upper-layer discretized control framework employs a discrete task scheduling mechanism to monitor the discrete status codes in real time and execute asynchronous routing control based on the discrete status codes. The discrete task scheduling mechanism is preferably a behavior tree engine, but can also be a finite state machine or a hierarchical state diagram.

[0034] This embodiment uses a dual-arm humanoid robot performing a fabric alignment operation as an example for illustration, but the present invention is not limited to this and can also be applied to other robot operation scenarios that require precise pose control.

[0035] The system hardware used in this embodiment is as follows: a 16-DOF dual-arm humanoid robot (or a 14-DOF robotic arm in other embodiments) and multi-view cameras (mounted on the hands and chest respectively) are used to collect visual data streams of the operation scene. The underlying servo controller is equipped with an RT-PREEMPT real-time kernel, and the servo driver is connected to the controller via an EtherCAT bus. The control frequency is set to 500Hz to ensure the real-time performance and determinism of high-frequency motion output. The image data collected by the multi-view cameras is transmitted to the upper computing platform via GigE Vision or USB 3.0 interface.

[0036] See attached document Figure 4 , attached Figure 4 This is a schematic diagram of the hardware-software decoupling architecture of the present invention. The present invention adopts a hardware-software decoupling architecture. The upper layer uses a behavior tree engine to uniformly schedule business logic, responsible for listening to discrete state signals and triggering corresponding routing decisions. The lower algorithm layer encapsulates the VLA inference master node, multi-state early-stop classifier node, and secondary fine-tuning recovery strategy node as independent asynchronous ROS2 nodes. Each node communicates through ROS2's topic and service mechanisms, thereby achieving logical decoupling between the high-frequency action flow and the upper-layer discrete scheduling. The behavior tree engine obtains the current fine-tuning state in real time by subscribing to the status code topics published by the classifier nodes.

[0037] In this embodiment, the multimodal sensing data stream specifically includes, for example: Visual data stream: RGB images of the operating scene are captured in real time by multi-view cameras mounted on the robot's hands and chest. The image resolution is 640×480 pixels, and the frame rate is 30fps. The multi-view setup ensures all-around observation of the fabric area, avoiding occlusion problems caused by a single viewpoint.

[0038] Task-driven natural language instructions: such as "align the fabric edge to the target area" or "stretch the fabric 5 mm to the left" and other high-level semantic instructions, are obtained through a speech recognition module or text input interface.

[0039] The real-time physical state parameters of the robotic arm body include the angular position, angular velocity, and joint torque of each joint, as well as the Cartesian space pose (position and attitude) of the end effector. These parameters are read back in real time from the servo driver via the EtherCAT bus at a frequency of 500Hz.

[0040] Multimodal data are aligned using a timestamp synchronization mechanism to ensure that the data from different modalities at the same time are strictly time-corresponding.

[0041] The multimodal sensing data stream is input into the pre-feature extraction network of the VLA model for representation alignment and modality fusion to generate a dimension-reduced intermediate feature manifold.

[0042] In this embodiment, the VLA model adopts a vision-language-action multimodal model based on the Transformer architecture. Specifically: Multi-view RGB images are used to extract visual features via a Vision Transformer encoder; natural language commands are used to extract semantic features via a language encoder (such as BERT or T5); and the physical state parameters of the robotic arm are used to extract proprioceptive features via a Multilayer Perceptron (MLP) encoder. The features from these three modalities are then aligned and fused through a cross-modal attention mechanism to generate a unified multimodal embedding representation.

[0043] Specifically, the cross-modal attention mechanism employs multi-head cross-attention, using the linguistic features as the query vector and the visual and proprioceptive features as the key and value vectors, respectively. A scaled dot product attention method is used to calculate the relevance score between the query vector and each key vector. This score, after Softmax normalization, serves as the weight, and the value vectors are then weighted and summed to obtain the aligned cross-modal feature representation. Using the linguistic features as the query vector signifies retrieving the most relevant feature components from the visual and physical states, guided by semantic information, to the current task instruction. Before being input into the cross-modal attention mechanism, the visual, linguistic, and proprioceptive features are mapped to the same dimension through independent linear projection layers to ensure compatibility in cross-modal attention computation. The multi-head cross-attention method uses multiple attention heads for parallel computation, and the outputs of each head are concatenated and output through a linear layer.

[0044] Subsequently, the features of the penultimate hidden state layer of the VLA backbone network are extracted as an intermediate feature manifold. Compared to the final output layer, this layer retains richer intermediate semantic information, while its dimensionality is significantly reduced, making it suitable as input to a lightweight classifier.

[0045] To capture temporal dynamics, a sliding temporal window of length τ is introduced (for example, τ=10, which includes the current time step and the previous 9 historical steps). The feature manifolds of the current time step t and its historical steps are concatenated to construct a high-dimensional temporal feature manifold sequence, which serves as the input to the subsequent multi-state early stopping classifier.

[0046] The intermediate state feature manifold is input into a multi-state early stopping classifier for dynamic evaluation, and the classifier outputs a discrete state code representing the current fine-tuning state.

[0047] In this embodiment, the multi-state early stopping classifier is constructed using a three-layer perceptron (MLP), with a LayerNormalization layer inserted between each layer for normalization. The activation function is GeLU. The classifier's final output uses a Softmax function to determine the probability distribution, corresponding to the probability values ​​for each of the three states. The classifier performs one inference operation on the temporal feature manifold for each sliding window, with the inference latency controlled to within 1 millisecond.

[0048] The classifier outputs a discrete state code representing the current fine-tuning state, the discrete state code including: First status code (status 0): Release status; Second status code (status 1): Peak truncation status; Third status code (status 2): Abnormal escape state.

[0049] The upper-level discretization control framework monitors the discrete status codes in real time and executes corresponding asynchronous routing control based on the discrete status codes.

[0050] In this embodiment, the upper-layer discretization control framework uses a behavior tree engine. The behavior tree engine runs as an independent ROS2 node, subscribing to the "status code" topics published by the multi-state early stopping classifier node to obtain the current fine-tuning state in real time, and triggering the corresponding routing branch based on the status code.

[0051] State 0: Release – Maintaining Reasoning When the behavior tree engine receives status code 0 (allowed state), it does not interrupt the current computation flow, allowing the VLA model to continue continuous motion decoding and servo output. At this time, the VLA model sends joint position / velocity commands to the lower-level servo controller, and the robotic arm continues to perform its operation task.

[0052] State 1: Peak cutoff - Hard stop lock The classifier outputs status code 1 (peak truncation state) the instant the target object reaches its optimal alignment. Upon capturing this state, the behavior tree engine immediately sends a high-priority preemptive signal, suspending the VLA decoding thread and issuing a hard stop command for the joint position to the lower-level servo controller, achieving zero-speed hold. This mechanism locks the robotic arm the instant the manipulated object reaches its optimal pose, effectively preventing deformation and overshoot of the manipulated object due to inertia in the end-to-end model.

[0053] State 2: Abnormal Escape - Secondary Fine-tuning Redirection When the VLA model gets stuck in a local optimum deadlock due to strong reflections, lack of texture, or other adverse conditions in the workspace, the classifier outputs status code 2 (abnormal escape state). Upon receiving this, the behavior tree engine automatically cuts off the current continuous control flow and seamlessly jumps to activate the "secondary fine-tuning" node.

[0054] The training and labeling rules for the multi-state early stopping classifier are as follows: This embodiment employs offline pre-training and online inference. During the offline preparation phase, a large amount of multimodal trajectory data for cloth operations is first collected, including various scenarios such as successful alignment, overstretching, and oscillation. Each trajectory is then automatically and objectively labeled time-by-time. Labeling rules for label 0 (release status): When the edge of the fabric is far from the target area (e.g., position error Δx > 2 mm), or when the position error within the area shows a significant monotonically decreasing trend and the motion gradient remains stable (i.e., in the stable stretching stage), it is labeled as status 0.

[0055] Labeling rule for label 1 (peak truncation state): When the fabric edge position error Δx ≤ ε (ε = 0.5 mm in this embodiment), and the derivative sign of the position error changes from negative to positive within the time window (i.e., the error changes from decreasing to increasing, indicating that the optimal alignment point has been exceeded), accompanied by a rapid jump in the joint torque characteristics (torque change rate exceeds 10 Nm / s), it is labeled as state 1.

[0056] Labeling rules for label 2 (abnormal escape state): When the end effector action exhibits high-frequency alternating positive and negative oscillations within a continuous time sequence (such as the movement direction command sign reversing more than 3 times within 0.5 seconds), and the mean change rate of the intermediate state feature manifold approaches zero (i.e., the implicit features stagnate in a macroscopic statistical sense), it indicates that the VLA model has fallen into a visual blind spot or local optimum deadlock, and is labeled as state 2.

[0057] The effects of the above technical solution are as follows: This embodiment simultaneously acquires visual data streams, natural language commands, and real-time physical state parameters of the robotic arm, and performs representation alignment and modal fusion via a VLA model pre-network, fully leveraging the complementary advantages of multi-source heterogeneous information. The visual data stream provides global and local spatial information of the operational scenario, the natural language commands provide high-level task constraints and semantic guidance, and the physical state parameters provide real-time force / position information of the interaction between the end effector and the environment. The fusion of these three elements generates an intermediate feature manifold rich in semantic and physical information, providing a more comprehensive feature foundation for subsequent classification decisions and significantly improving perceptual robustness in complex operational scenarios.

[0058] By bypassing the complete VLA action decoding stream through a lightweight multi-state early-stop classifier, three-state classification evaluation is performed directly from the intermediate-state feature space, achieving inference latency as low as milliseconds. Compared to traditional schemes that run complete VLA inference in every frame, this embodiment significantly reduces computational overhead while maintaining a high-frequency control output of 500Hz. Furthermore, by encapsulating the classifier, VLA inference master node, and secondary fine-tuning recovery strategy node as independent asynchronous ROS2 nodes, and combining this with an upper-layer discrete task scheduling mechanism to monitor status codes and trigger routing in real time, asynchronous decoupling of control logic and inference computation is achieved, ensuring high frequency and determinism of the underlying control.

[0059] By capturing peak truncation states in real time through a multi-state early stop classifier, and coordinating the high-priority preemptive signals of the upper-level discrete control framework with the lower-level servo hard stop commands, the joint achieves zero-velocity maintenance the instant the target object reaches its optimal pose. This mechanism fundamentally eliminates the deformation of the manipulated object and pose overshoot caused by control inertia in end-to-end continuous inference models, locking the end-effector positioning accuracy at the optimal value. It is particularly suitable for fine-grained operation scenarios with stringent edge accuracy requirements, such as fabric alignment.

[0060] By abstracting the underlying high-frequency control flow into a state code topic interface that can be directly listened to by the upper-level discrete scheduling framework, a standardized bridge is built between continuous action flow and discrete task scheduling. The VLA model can be seamlessly integrated into industrial-grade control frameworks such as behavior trees and finite state machines without internal modification, which greatly reduces the deployment difficulty and integration cost of embodied intelligence large models in real industrial scenarios and has good prospects for industrial application.

[0061] Preferably, during the construction of the high-dimensional temporal feature manifold sequence, the trajectory curvature of the feature manifold within the sliding temporal window is calculated simultaneously, and the trajectory curvature characterizes the degree of bending of the feature manifold in the latent space; Based on the real-time value of the trajectory curvature, the contribution weight of each historical step feature within the sliding time window to the output of the multi-state early stopping classifier is dynamically adjusted: the greater the curvature, the greater the contribution weight of the feature manifold of the recent moment, so as to enhance the immediate response to drastic state changes.

[0062] In the continuous control of a robotic arm driven by a VLA model, the trajectory of the intermediate characteristic manifold in the latent space is not uniform linear motion. The motion characteristics of the characteristic manifold differ significantly across different operational stages: Coarse positioning stage: The robotic arm moves away from the target, and the feature manifold drifts slowly along an approximately straight line in the latent space with a small trajectory curvature; Precision alignment phase: As the robotic arm approaches the target, the characteristic manifold undergoes drastic changes and fluctuations, and the trajectory curvature increases significantly; Contact assembly stage: The characteristic manifold oscillates slightly around the target point, and the curvature exhibits high-frequency jitter characteristics.

[0063] Traditional sliding time windows typically assign equal or fixed-decreasing contribution weights to features at each historical step, failing to distinguish the differences and information content of feature manifold changes at different stages. When the feature manifold undergoes a sharp shift during the fine alignment stage, features from distant historical steps have lost their reference value, yet they still participate in classification decisions with the same weights, diluting the influence of recent key features and reducing the sensitivity of the early stopping classifier to critical states.

[0064] To address the aforementioned issues, this embodiment proposes a dynamic weighting mechanism based on trajectory curvature: by calculating the degree of trajectory curvature of the feature manifold within the sliding window in real time, the contribution weight of each historical step feature within the window is dynamically adjusted according to the magnitude of the curvature.

[0065] In each control cycle, the feature vectors of all historical moments contained in the current sliding time window are first obtained. These feature vectors are arranged in chronological order to form a high-dimensional trajectory in the latent space.

[0066] For each intermediate moment within the window (excluding the beginning and end), take three adjacent feature points: that moment, the moment before it, and the moment after it. Calculate the angle between the directions of the preceding and following displacement vectors. Divide this angle by the sum of the lengths of the two displacement vectors. The result is the degree of local curvature of the feature trajectory at that moment. The larger this ratio, the sharper the turn and the greater the curvature of the feature manifold at that location; the smaller the ratio, the closer the feature manifold is to linear motion at that location.

[0067] For the beginning and end moments of the window boundary, since there are no complete adjacent points, a mirror filling method is used to complete them. That is, with the boundary point as the center of symmetry, the displacement of the adjacent points on the inside is copied in reverse to the outside, and the curvature is calculated in the same way as above; or the curvature value of the nearest inner point is directly taken as the approximate value of the boundary point.

[0068] After calculating the degree of curvature at each time point within the window, perform the following steps to assign weights: Step 1: Calculate the baseline curvature level. Take the arithmetic mean of the curvature at all times within the window and use it as the baseline curvature level for the current window. At the same time, record the maximum value of the curvature at each time within the window, which will be used to determine whether the curvature at the current time is significantly higher than the average level.

[0069] Step 2: Calculate the relative curvature factor at the current moment. Divide the current curvature by the average curvature level within the window (to prevent the denominator from being zero, the relative curvature factor is set to 1 by default when the average curvature level is extremely small or zero). The resulting ratio is the relative curvature factor at the current moment. If the relative curvature factor is greater than 1, it indicates that the curvature of the characteristic manifold at the current moment exceeds the average level within the window, meaning that it is currently in a phase of rapid state change; if the factor is less than or equal to 1, it indicates that the curvature at the current moment is at or below the average level.

[0070] Step 3: Assign contribution weights based on the relative curvature factor; dynamically adjust the contribution weights of features from each historical step within the window according to the magnitude of the relative curvature factor at the current moment. A large relative curvature factor indicates that the feature manifold is undergoing a dramatic shift. In this case, contribution weights are tilted towards more recent features within the window, giving them a larger weighting and compressing the weights of features from more distant time periods. The greater the curvature, the more pronounced this tendency to concentrate on recent features.

[0071] When the relative curvature factor is close to 1, it means that the curvature of the feature manifold is comparable to the average level of the window, thus maintaining a relatively balanced distribution of weights over time.

[0072] When the relative curvature factor is less than 1, it means that the current feature manifold movement is gentle. At this time, the reference value of the long-term historical features should be appropriately retained, and the weight distribution is relatively uniform.

[0073] The specific implementation method of the above weight allocation is as follows: using the relative curvature factor at the current moment as an adjustment parameter, assigning incremental baseline weights to each moment within the window in order from farthest to nearest, and then normalizing the sum of all weights to equal 1. The larger the relative curvature factor, the greater the weight difference between recent and distant moments; the smaller the relative curvature factor, the smaller the weight difference between moments.

[0074] As a simplified alternative to the continuous adjustment method described above, segmented coarse-grained adjustment can also be used: When the relative curvature factor exceeds the preset high threshold, it is determined that the current state is in a stage of drastic change. At this time, only the features of the most recent half of the window are retained, and the feature weights of the other half of the earlier time are reset to zero. Only the recent features are used for classification. When the relative curvature factor is between a preset low threshold and a high threshold, a weight allocation method that increases linearly from the long term to the near term is adopted. When the relative curvature factor is lower than the preset low threshold, it is determined that the current movement is in a smooth motion phase, the feature weights of all moments within the window are equal, and complete historical information is preserved.

[0075] The high and low thresholds mentioned above are preset empirical parameters that can be determined through offline simulation or on-site calibration according to specific work scenarios. For example, a feasible configuration is to set the high threshold to 2.0 and the low threshold to 0.8.

[0076] After weighting, the feature vectors at each time point within the window are multiplied by their corresponding contribution weights to obtain a weighted sequence of feature vectors. The original chronological order of each time point is maintained after weighting.

[0077] Subsequently, all the weighted feature vectors are concatenated end-to-end in chronological order to form a single high-dimensional vector. This vector replaces the original unweighted concatenated feature vectors and serves as the input to the multi-state early stopping classifier for subsequent discrete state code output.

[0078] The entire process described above is repeated in each control cycle, thereby achieving online, real-time dynamic updates of the sliding window weights.

[0079] The key parameters and their configuration methods involved in this embodiment are as follows: The sliding window length, i.e., the number of historical time steps contained within the window, needs to be set according to the inference frame rate of the VLA model. In one feasible implementation, the window length is set to cover a range of historical information from 0.5 seconds to 2 seconds. For example, when the VLA inference frame rate is 20 frames per second, the window length can be set to 10 to 40 time steps.

[0080] The curvature sensitivity coefficient controls the influence of the relative curvature factor on the weight distribution, and its value is typically set between 1 and 5. The specific value is determined through offline simulation testing or cross-validation on actual operational data, with the early stopping accuracy of the classifier on the validation set as the optimization objective.

[0081] High curvature threshold and low curvature threshold are used in the segmented simplification scheme. Their specific values ​​can be determined based on the statistical analysis of the degree of curvature of characteristic manifolds in historical operation data. For example, they can be set with the upper quartile and lower quartile of the historical curvature distribution as a reference.

[0082] The calibration process for the above parameters is completed before deployment, and after deployment, online fine-tuning can be performed based on the actual performance under different operating conditions.

[0083] The effects of the above technical solution are as follows: By automatically concentrating weights on recent features when the feature manifold undergoes a sharp shift, interference from outdated historical information can be effectively suppressed, enabling the classifier to capture feature changes at critical states in a timely manner. Without changing the window length or sampling rate, selective attention is achieved through dynamic weight redistribution, avoiding the additional computational overhead of blindly shortening the window or increasing the sampling frequency to improve response sensitivity. During the coarse localization stage, the feature manifold is relatively flat, retaining long-term historical information to perceive the overall trend. During the fine alignment stage, the feature manifold undergoes a sharp shift, focusing on recent features to capture critical states. The same mechanism automatically adapts to the differentiated needs of different operational stages without requiring manual switching of strategies or parameters. In high-frequency micro-oscillation scenarios, although the degree of feature manifold curvature fluctuates frequently, the amplitude is small, and the relative curvature factor is not significantly high, preventing excessive weight concentration on recent features. This avoids misjudgments triggered by over-focusing on oscillation details, effectively adding a filter against accidental triggering at the feature input level.

[0084] Preferably, after extracting the hidden state features of the penultimate layer of the VLA backbone network as an intermediate feature manifold, the method further includes: Based on the response distribution of the attention head to the input token in the VLA model, the hidden state features are decoupled into a semantic-dominant subspace and an action-dominant subspace. A time-dimensional low-pass filter is applied to the semantic-dominant subspace to suppress high-frequency semantic fluctuations while preserving the original high-frequency response of the action-dominant subspace. The filtered semantic subspace features and action subspace features are selectively fused along the feature dimension or input separately to different branches of the multi-state early stopping classifier to achieve decoupling of task semantics and physical state in early stopping determination.

[0085] The step of decoupling the hidden state features into a semantic-dominant subspace and an action-dominant subspace based on the response distribution of the attention head to the input token in the VLA model specifically includes: Obtain the attention weights of each attention head in the multi-head self-attention mechanism of the VLA model backbone network for various types of input tokens; The semantic-action response tendency ratio of each attention head is calculated based on the ratio of the average attention weight of each attention head on semantically relevant tokens to the average attention weight on action-related tokens. Attention heads whose response tendency ratio is higher than a first preset threshold are classified as semantically dominant attention heads, and attention heads whose response tendency ratio is lower than a second preset threshold are classified as action-dominant attention heads, wherein the first preset threshold is greater than the second preset threshold; The output features of all semantically dominant attention heads are concatenated to construct the semantically dominant subspace; the output features of all action-dominant attention heads are concatenated to construct the action-dominant subspace.

[0086] The application of a time-dimensional low-pass filter to the semantic-dominant subspace specifically involves: Maintain a historical cache of the semantic dominant subspace features, and perform a weighted average of the semantic features at the current time and the semantic features at the previous time, wherein the weight of the features at the current time is less than the weight of the features at the previous time. The selective fusion of filtered semantic subspace features and action subspace features specifically includes: The filtered semantic subspace features and action subspace features are respectively input into two independent branches of the multi-state early stopping classifier. The two branches maintain parameter independence during the forward propagation of the classifier and feature fusion or decision fusion is performed before the output layer of the classifier.

[0087] The application of a time-dimensional low-pass filter to the semantic-dominant subspace specifically involves: Maintain a historical cache of the semantic dominant subspace features, and perform a weighted average of the semantic features at the current time and the semantic features at the previous time, wherein the weight of the features at the current time is less than the weight of the features at the previous time. The selective fusion of filtered semantic subspace features and action subspace features specifically includes: The filtered semantic subspace features and action subspace features are respectively input into two independent branches of the multi-state early stopping classifier. The two branches maintain parameter independence during the forward propagation of the classifier and feature fusion or decision fusion is performed before the output layer of the classifier.

[0088] In the VLA (Vision-Language-Action) model, the hidden state features of the backbone network are a comprehensive encoding of multimodal inputs. They include information closely related to the semantic understanding of the task, such as the category attributes of the target object, spatial positional relationships, and high-level descriptions of the task intent, as well as information closely related to the execution of the underlying physical actions, such as the instantaneous pose of each joint of the robotic arm, the motion trend of the end effector, and the real-time changes in contact force.

[0089] These two types of information have fundamentally different characteristics in terms of change over time: semantic information is usually relatively stable and will not change drastically during the execution of a single subtask—the target object will not disappear out of thin air, and the task intent will not switch repeatedly within milliseconds; while motion information changes rapidly over time, especially in stages such as precision alignment and contact assembly, where physical state parameters such as joint angles, end-effector velocities, and torques may fluctuate significantly every millisecond.

[0090] However, in the standard VLA architecture, these two types of information are co-encoded in the same hidden state vector and used as input to subsequent modules. When this hidden state is directly used for early stopping decisions, the slowly changing semantic information acts as background noise, drowning out the key signals in the rapidly changing action information that are truly relevant to the stopping decision, thus reducing the classifier's sensitivity to critical states.

[0091] This embodiment utilizes the differences in the responses of different attention heads to input tokens in the VLA model to separate the hybrid hidden state features into a semantic-dominated subspace and an action-dominated subspace. Subsequently, differential processing is applied to the features of the two types of subspaces to achieve decoupling of task semantics and physical state in early stopping determination.

[0092] The VLA model is built on the Transformer architecture, and its backbone network includes a multi-head self-attention mechanism. During training, different attention heads spontaneously focus on different aspects of the input information. Some attention heads tend to focus on visual and linguistic tokens related to semantic attributes such as object category, color, and shape, while others tend to focus on tokens related to physical attributes such as spatial location, direction of motion, and contact state.

[0093] This embodiment utilizes this characteristic to achieve feature decoupling. The specific steps are as follows: The first step is to obtain the attention response distribution. During the forward inference process of the VLA model, as multimodal input data (including visual image tokens, language command tokens, robot body state tokens, etc.) passes through each Transformer module of the backbone network, the attention weight distribution of each attention head in the multi-head self-attention mechanism for each input token is recorded. This weight distribution reflects which input information the attention head pays more attention to at the current position.

[0094] The second step is to calculate the semantic-action response tendency score for each attention head. For each attention head, based on the distribution pattern of its attention weights across various input tokens, the semantic response tendency score and action response tendency score are calculated. Specifically, a set of semantically relevant token categories (such as region tokens belonging to known object categories in an image, noun and verb tokens in language commands) and a set of action-related token categories (such as robot joint status tokens, end-effector pose tokens, and force sensor tokens) are predefined. The average attention weight of each attention head on semantically relevant tokens is calculated, and the ratio of this ratio to the average attention weight on action-related tokens is the semantic-action response tendency ratio for that head. A ratio greater than 1 indicates that the attention head is more inclined to focus on semantic information; a ratio less than 1 indicates that the attention head is more inclined to focus on action information.

[0095] The third step is to select semantically dominant attention heads and action-dominant attention heads. All attention heads are sorted from highest to lowest according to their semantic-action response tendency ratio. A certain percentage (e.g., the top 30%) of these attention heads are selected as semantically dominant attention heads, and a certain percentage (e.g., the bottom 30%) are selected as action-dominant attention heads. Attention heads in the middle region, due to their unclear response tendency, can be discarded or participate in feature construction in both subspaces simultaneously. The values ​​of each percentage can be determined through offline experiments based on the specific model structure and task type. For example, in one implementation, for a VLA model with 7 billion parameters, the top 25% of attention heads are classified as semantically dominant attention heads, and the bottom 25% as action-dominant attention heads.

[0096] After completing the classification and filtering of attention heads, the subspace features are extracted and constructed: Construct a semantically dominant subspace. For each attention head labeled as semantically dominant, extract the attention-weighted feature vector output by that head at the current layer. Concatenate the output feature vectors of all semantically dominant attention heads along their feature dimensions to form the semantically dominant subspace feature vector. This subspace primarily carries high-level semantic information related to task understanding, object recognition, and intent parsing.

[0097] Construct a motion-dominant subspace. For each attention head labeled as motion-dominant, extract its output feature vector in the same way and concatenate them to form the motion-dominant subspace feature vector. This subspace mainly carries the underlying motion information related to the robot arm's physical state, motion trend, and force perception interaction.

[0098] In one implementation, the aforementioned subspace is constructed at the penultimate layer of the VLA backbone network, that is, the multi-head attention output of this layer is directly split and recombined. In another implementation, to avoid the randomness of single-layer partitioning, the classification results of the attention heads in the last three or more layers of the backbone network can be comprehensively voted on, and the consistency of the response tendency of each head in the multiple layers can be used as the final classification criterion.

[0099] Semantic information and action information play different roles in early stop decision-making, therefore differentiated preprocessing is required: A low-pass filter is applied to the semantic dominant subspace. Semantic information should not change drastically in a short period, but its representation in the hidden state may generate high-frequency noise due to minor fluctuations in visual input (such as slight camera shake or changes in lighting). To suppress this noise, a low-pass filter is applied to the feature vector of the semantic dominant subspace in the time dimension. Specifically, a historical cache queue of semantic subspace features is maintained, and a weighted average is taken between the semantic features at the current time and those at the previous time. The current time feature has a smaller weight (e.g., 0.3), and the previous time feature has a larger weight (e.g., 0.7), thereby suppressing high-frequency fluctuations in semantic features. In another implementation, a moving average filter can be used, where the average of the semantic features at the current time and several previous times is taken as the filtered current semantic feature. The filter strength (i.e., the proportion of historical information retained) can be preset according to the dynamic nature of the work scenario. For high-speed dynamic operations, the filter strength can be appropriately reduced to retain a certain semantic change response capability, while for static precision operations, the filter strength can be increased to suppress noise to the greatest extent.

[0100] Preserving the original high-frequency response of the motion-dominant subspace. In contrast to semantic information, the rapid changes in motion information are precisely the key basis for early stopping decisions. The sharp drop in speed when the robotic arm approaches the target, the sudden increase in torque at contact, and the frequency changes in the end-effector oscillations all exist in the motion-dominant subspace as high-frequency signals. Therefore, no time-dimensional smoothing or filtering is applied to the motion-dominant subspace, fully preserving its original high-frequency response characteristics to ensure that the early stopping classifier can capture key changes in the physical state in real time.

[0101] After completing the above differentiation process, we obtain the filtered and smoothed semantic subspace features and the unfiltered original action subspace features. These two feature sets are then input into the at most-state early stopping classifier using either of the following two methods: Method 1: Unified Input After Feature Concatenation and Fusion. The filtered semantic subspace feature vector and action subspace feature vector are concatenated end-to-end along the feature dimension to form a single feature vector, which serves as the single input to the multi-state early stopping classifier. This method is simple to implement, requires no modification to the classifier structure, and is suitable for deployment with minimal changes to existing systems.

[0102] Method 2: Inputting features into different branches of the classifier. Semantic subspace features and action subspace features are fed into two independent input branches of a multi-state early stopping classifier. The two branches maintain parameter independence during forward propagation, with feature fusion or decision fusion only occurring before the final output layer. This method allows the classifier to learn different weight mappings for the two types of features, more fully leveraging the advantages of decoupling, but requires corresponding adjustments to the classifier structure.

[0103] In a preferred embodiment, method two is adopted, and independent batch normalization layers and dropout layers are set in the two branches of the classifier for semantic features and action features respectively, in order to adapt to the differences in numerical distribution and statistical characteristics of the two types of features.

[0104] Considering that the VLA model's reliance on semantic and action information may change at different stages of the task—for example, semantic understanding ("which object I want to grab") is more critical at the initial stage of the task, while action monitoring ("have I already made contact with the target") is more critical in the later stages—this embodiment also provides an optional dynamic reclassification mechanism: In each control cycle, the rate of change of the magnitude of the feature vector in the action-dominant subspace is monitored synchronously. When this rate of change exceeds a preset threshold, the system is determined to be in an action-sensitive phase. At this point, the contribution weight of the action-dominant subspace features in the final classification decision is increased, while the weight of the semantic-dominant subspace is correspondingly decreased. Conversely, when the rate of change of the action feature magnitude is less than or equal to the preset threshold, the default weight ratio of the two types of features is maintained. This dynamic weight adjustment is implemented in the input layer or fusion layer of the classifier, enabling the decoupled feature fusion weights to adaptively adjust as the operation progresses.

[0105] The key parameters and their configuration methods involved in this embodiment are as follows: The division ratio between semantically dominant and action-dominant attention heads refers to selecting the top percentage of all attention heads as semantically dominant and the bottom percentage as action-dominant. This ratio can be determined based on the total number of attention heads in the specific VLA model and the characteristics of the task. A feasible configuration is to select 25% for each, with the middle 50% of attention heads discarded due to unclear response tendencies.

[0106] The intensity coefficient of the low-pass filter represents the weight ratio of the current semantic features to the historical semantic features in a weighted average. This coefficient determines the temporal smoothness of the semantic information, and its value typically ranges from 0.2 to 0.5 for the current feature weights. Specific values ​​are determined through offline simulation testing, with the accuracy of the early stopping classifier on the validation set as the optimization objective.

[0107] The threshold for determining the action-sensitive stage is used in the dynamic reclassification mechanism to determine whether the current stage is the action-sensitive stage. This threshold is determined based on the distribution statistics of the rate of change of action feature magnitude in historical operation data, for example, using the upper quartile of the historical rate of change distribution as a reference threshold.

[0108] The effects of the above technical solution are as follows: By separating slowly changing semantic information from rapidly changing action information and filtering and suppressing the semantic information, the dilution effect of semantic background noise on key action signals is eliminated. The early stopping classifier can thus focus on the physical state changes truly relevant to the stopping decision, significantly improving its sensitivity in discriminating critical states.

[0109] To reduce the interference of high-frequency noise in semantic information on early stopping decisions, non-task factors such as minor jitter and lighting changes in visual input may cause random fluctuations in semantic features. If these are directly input into the classifier without processing, they may lead to misjudgments. By applying a low-pass filter to the semantic subspace, these high-frequency noises that are irrelevant to early stopping decisions are effectively filtered out, improving the stability and robustness of the decision.

[0110] Preferably, the specific process of maintaining continuous motion decoding and servo output of the VLA model includes: In the action decoding stage of the VLA model, the prediction dimension is jointly expanded on the basis of the original action dimension, and the stopping confidence and subtask completion progress percentage of the current step are output simultaneously. The system monitors the completion percentage of the subtask in real time. When the percentage reaches a preset progress threshold (e.g., 90%), it immediately calls the Visual Language Model (VLM) asynchronously. It then merges the currently synchronously acquired multi-view observation images with the text description of the subtask list to be executed, and performs fault probability inference for the next operation. If the failure probability of the VLM inference output is lower than the preset risk threshold, then the current decoding thread is maintained and the action generation of the next subtask is smoothly transitioned. If the failure probability is higher than the preset risk threshold, a hard stop is not triggered directly. Instead, a ramp deceleration command (soft stop) is sent to the lower-level servo controller in advance to reduce the end speed gradient to a safe range before the decoding thread is completely suspended.

[0111] In its implementation, during the motion decoding phase of the VLA model, the prediction dimension is jointly expanded based on the original motion dimension (such as 7-DOF joint angles or 6-DOF end-effector pose). In each decoding step, in addition to outputting the current motion command, the VLA model simultaneously outputs the following two auxiliary prediction values: Stop confidence: A scalar value between 0 and 1 that represents the level of confidence the model has in whether the current subtask should stop at this moment; Subtask completion percentage: A scalar value between 0 and 100%, representing the model's estimate of the overall completion progress of the current subtask.

[0112] The two auxiliary prediction values ​​mentioned above are achieved by adding two independent lightweight prediction heads (each consisting of two MLP layers) to the end of the VLA model decoder. These prediction heads are output in parallel with the main action decoder and share the same backbone network features. The two auxiliary prediction heads are jointly fine-tuned with the VLA main model during the offline phase. The training data is labeled with the true stop label (whether it should stop) and the true progress percentage (manually labeled or obtained through simulation) at each time step.

[0113] The upper-level discretization control framework monitors the completion percentage of the subtask in real time. When the percentage reaches a preset progress threshold (preferably 90% in this embodiment), it immediately calls the VLM (Vision-Language Model) asynchronously without blocking the current VLA decoding thread.

[0114] The inputs to a VLM include: Multi-view observation images acquired synchronously at the current moment (real-time acquisition by multi-view cameras); A text description of the list of subtasks currently pending execution (obtained from the upper-level task planner).

[0115] After receiving the above multimodal input, the VLM performs the next step of fault probability inference and outputs a fault probability value between 0 and 1.

[0116] If the failure probability of the VLM inference output is lower than the preset risk threshold (preferably 0.7 in this embodiment), it is determined that the risk of executing the next subtask is controllable, the current VLA decoding thread is not interrupted, and the process smoothly transitions to the action generation of the next subtask.

[0117] The specific implementation of smooth transition is as follows: At the moment of action switching, the end velocity vector at the end of the previous subtask is used as the initial value of the starting velocity of the next subtask, and fifth-order polynomial interpolation is used to perform transition planning on the trajectory between the two actions to ensure that position, velocity, and acceleration are continuous at the switching point. After the transition trajectory is generated, it is decoded by the VLA model and continues to output joint position / velocity commands to the lower-level servo controller.

[0118] If the failure probability output by the VLM inference is higher than the preset risk threshold, the next subtask is determined to have a high risk of failure. In this case, a hard stop is not triggered directly (hard stop is triggered by state 1 and is a reactive measure), but a ramp deceleration command (i.e., soft stop) is sent to the lower-level servo controller in advance.

[0119] The soft stop is implemented as follows: the servo controller gradually reduces the end effector velocity to zero along the current direction of motion using a preset deceleration rate (e.g., 2 m / s²), and smoothly reduces the end effector velocity to a safe range before completely suspending the VLA decoding thread. After the soft stop is completed, the system pauses the current operation and reports a risk warning to the upper-level task planner, awaiting manual intervention or replanning.

[0120] The effects of the above technical solution are as follows: This solution achieves proactive risk awareness by asynchronously calling VLM to perform the next operation when the subtask completion progress reaches a threshold, thus significantly improving the overall success rate of long-term tasks.

[0121] The introduced soft-stop mechanism complements the main solution's hard-stop mechanism: soft-stop uses a ramp-like deceleration method to achieve preventative safety braking, avoiding sudden locking that could impact the robotic arm and the manipulated object; hard-stop maintains zero speed upon reaching the optimal pose, ensuring locking accuracy. The two mechanisms differ in their triggering conditions and execution methods, addressing potential risks and achieved goals respectively, together constructing a complete safety protection chain from risk warning to precise locking.

[0122] During subtask switching, a three-dimensional continuous transition of position, velocity, and acceleration of the motion trajectory is achieved through fifth-order polynomial interpolation, completely eliminating the mechanical shock and pose jumps caused by traditional stop-restart task switching. For delicate operation scenarios such as cloth handling, which have high requirements for continuity and stability, the smooth transition mechanism effectively avoids cloth pulling or wrinkling caused by sudden speed changes during task switching, ensuring the continuity and quality of operation.

[0123] The VLM call is designed to execute asynchronously. After the call is triggered, the main VLA decoding thread continues to run, while VLM inference is completed in parallel in the background. The inference result is only used to decide whether to trigger a soft stop, without blocking the high-frequency control output. This design ensures that even when calling a large model for risk prediction, the underlying 500Hz control frequency remains unaffected, thus balancing the dual requirements of intelligent decision-making and real-time control.

[0124] In one possible implementation, the criteria for determining the peak cutoff state are: the positional error between the end effector and the target object is less than or equal to the error threshold (e.g., 0.5 mm), the derivative sign of the positional error is reversed within the timing window, and the torque change rate in the real-time physical state parameters exceeds the torque change rate threshold (torque characteristic jump). The criteria for determining the abnormal escape state are as follows: within a continuous time window of a preset duration, the motion direction command symbol of the end effector alternates between positive and negative at least N times (entering a visual blind spot or failing to recognize fabric features), and the rate of change of the mean of the intermediate state feature manifold within the time window is less than a preset rate of change threshold (the rate of change of the mean of the implicit feature tends to zero).

[0125] Preferably, the position error threshold and the torque change rate threshold are adaptive thresholds that are dynamically adapted based on the current working conditions. The working conditions include at least the material type of the current working target, the real-time spatial distance between the end effector and the target object, and the load mass currently carried by the robotic arm. By acquiring in real time the material hardness coefficient of the current working target, the real-time spatial distance between the end of the robotic arm and the target object, and the current load mass of the robotic arm, and inputting the acquired parameters into a pre-fitted nonlinear mapping function to calculate and generate the position error threshold and the torque change rate threshold, a judgment boundary that is adaptive to the current working condition is formed. Furthermore, after the multi-state early stopping classifier outputs the second state code and suspends the VLA decoding thread due to satisfying the peak truncation state condition, if the following two conditions are simultaneously detected within a preset continuous time window: The current position error is greater than the dynamically updated position error threshold again; Furthermore, the rate of change of torque at the current moment decreases to below a preset percentage of the dynamically updated rate of change of torque threshold; Then the upper-level discretization control framework automatically releases the zero-speed holding state, sends a recovery enable signal to the VLA decoding thread, and reactivates the continuous action decoding output.

[0126] The number of alternations N of the motion direction command symbol is a dynamic parameter that is adaptively tuned based on the real-time motion state of the end effector of the robotic arm; wherein, the real-time motion state includes at least the real-time moving speed of the end effector, and the tuning rule of the number of alternations N is that it is positively monotonically correlated with the real-time moving speed, and the specific value of N is obtained online through a preset speed-number mapping curve or piecewise function.

[0127] The effects of the above technical solution are as follows: The adaptive threshold and automatic unlocking mechanism adapt to various working conditions, avoiding misjudgments and missed judgments caused by fixed thresholds under different materials, distances, and loads. For precision operations, the threshold is automatically tightened to ensure sensitivity, while for flexible operations, the threshold is automatically relaxed to avoid accidental stops.

[0128] After triggering a stop, the system continuously monitors the position error and torque change rate. When physical conditions recover, it automatically releases the zero-speed hold and restarts VLA decoding, allowing operation to resume from transient overshoot events without manual intervention. With thresholds adaptive to operating conditions, a hard stop is only triggered when the system truly enters the critical range, avoiding over-triggering under boundary conditions.

[0129] The adaptive N-value tuning rule resolves the adaptability contradiction of a fixed N-value in variable-speed operations. The N-value is positively correlated with the real-time movement speed; at low speeds, it automatically selects a smaller value to ensure sensitivity in anomaly detection, while at high speeds, it automatically selects a larger value to filter out normal random jitter and avoid false triggers. This significantly reduces the false alarm rate caused by normal high-frequency jitter in high-speed segments. Anomaly escape is only triggered when the number of directional alternations truly exceeds the normal adjustment range.

[0130] Together, they achieve an early stop control closed loop that ensures that the machine stops when it should, does not stop when it should not, and can automatically recover after an erroneous stop. The front end makes accurate judgments, and the back end is fault-tolerant and automatic, adapting to complex work scenarios with multiple working conditions and variable speeds.

[0131] Preferably, refer to the appendix Figure 5 A schematic diagram of the offline training and manifold extraction process for a multi-state early stopping classifier. The multi-state early stopping classifier is trained in the following manner: Acquire an offline multimodal operation trajectory dataset, which includes visual data at multiple time points, natural language command data, and physical state parameter data of the robotic arm; Each trajectory in the multimodal operation trajectory dataset is labeled with a state label at each time step. The state label includes a first label, a second label, and a third label. The penultimate hidden state features of the multimodal operation trajectory after encoding by the VLA model backbone network are extracted and used as the training feature manifold; A sliding time window is introduced to cascade the training feature manifolds of the current time step and its historical steps to construct high-dimensional time-series training samples; Using the high-dimensional time-series training samples as input and the state labels as supervision signals, the initial classifier is trained using a hybrid loss function to obtain the multi-state early stopping classifier.

[0132] The hybrid loss function is: ;in: as well as ; Where L is the mixture loss function, Represents the multi-class cross-entropy loss used to supervise classification accuracy; One-hot encoding for the real label; The probability of class i is output by Softmax; To constrain abrupt changes in adjacent predicted probabilities, the temporal smoothing regularization term is used to constrain the difference between the probability distributions output by the multi-state early stopping classifier at adjacent time points, so as to suppress frequent switching of the discrete state code at the action boundary. Adjustment coefficient The value is preferably set between 0.15 and 0.35. Let be the probability distribution at time t. This represents the probability distribution at the previous time step (t-1).

[0133] Specifically, the multi-state early stopping classifier is trained during the offline preparation phase and then deployed to the online control system after training. The specific offline training process is as follows: First, an offline multimodal operation trajectory dataset was acquired. This dataset was obtained by running a VLA model to drive a robotic arm to perform operations under various typical operation scenarios. Each scenario includes at least one complete operation process (starting from the initial state, sequentially experiencing the normal propulsion phase, the critical arrival phase, and the abnormal oscillation phase). During each complete operation, the system continuously recorded multimodal data at all times, including: RGB images captured by multi-view cameras, task-driven natural language commands, the angular positions and angular velocities of each joint of the robotic arm, joint torques, and the Cartesian space pose of the end effector. The data at each moment are arranged in chronological order to form a multimodal operation trajectory.

[0134] Secondly, each trajectory is automatically and objectively labeled at each time step, generating a status label for each time step. The labeling rules are as follows: The conditions for labeling an object as the first label are: the target object is far from the target area, or the positional error within the area shows a significant monotonically decreasing trend and the motion gradient remains stable.

[0135] The conditions for being labeled as the second label are: the position error of the target object is less than or equal to the set error threshold, and the sign of the derivative of the position error changes from negative to positive within the time window, accompanied by a rapid jump in the joint torque.

[0136] The condition for being labeled as the third label is: the end effector action exhibits high-frequency alternating positive and negative oscillations within a continuous time sequence, and the rate of change of the implicit feature mean approaches zero.

[0137] After annotation, the time-series data for each trajectory are sequentially input into the VLA model backbone network. The hidden state features from the penultimate layer output of the backbone network are extracted and used as the training feature manifold for that time step. Compared to the final output layer, this layer retains richer intermediate semantic information while significantly reducing dimensionality, making it suitable as input for a lightweight classifier.

[0138] Subsequently, a sliding temporal window is introduced. For each time step, the training feature manifolds of that time step and its historical time steps are concatenated sequentially to form a single high-dimensional vector, which serves as the high-dimensional temporal training sample for that time step. The window length can be set according to the inference frame rate of the VLA model. For example, when the VLA model's inference frame rate is 20 frames per second, the window length can be set to 10 to 40 time steps, so that the duration of historical information covered by the window corresponds to the aforementioned time range of 0.5 seconds to 2 seconds. For different frame rates, those skilled in the art can calculate and determine the corresponding window length value using the same duration range.

[0139] An initial classifier is constructed using a three-layer perceptron structure, with normalization layers inserted between each layer for normalization processing. Gaussian error linear units are used as activation functions, and normalized exponential functions are used as output layers.

[0140] Finally, using high-dimensional time-series training samples at each time step as input and corresponding state labels as supervision signals, the initial classifier is trained using a hybrid loss function. During training, mini-batch stochastic gradient descent is used for optimization, with a batch size of 256 and an initial learning rate of 1e-3. A cosine annealing learning rate decay strategy is used to gradually reduce the learning rate, with a weight decay coefficient of 1e-4. After processing each batch of training samples, the current hybrid loss function value is calculated and the network weights are updated. After each iteration through all training samples, the classification accuracy of the current classifier is calculated on the validation set. Training stops when the validation set accuracy no longer improves for N consecutive iterations (e.g., 5 consecutive iterations), or when the decrease in the hybrid loss function value is less than a preset change threshold for M consecutive iterations (e.g., 10 consecutive iterations, threshold 0.001), resulting in a trained multi-state early stopping classifier. The values ​​of the above hyperparameters can be adaptively adjusted according to the specific amount of training data and model size.

[0141] The hybrid loss function introduces a temporal smoothing regularization term on the basis of multi-class cross-entropy loss, which constrains the probability distribution of the classifier output at adjacent time steps to prevent drastic changes, thereby improving the temporal continuity of decision-making while ensuring classification accuracy.

[0142] The hybrid loss function consists of two terms: the first term is the multi-class cross-entropy loss, and the second term is the temporal smoothing regularization term. The two terms are weighted and summed by an adjustment coefficient.

[0143] The first term is the multi-class cross-entropy loss. This is a standard classification loss function used to supervise the classifier's correct classification of the input features at the current time step. Its basic calculation logic is as follows: For each training sample, the classifier outputs the predicted probabilities of the three classes through the Softmax function, and the sum of the three probabilities equals 1. The cross-entropy loss compares the probability distribution predicted by the model with the true one-hot encoded label. The closer the probability of the class corresponding to the true label is to 1, and the closer the probabilities of the other two classes are to zero, the smaller the loss value; conversely, the larger the loss value. By minimizing this loss, the classifier is trained to determine the appropriate state code to output as accurately as possible at each independent time step.

[0144] The second term is the temporal smoothing regularization term. Its basic calculation logic is as follows: For two adjacent control moments, i.e., the current moment t and the previous moment t-1, obtain the probability distribution vectors output by the classifier at these two moments, and calculate the degree of difference between the two probability distribution vectors. The greater the degree of difference, the more drastic the jump in the classifier's decision at adjacent moments; the smaller the degree of difference, the smoother and more continuous the classifier's decision is in time. This regularization term is added to the total loss function in the form of a penalty term. When the probability distribution difference between adjacent moments is large, the total loss increases accordingly, thereby forcing the model to learn the decision boundary that evolves smoothly over time during training.

[0145] The effects of the above technical solution are as follows: Traditional cross-entropy loss only focuses on the correctness of classification at a single time step, without constraining the consistency of decisions between adjacent time steps. This scheme introduces a temporal smoothing regularization term, allowing the classifier's decision boundary to evolve smoothly over time. The output probability distribution will not undergo drastic changes due to small noises or feature fluctuations, making the decision-making process more stable and reliable. Dramatic changes in the probability distribution directly lead to high-frequency switching between continued operation and emergency stop in the upper-level control framework. This scheme eliminates such non-physical decision jumps through smoothing constraints, ensuring that control commands are continuous and consistent in time, avoiding the cumulative impact damage of high-frequency oscillations on the robotic arm's transmission mechanism and deceleration device. During the transition of the robotic arm from normal movement to a critical stop state, the feature manifold is precisely near the class decision boundary, which is the moment most prone to probability jumps. This scheme, through temporal smoothing constraints, ensures that the classifier maintains gradual changes in output rather than abrupt jumps during such transition phases, reserving sufficient response time for the upper-level controller.

[0146] Preferably, the secondary fine-tuning strategy includes: At the input end, a local image region of a preset size is extracted from the original image, centered on the current position of the robotic arm end, and then the local image region is upsampled and re-input into the VLA model; At the output end, the motion commands output by the VLA model are multiplied by a preset scaling factor for gain compression.

[0147] In the aforementioned secondary fine-tuning strategy, at the input end, the cropping center of the local image region is not directly mapped to the end position of the robotic arm. Instead, the attention heatmap generated from the original image is weighted and calculated through the intermediate layer of the VLA model to obtain the visual saliency center offset. The end position is then superimposed with the offset to serve as the dynamic cropping center. Simultaneously, at the output end, the scaling factor is not a fixed value but is adjusted in reverse according to the real-time collected contact force amplitude of the robotic arm end. The larger the contact force amplitude, the smaller the scaling factor, forming force-guided variable gain compliant control.

[0148] In its implementation, during the forward inference process of the VLA model, after the original image is input to the model's visual encoder and processed by multiple Transformer modules, the multi-head self-attention mechanism generates an attention weight distribution on the image feature map. This weight distribution reflects the degree of attention the model pays to different spatial regions in the image at the current moment; regions with higher weights indicate that the model believes that the region contributes more to the current decision.

[0149] In this embodiment, during VLA model inference, the average attention weights of the last one or more self-attention heads in the visual encoder part on the image feature map are recorded simultaneously. After upsampling to restore the original image resolution, a two-dimensional attention heatmap with the same size as the input image is formed. The value of each pixel position in this heatmap represents the visual saliency score of that position in the current decision.

[0150] After obtaining the attention heatmap, it is first normalized so that the sum of all weight values ​​in the heatmap equals 1, forming a probabilistic saliency distribution. Then, the weighted centroid of this saliency distribution is calculated. This is achieved by averaging the horizontal and vertical coordinates of all pixels across the entire image plane, using the saliency score of each pixel as a weight, to obtain the saliency center point. This center point represents the center of the image region that the VLA model considers visually most worthy of attention at the current moment.

[0151] The pixel coordinates of the saliency center are compared with the projected pixel coordinates of the robotic arm's end effector position on the image, and the offset vector between the two is calculated. This offset vector is the visual saliency center offset. This offset includes two components: direction and distance, reflecting the visual difference between where the model thinks it should look and where the end effector is actually located.

[0152] The dynamic clipping center is obtained by vector superimposing the original projected coordinates of the robotic arm's end effector position with the calculated offset of the visual saliency center. This center is not simply equal to the end effector position or the saliency center, but rather a compromise between the two. When the model's attention to a certain visual region is highly concentrated (i.e., the heatmap distribution is sharp and the variance is small), the weight of the offset increases, and the clipping center moves closer to the saliency center. When the model's attention distribution is relatively dispersed (i.e., the heatmap distribution is flat and the variance is large), the weight of the offset decreases, and the clipping center is closer to the original end effector position.

[0153] Using the dynamic cropping center as the geometric center, a local image region of a preset size (e.g., one-third or one-quarter of the original image size) is extracted from the original image. This preset size can be pre-set according to the task type; precision assembly tasks require a smaller cropping range to obtain high-resolution details, while coarse positioning tasks can use a larger cropping range to preserve contextual information.

[0154] The extracted local image region is upsampled using methods such as bilinear interpolation or bicubic interpolation to restore it to the same image size as the original input of the VLA model. The upsampled local image retains high-frequency detail information (such as edges, textures, and contours) of key regions in the original image, while eliminating a large number of irrelevant background regions.

[0155] The upsampled local image replaces the original full-size image and is re-inputted into the visual encoder of the VLA model for a new round of forward inference and action generation. Since the spatial resolution of the input image remains unchanged but the effective information density is significantly improved, the VLA model can focus more on the visual region most relevant to the current operation during the secondary fine-tuning stage, generating more precise action commands.

[0156] A six-dimensional force / torque sensor is installed on the end effector of the robotic arm to collect contact force information between the end effector and the environment or target object in real time. During the secondary fine-tuning stage, the force sensor data is read at the same control frequency as VLA inference (e.g., once every fifty milliseconds) to calculate the contact force amplitude at the current moment, that is, the composite magnitude of the three-dimensional force vector (force components in three orthogonal directions).

[0157] At the output, the motion command generated by the VLA model (usually the expected displacement increment or velocity command of the end effector) is multiplied by a scaling factor for gain compression to reduce the amplitude of a single step motion and avoid collisions or loss of control due to excessive motion during sensitive phases. The value of this scaling factor is not fixed, but is adjusted inversely based on the real-time contact force amplitude.

[0158] The specific mapping rule is as follows: the contact force amplitude and the proportional coefficient have an inverse monotonic relationship; the larger the contact force amplitude, the smaller the proportional coefficient. When the contact force amplitude is low (the robotic arm has not yet contacted the target or has only made slight contact), the proportional coefficient remains at a large value, allowing for a large single-step motion increment to achieve rapid approximation; as the contact force amplitude gradually increases (the robotic arm gradually presses against the target), the proportional coefficient decreases smoothly, and the motion increment is gradually compressed to a tiny step size, achieving fine-tuning and compliant contact.

[0159] In one implementation, the above-mentioned reverse mapping is achieved through a preset piecewise linear function: when the contact force amplitude is lower than the first force threshold, the proportional coefficient takes the maximum value (e.g., 0.8); when the contact force amplitude is higher than the second force threshold, the proportional coefficient takes the minimum value (e.g., 0.1); when the contact force amplitude is between the first force threshold and the second force threshold, the proportional coefficient decreases linearly with the contact force amplitude.

[0160] In another implementation, the aforementioned reverse mapping is achieved through a preset nonlinear function (such as an exponential decay function or an S-curve), so that the proportional coefficient decreases smoothly and continuously as the amplitude of the contact force increases, avoiding abrupt changes that the piecewise function may produce at the threshold.

[0161] The motion commands output from the original VLA model are multiplied by the scaling factor calculated through the mapping process described above to obtain compressed motion commands, which are then sent to the lower-level servo controller for execution. These compressed motion commands are more conservative in amplitude but maintain the same direction as the original commands, ensuring that the robotic arm approaches the target safely and gradually during the secondary fine-tuning phase.

[0162] The input-side dynamic visual cropping and the output-side variable gain control do not operate in isolation; they form a collaborative closed loop during the secondary fine-tuning stage. On the one hand, dynamic visual clipping enables the VLA model to focus on the most relevant visual region in each round of inference, generating more accurate raw action commands. On the other hand, force-guided variable gain control performs adaptive compression of the raw command with a safe amplitude based on the contact state. When visual clipping allows the model to accurately target the object, the direction of the raw action command is more accurate; when force feedback indicates that the contact has become closer, gain compression makes the execution more refined. The two work together to give the secondary fine-tuning stage both accurate targeting and stable movement.

[0163] In a preferred embodiment, the adjustments to the input and output ends are performed synchronously in each secondary fine-tuning iteration. In each iteration, the dynamic cropping center is first calculated based on the current attention heatmap, and an upsampled local image is extracted. The local image is then input into the VLA model to obtain the original action command. Next, the scaling factor is calculated based on the current contact force amplitude, and the action command is compressed. Finally, the compressed action command is executed, and the next iteration begins. This process is repeated until the early-stop classifier re-outputs the first status code (release status) or the preset maximum number of iterations is reached.

[0164] The key parameters and their configuration methods involved in this embodiment are as follows: The cropped image size refers to the ratio of the cropped portion of the image to the original image. For precision assembly tasks, a smaller ratio (e.g., one-quarter) is recommended, while for coarse positioning tasks, a larger ratio (e.g., 1 / 3 to 1 / 2) is suitable. This parameter can be preset before system deployment based on the task type, or automatically switched during operation based on the current subtask type.

[0165] Upsampling methods can include bilinear interpolation, bicubic interpolation, or deep learning-based super-resolution methods. On embedded platforms with limited computing power, bilinear interpolation is the preferred option due to its low computational cost and good performance.

[0166] The first and second force thresholds are used to define the range of variation of the proportional coefficient in piecewise linear mapping. These thresholds are determined based on the force sensor range of the end effector and the safe contact force range for the specific task. For example, in precision assembly tasks, the first force threshold can be set to 10% of the rated maximum contact force, and the second force threshold can be set to 60% of the rated maximum contact force.

[0167] The maximum and minimum values ​​of the proportionality coefficient correspond to the compression ratio of the motion under no-contact and maximum safe contact force conditions, respectively. The maximum value is recommended not to exceed 0.9 to retain a certain safety margin, and the minimum value is recommended not to be lower than 0.05 to ensure effective motion can still be generated at extremely slow speeds.

[0168] The effects of the above technical solution are as follows: This solution guides the cropping center offset through the VLA model's own attention heatmap, ensuring the cropped area always focuses on the visual region deemed most valuable for decision-making by the model, effectively improving the information density and task relevance of the input image. By adjusting the proportional coefficient inversely using the contact force amplitude, the robotic arm can approach the target at a speed adapted to the contact state during the secondary fine-tuning phase, ensuring both approach efficiency and contact safety. Dynamic visual cropping guarantees that the VLA model acquires the most relevant visual information in each round of inference to generate accurate original action commands. Force-guided variable gain control safely adapts the command based on the physical contact state. Both are updated synchronously and cooperate with each other in each iteration of the secondary fine-tuning phase, jointly improving the success rate and safety of the fine-tuning stage.

[0169] This approach directly utilizes the attention heatmap generated during the inference process of the VLA model itself as a pruning guide signal, without introducing any additional models or parameters, thus achieving dynamic focusing of visual attention while maintaining the system's lightweight nature.

[0170] The subtask backtracking and action resampling mechanism in the secondary fine-tuning strategy specifically includes: Backtracking phase: When the secondary fine-tuning strategy is activated, the most recent K-step incremental action sequence stored in the memory pool is extracted. Starting from the current pose of the end effector, the incremental actions are canceled in reverse step by step until the state of the robotic arm is restored to the earliest target keyframe pose that can meet the prerequisites of the current subtask. Resampling stage: Under the pose of the target keyframe, M candidate action block hypothetical trajectories are generated in parallel using a random policy network. A minimum Bayesian risk (MBR) decoder is introduced, and the posterior expected loss of each trajectory is calculated in combination with the current observation features. The trajectory with the lowest expected loss is selected as the main recovery trajectory. Safety constraints: During the execution of the backtracking phase and the resampling phase, a preset virtual repulsive potential field is accessed simultaneously. If interference is detected between the backtracking path or candidate trajectory and a static obstacle in the work space, the current backtracking is forcibly terminated and a replanning request is reported to the upper layer.

[0171] In the specific implementation, during the normal continuous control process of the VLA model, an action memory pool is pre-established. This memory pool stores the most recently executed K-step incremental action sequence in a fixed-length circular queue. Each incremental action is recorded as a six-dimensional vector, which includes the translational increments (displacements along the x, y, and z axes) of the end effector in three orthogonal directions and the attitude increments (rotation angles around the x, y, and z axes) in three rotational directions.

[0172] The value of K is set according to the specific task. In one implementation, K is set to 50 to 200 steps, corresponding to approximately 2.5 to 10 seconds of historical action information (calculated in a control cycle of 50 milliseconds). This duration covers the typical number of action steps required for most subtasks from start to finish, providing sufficient backtracking range while avoiding the storage of outdated action data that has lost its reference value.

[0173] In addition to the incremental action itself, the memory pool also synchronously records the end effector pose (position coordinates and attitude angle) before each step of execution, as well as the flag indicating whether the robotic arm meets the prerequisite conditions for the current subtask after each step of execution.

[0174] When the secondary fine-tuning strategy is activated, the precondition flags recorded in the memory pool are queried first. Starting from the current moment, the system traces back frame by frame to find the earliest key frame that can meet the preconditions of the current subtask, and sets the key frame as the target pose to be restored.

[0175] If the robotic arm can meet the preconditions of a subtask in a certain pose, it means that, theoretically, the subtask can be re-executed according to the normal VLA control flow from that pose. The farther the pose is from the current pose (i.e., the earlier the backtracking), the larger the error correction space is obtained, but too far backtracking distance will also lead to efficiency loss. Therefore, the criterion of meeting the preconditions as early as possible strikes a balance between recovery probability and recovery efficiency.

[0176] In one specific implementation, if a key frame that meets the preconditions cannot be found by tracing back from the current frame to the earliest record in the memory pool, it is determined that the current memory pool record is insufficient. Instead, a more conservative emergency strategy is adopted, in which the robotic arm is slowly raised to a preset safe retraction pose, and a request to re-plan the task is reported to the upper-level controller.

[0177] After determining the target keyframe pose, starting from the current pose of the end effector, all incremental motion records from the target keyframe to the current moment are retrieved from the memory pool. Undo operations are then performed frame-by-frame in strict reverse chronological order, meaning the last incremental motion executed is undone first, and the earliest executed incremental motion is undone last. Each undo operation applies the previous incremental motion in reverse to the current pose with a negative gain, causing the end effector to backtrack point-by-point along the original path.

[0178] During the undo process, reverse action commands are issued at the same control frequency as normal control to ensure the smoothness and controllability of the backtracking motion. After each undo is completed, it is verified whether the current pose matches the target keyframe pose; once the match is confirmed, the backtracking phase is complete.

[0179] After tracing back to the target keyframe pose, the resampling stage begins. This stage utilizes a pre-trained stochastic policy network, taking the multimodal observations corresponding to the target keyframe pose (including the current visual image, the robotic arm joint state, contact force information, etc.) as input, to generate M candidate motion block hypothetical trajectories in parallel.

[0180] Each candidate trajectory is an action block sequence containing continuous action instructions for the next L steps from the current moment (each step is also a six-dimensional action vector). In one implementation, M is set to 16 to 64, with the specific value determined based on available computing resources and real-time requirements. When computing resources are sufficient, a larger value can be used to cover more trajectory hypothesis space, while a smaller value can be used when real-time requirements are high to ensure that the total time of the resampling stage does not exceed the preset time budget.

[0181] The stochastic policy network introduces a suitable amount of random noise during the generation process, resulting in a differentiated distribution of the M candidate trajectories in the action space. Some trajectories are conservative (smaller action amplitude), some are aggressive (larger action amplitude), and the remaining trajectories fall somewhere in between. This diversified generation strategy ensures that the candidate trajectory set can cover different styles of recovery strategies, providing a rich set of alternatives for subsequent selection.

[0182] After generating M candidate trajectories, a minimal Bayesian risk decoder is introduced to evaluate and rank all candidate trajectories. This decoder combines the multimodal observation features at the current time to calculate the expected risk of each candidate trajectory, that is, the posterior expected loss that may lead to adverse consequences after executing the trajectory.

[0183] Specifically, the evaluation process is as follows: For each candidate trajectory, the decoder first predicts the pose and contact state that the end effector will reach after executing the trajectory; then, based on the current observation features, it evaluates whether the predicted state meets the task success conditions (such as whether the position error is small enough, whether the contact force is within the safe range, whether the visual features match the target template, etc.); the degree of failure to meet the success conditions is quantified into a loss value, which comprehensively considers multiple factors such as pose deviation, the degree of contact force exceeding the limit, and the difference in matching with the target image features; the smaller the loss value, the lower the risk of the trajectory.

[0184] In one implementation, the calculation of the loss value also takes into account the motion smoothness factor. When the rate of change (e.g., acceleration) between adjacent motion commands exceeds the motion change rate threshold (acceleration threshold), an additional smoothness penalty term is added to suppress the generation of unstable trajectories with excessive jitter or excessive acceleration, ensuring that the selected trajectory is physically executable and friendly to the robotic arm body.

[0185] After completing the loss evaluation of all M candidate trajectories, the decoder selects the trajectory with the minimum expected loss as the master recovery trajectory.

[0186] Static obstacles in the workspace (including but not limited to workbench edges, fixtures, other equipment, safety fences, etc.) have their spatial locations and geometric contours obtained through 3D modeling or online mapping before deployment. For each obstacle, a virtual repulsive potential field is defined around it. The strength of this potential field increases exponentially or exponentially as the distance from the spatial point to the obstacle surface decreases. When the distance is greater than the preset safe operating distance, the potential field strength decays to approximately zero.

[0187] During the backtracking and resampling process, the virtual repulsive potential field is synchronously connected to the control loop to monitor the desired position of the end effector in real time for each cycle.

[0188] Before executing each reverse action in the backtracking phase, a line segment path is first constructed from the starting point to the ending point, using the current pose as the starting point and the reverse action of that shot as the displacement vector. It is then checked whether any point on this line segment path enters the repulsive potential field range of any obstacle. If so, it is determined that the backtracking path interferes with the obstacle.

[0189] During the trajectory evaluation process in the resampling phase, in addition to calculating the posterior expected loss of each candidate trajectory, interference detection is also performed on the complete action sequence of each candidate trajectory, checking frame by frame whether each intermediate pose in the trajectory has entered the range of action of the virtual repulsive potential field.

[0190] If interference is detected during the backtracking phase, the current backtracking operation is immediately terminated, causing the robotic arm to remain in its current pose and maintain a zero-speed state. At the same time, a request is sent to the upper-level controller that the backtracking path is blocked and needs to be replanned.

[0191] If an interference is found in a candidate trajectory during the trajectory evaluation in the resampling phase, that trajectory is directly marked as infeasible and removed from the candidate set, and will not participate in the subsequent least Bayesian risk ranking. If all M candidate trajectories are determined to interfere with obstacles, the resampling fails, a replanning request is reported to the upper layer, and safety maintenance measures are taken (such as maintaining the current position and decelerating to zero).

[0192] The complete execution flow of the entire subtask backtracking and action resampling mechanism is as follows: The first step is triggered by the third status code, which activates the secondary fine-tuning strategy and initiates the backtracking phase.

[0193] The second step is to query the precondition flags in the memory pool to determine the target keyframe pose.

[0194] The third step involves reversing the incremental motion frame by frame from the current pose and moving towards the target keyframe pose. Before each step, interference is detected using a virtual repulsive potential field; if interference is detected, the process is terminated and reported.

[0195] The fourth step, after reaching the target keyframe pose, involves resampling. A random policy network generates M candidate trajectories in parallel.

[0196] The fifth step is to perform interference detection on each candidate trajectory and eliminate all trajectories that interfere with obstacles.

[0197] The sixth step is to use the minimum Bayesian risk decoder to evaluate the posterior expected loss of the remaining candidate trajectories and select the trajectory with the minimum loss as the main recovery trajectory.

[0198] The seventh step is to execute the first action in the main recovery trajectory, and then re-enter the normal continuous control mode of the VLA model, that is, starting from the target keyframe pose, continue to execute subsequent actions according to the normal decoding and servo output of the VLA model.

[0199] If the early stop classifier outputs a second status code (peak truncation) or a third status code (abnormal escape) again at any stage of the above process, the corresponding status code processing logic will be used to respond (suspend hard stop or reactivate secondary fine-tuning).

[0200] The effects of the above technical solution are as follows: By retracting the robotic arm to a keyframe pose that clearly satisfies the preconditions of the subtask, the accumulation of state deviations that lead to anomalies is fundamentally eliminated, providing a reliable foundation for subsequent recovery execution. The multiple candidate trajectories generated by the random policy network exhibit differentiated distributions in motion style, covering different recovery paths from conservative to aggressive. The minimum Bayesian risk decoder selects the scheme with the lowest overall risk from multiple candidate trajectories, avoiding a dead end after a single strategy failure and ensuring that the selection process has a quantifiable optimization objective rather than blind experimentation. When the robotic arm moves backward along the retracing path, it may encounter obstacles, and the candidate trajectories may also contain infeasible schemes that interfere with obstacles. The full-process access of the virtual repulsive potential field allows these potential collision risks to be detected and avoided before the action is executed, preventing physical collisions and ensuring the safety of the secondary fine-tuning process.

[0201] Example 2: This application provides a VLA multi-state early stop control system based on multimodal sensing, used to execute the method described in Example 1. The system includes: The multimodal data acquisition interface is used to connect to multi-view cameras and the physical state sensors of the robotic arm to acquire multimodal sensing data streams in the operating space environment in real time. Memory, which stores computer programs; A processor that, when executing the computer program, implements the steps of the method described above.

[0202] The processor is configured with a multi-state early stopping classifier node, a VLA inference master node, and a secondary fine-tuning recovery strategy node. The multi-state early stopping classifier node, VLA inference master node, and secondary fine-tuning recovery strategy node are each encapsulated as independent asynchronous ROS 2 nodes. Each node interacts with data through the ROS 2 topic communication mechanism.

[0203] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0204] This application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented using any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0205] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0206] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element qualified by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A VLA multi-state early stopping control method based on multimodal sensing, characterized in that, Includes the following steps: Real-time acquisition of multimodal sensing data streams within the operating space environment, wherein the multimodal sensing data streams include real-time physical state parameters of the robotic arm; The multimodal sensing data stream is input into the pre-feature extraction network of the VLA model for representation alignment and modality fusion to generate a dimension-reduced intermediate feature manifold. The intermediate state feature manifold is input into a multi-state early stopping classifier for dynamic evaluation. The classifier outputs a discrete state code that represents the current fine-tuning state, wherein the discrete state code includes a first state code, a second state code, and a third state code. The upper-level discretized control framework monitors the discrete status codes in real time and executes corresponding asynchronous routing control based on the discrete status codes: When the first status code is received, the continuous action decoding and servo output of the VLA model are maintained; When the second status code is received, the VLA decoding thread is suspended and a hard stop command is sent to the lower-level servo controller to achieve zero speed maintenance; When the third status code is received, the current continuous control flow is cut off and the secondary fine-tuning strategy is activated.

2. The control method according to claim 1, characterized in that, The intermediate state feature manifold is obtained as follows: The features output after the representation alignment and modality fusion are input into the VLA model backbone network for encoding, and the hidden state features of the penultimate layer of the backbone network are extracted as intermediate feature manifolds. A sliding time window is introduced to cascade the intermediate state feature manifolds of the current time t and its historical steps to construct a high-dimensional time-series feature manifold sequence, which serves as the input to the subsequent classifier.

3. The control method according to claim 1, characterized in that, The multi-state early stopping classifier is constructed using a three-layer perceptron; the three-layer perceptron integrates a normalization layer, uses Gaussian error linear units as activation functions, and uses a normalized exponential function as the output layer.

4. The method according to claim 1, characterized in that, The upper-level discretized control framework includes a behavior tree engine.

5. The control method according to claim 1, characterized in that, The first status code indicates a release status, the second status code indicates a peak cutoff status, and the third status code indicates an abnormal escape status.

6. The control method according to claim 5, characterized in that, The criteria for determining the peak cutoff state are: the position error between the end effector and the target object is less than or equal to the error threshold, the derivative sign of the position error is reversed within the timing window, and the torque change rate in the real-time physical state parameters exceeds the torque change rate threshold. The criteria for determining the abnormal escape state are as follows: within a continuous time window of a preset duration, the motion direction command symbol of the end effector alternates between positive and negative at least N times, and the rate of change of the mean of the intermediate state feature manifold within the time window is less than a preset rate of change threshold.

7. The method according to claim 1, characterized in that, The multi-state early stopping classifier is optimized using a hybrid loss function, which is: ,in: as well as ； Where L is the mixture loss function, Represents the multi-class cross-entropy loss used to supervise classification accuracy; One-hot encoding for the real label; The probability of the i-th class output by the classifier; To constrain the temporal smoothing regularization term for abrupt changes in adjacent prediction probabilities; Adjustment coefficient ; Let be the probability distribution at the current moment. This represents the probability distribution at the previous moment.

8. The method according to claim 1, characterized in that, The secondary fine-tuning strategy includes: At the input end, a local image region of a preset size is extracted from the original image, centered on the projection pixel coordinates of the current robotic arm end position in the image. The local image region is then upsampled and re-input into the VLA model. At the output end, the motion commands output by the VLA model are multiplied by a preset scaling factor for gain compression.

9. A VLA multi-state early stop control system based on multimodal sensing, characterized in that, The system for performing the method as described in any one of claims 1 to 8 comprises: The multimodal data acquisition interface is used to connect to multi-view cameras and the physical state sensors of the robotic arm to acquire multimodal sensing data streams in the operating space environment in real time. Memory, which stores computer programs; A processor that, when executing the computer program, implements the steps of the method as claimed in any one of claims 1 to 8.

10. The system according to claim 9, characterized in that, The processor is configured with a multi-state early stopping classifier node, a VLA inference master node, and a secondary fine-tuning recovery strategy node. The multi-state early stopping classifier node, VLA inference master node, and secondary fine-tuning recovery strategy node are each encapsulated as independent asynchronous ROS 2 nodes.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1-8.

Images