Dexterous hand motion planning and control method based on visual language motion model

Abstract

The invention relates to the technical field of intelligent control, and discloses a dexterous hand motion planning and control method based on a visual language motion model, and the method comprises the steps: S1, collecting data, and carrying out the time sequence synchronization and consistency verification of multi-source data; s2, performing dynamic adaptive exponential moving average filtering and calculation processing on the joint sequence, and reconstructing a processing result into adaptive structured features; s3, fusing joint states, inputting the fused joint states into an action head network, splicing online and offline samples into a training batch according to a preset proportion, and determining results to jointly optimize model parameters; s4, performing model reasoning to obtain target action information, performing processing in sequence to generate a smooth low-jitter control sequence, and sending the smooth low-jitter control sequence to the dexterous hand for execution; and S5, the deployment thread issues a control instruction by aligning the annular buffer and the timestamp, triggers a rollback track and allows manual intervention when the control instruction exceeds a threshold value, and writes manual intervention data into an offline buffer. According to the method, efficient, stable and smooth motion control of the dexterous hand can be realized under the condition of extremely few teaching data sets.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligent control technology, specifically a method for dexterous hand motion planning and control based on a visual language action model. Background Technology

[0002] Traditional dexterous hand manipulation methods face multiple bottlenecks in terms of data and performance: imitation learning-based strategies heavily rely on massive, stable, and finely labeled high-quality finger movement trajectory data, resulting in extremely high collection and cleaning costs; reinforcement learning suffers from low sampling efficiency and is susceptible to instability or even policy degradation when performing fine-grained finger interactions in real physical environments, and is affected by sparse rewards and dynamic uncertainties; single-vision or language modal models struggle to simultaneously grasp the 3D structural details of the target object and the task intent, failing to generate action sequences that meet the requirements for fine control of joint angles and force application timing; and in the real deployment stage, jitter in action output, latency, and error accumulation often occur, weakening the success rate and stability of grasping and manipulation.

[0003] The aforementioned problems collectively restrict the improvement of dexterity's generalization ability, real-time performance, and reliability in complex multimodal human-machine collaborative scenarios, necessitating a unified method and system that integrates vision, language understanding, and fine motion generation while balancing training efficiency and robust deployment. To overcome the limitations of existing technologies, this patent proposes a dexterity motion planning and control method based on a visual-language-action model. Summary of the Invention

[0004] The purpose of this invention is to provide a method for dexterous hand motion planning and control based on a visual language action model, so as to solve the problems in the prior art.

[0005] The objective of this invention can be achieved through the following technical solutions: A method for dexterous hand motion planning and control based on a visual language action model, the method comprising: S1. Collect images from multiple cameras, voice task commands, finger joint positions and timestamps, and perform time-series synchronization and consistency verification on the multi-source data; S2. Perform dynamic adaptive exponential moving average filtering on the joint sequence and calculate the difference between adjacent frames, image cropping and color normalization. Reconstruct the processing results into structured features that fit the input of the adaptation model. S3. The visual encoder extracts image features, the language encoder parses instructions, and the joint states are fused and then input into the action head network. The training adopts a double buffering mechanism, which splices online and offline samples into training batches according to a preset ratio. The model parameters are jointly optimized by imitation loss and the judgment results based on the reward classifier. S4. The model infers the target action information and performs temporal fusion, velocity limiting, dead zone suppression and multi-step Bezier interpolation in sequence to generate a smooth and low-jitter control sequence, which is then sent to the dexterous hand for execution. The reward classifier outputs the success probability based on vision and joint state, which serves as a weighted signal for training and runtime. S5: The deployment thread issues control commands through a circular buffer aligned with the timestamp, monitors reward and jitter metrics in real time, triggers a rollback trajectory when thresholds are exceeded and allows manual intervention, and writes the manually intervened data to an offline buffer for subsequent training.

[0006] Furthermore, in S1, the real-time video data is generated by multiple camera images, which consists of multiple frames from each camera.

[0007] Furthermore, the formula for the adaptive exponential moving average filtering algorithm in S2 is:

[0008] Wherein, the smoothing coefficient dynamically adjusted at time t is the basic smoothing coefficient, is the joint jitter signal at time t, is the instantaneous angular velocity of the joint at time t, is the adjustment weight, is the filtered joint position output value at time t, is the original joint position input value collected at time t, and is the filtered output value at time t-1.

[0009] Furthermore, the visual encoder in S3 includes a first visual encoder and a second visual encoder; The first visual encoder is a self-supervised learning-based visual Transformer model, which is responsible for extracting global semantic features of the input image and outputting the first feature sequence. Among them, the second visual encoder is the backbone network of the single-stage object detection network, which is responsible for extracting local spatial features of the input image and outputting the second feature map; The first feature sequence is processed by dimensional projection through the first scale alignment module to output the first aligned feature; the second feature map is processed by adaptive average pooling and sequence rearrangement through the second scale alignment module to output the second aligned feature; so that the two have the same spatial size and feature dimension.

[0010] Furthermore, S3 uses a multi-scale adaptive cross-attention mechanism, taking the first alignment feature as the query vector Q and the second alignment feature as the key vector K and value vector V, to calculate attention weights and generate attention-enhancing features. The formula for calculating attention weights is as follows:

[0011] Where L is the number of multi-scale layers, the learnable weights for the i-th scale, and B is the relative position bias code matrix. is the bias scaling factor, is the learnable scaling matrix at the i-th scale, is the value projection matrix at the i-th scale, M is the dynamic mask matrix, τ(t) is the adjustment function, and Gi is the gating matrix of the i-th matrix.

[0012] Furthermore, after calculating the attention weights, S3 generates attention-enhanced features, wherein the formula involved is:

[0013] in, For learnable residuals; The first alignment feature and attention enhancement feature are then input into the gating network to calculate the fusion weights. Through multimodal feature concatenation and nonlinear transformation, the gating weights used for feature fusion are adaptively calculated. The hidden layer features in the middle are:

[0014] Among them, Linear compresses the splicing features to 512 dimensions; The gating function formula for modal feature differentiation and residual connection is introduced as follows:

[0015] Where Sigmoid is the activation function, and Linearres(Fenhanced) is the attention feature residual branch. Compress the feature map into a low dimension; The generated gating weights are used to dynamically weight and fuse the attention-enhanced features and global features to obtain the final fused features. The formula is as follows:

[0016] Where ⊙ represents element-wise multiplication; Subsequently, based on the fusion weights, the first alignment feature and the attention enhancement feature are weighted and fused to obtain the fused visual feature.

[0017] Furthermore, the adaptive Bezier interpolation optimization in S4 specifically includes: At the initial time point, (k=0,1,...,N−1, where N is the number of interpolation steps) in, Corresponding trajectory starting point , Corresponding trajectory endpoint Only intermediate nodes participate in iterative optimization; For the intermediate nodes of the i-th iteration Based on whether the instantaneous angular velocity satisfies Adjustments are made to obtain the time node of the (i+1)th iteration:

[0018] in The corresponding instantaneous angular velocity , The convergence threshold is calculated using the following formula:

[0019] in, , , , For Bessel control nodes; To avoid timing errors, use Monotonicity constraint formula limitations; Finally, it satisfies the convergence formula The iteration process is terminated. Substitute the non-uniform time node sequence into the Bessel interpolation formula to generate a compliant trajectory; The compliance trajectory is sent to the dexterous hand execution terminal through the library interface.

[0020] Furthermore, in S5, the deployment thread uses a fixed-capacity circular buffer as the time-sequential storage for action A and timestamp T, and the write pointer increments cyclically modulo the capacity; during instruction query and issuance, the query time is used as the reference. Choose the benchmark The smallest element is used to output the corresponding action. If the minimum difference exceeds the timestamp tolerance δ, it is set to empty to avoid timing out-of-bounds errors. The control loop retrieves instructions and their timestamps (A, T) from the queue and reads the current time. And perform time delay constraint verification, when The instruction is discarded if it is not in use; otherwise, it is sent to the execution end through the predetermined interface, thereby suppressing the interference of expired or mismatched instructions on closed-loop control. In abnormal threshold triggering, success probability Below the threshold During this process, manual intervention is permitted, and the pre-intervention status, remote operation actions, post-intervention status, timestamps, and triggering reasons are recorded throughout. These are synchronously written to online and offline buffers according to the intervention identifier and periodically solidified into a dataset for subsequent training and calibration of the closed loop.

[0021] The beneficial effects of this invention are: 1. The present invention provides a method for dexterous hand motion planning and control based on visual language action models. It effectively improves the practicality and engineering deployability of high-degree-of-freedom dexterous hand control systems under the condition of a small amount of teaching data. By storing offline teaching data and online human-machine collaboration data in offline buffer and online buffer respectively and mixing and sampling them according to a preset ratio for training, usable strategies can be quickly obtained with a very small amount of teaching, significantly reducing data acquisition costs. 2. The present invention is a dexterous hand motion planning and control method based on visual language action model. The proposed four-layer motion smoothing pipeline, which includes timing consistency fusion, speed limiting, dead zone suppression and multi-step Bezier interpolation, effectively suppresses high-frequency jitter and reduces joint micro-amplitude and energy peak, thereby improving grasping stability and reducing mechanical and electrical impact on the actuator. 3. The present invention provides a dexterous hand motion planning and control method based on a visual language action model. The adaptive interpolation step count and the speed constraint based on the task amplitude maintain control continuity while ensuring real-time response and avoiding lag caused by excessive smoothing. 4. The present invention provides a dexterous hand motion planning and control method based on a visual language action model, which integrates image enhancement, temporal perturbation and motion noise into the training process to improve the robustness of the model to changes in illumination, frame loss and slight external perturbations. 5. The present invention provides a dexterous hand motion planning and control method based on a visual language action model. It uses a visual reward classifier to automatically determine the task completion result, reducing manual annotation and accelerating training iteration; it monitors low reward and jitter thresholds to trigger automatic trajectory regression and request manual intervention, enhancing operational safety and fault recovery capabilities; and finally outputs multi-step joint control commands to the dexterous hand hardware for execution. 6. The dexterous hand motion planning and control method based on visual language action model of the present invention can achieve efficient, stable and smooth motion control of dexterous hand under the condition of very little teaching dataset. Attached Figure Description

[0022] The invention will now be further described with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the initial grasping posture of the RY-H1 dexterous hand of the present invention; Figure 3 This is a schematic diagram of the RY-H1 dexterous hand grasping fruit according to the present invention; Figure 4 This is a schematic diagram of the RY-H1 dexterous hand grasping a water bottle according to the present invention; Figure 5 This is a schematic diagram of the RY-H1 dexterous hand grasping a tennis ball according to the present invention; Figure 6This is a schematic diagram of the RY-H1 dexterous hand grasping a matchbox according to the present invention; Figure 7 This is a schematic diagram of the RY-H1 dexterous hand grasping an egg according to the present invention; Figure 8 This is a schematic diagram of the RY-H1 dexterous hand grasping express parcels according to the present invention; Figure 9 This is a schematic diagram of the RY-H1 dexterous hand grasping a pill according to the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To address the issues of high-frequency jitter and slow policy convergence in real-world environments where dexterous hands are prone to problems when teaching data is scarce, this invention constructs a multimodal VLA model that includes a visual reward classifier, a reinforcement learning training mechanism that combines online and offline buffers, and multi-layered policy post-processing modules such as temporal fusion, velocity limiting, dead zone suppression, and Bezier interpolation. This effectively enhances the dexterous hand's ability to achieve high success rates, low jitter, and real-time deployment for different engineering tasks with limited samples.

[0026] like Figures 1-9 As shown, a method for dexterous hand motion planning and control based on a visual language action model is presented, comprising steps S1 to S5: S1. Collect images from multiple cameras, voice task commands, finger joint positions and timestamps, and perform time-series synchronization and consistency verification on the multi-source data; The images from the multiple cameras are real-time video data, consisting of multiple frames from each camera. The tokenizer of the Visual Language Model (VLM) converts language task instructions into token sequences, and tags image placeholders, global images, and image-specific sequences. The finger joint positions are acquired and stored by reading 16 motor encoder count values ​​via the SocketCAN bus. The conversion formulas for the angles and curvatures of the thumb and finger joints are as follows: θrad = θdeg × n / 180° The timestamp synchronization mechanism generates timestamps for all clock sources using a unified clock source, and then stores the timestamped data in a circular buffer and reads the nearest neighbor for alignment.

[0027] S2. Perform dynamic adaptive exponential moving average filtering on the joint sequence and calculate the difference between adjacent frames, image cropping and color normalization. Reconstruct the processing results into structured features that fit the input of the adaptation model. The formula for the dynamic adaptive exponential moving average filtering algorithm is as follows:

[0028] in, The smoothing coefficient dynamically adjusted at time t. Based on the smoothing coefficient, Let t be the joint vibration signal at time t. The instantaneous angular velocity of the joint at time t. To adjust the weights, The filtered joint position output value at time t. Input values ​​for the original joint positions acquired at time t. This represents the filtered output value at time t-1; The structured feature set contains five elements: a four-dimensional tensor of camera images with dimensions (B, 3, H, W); a state vector of the 21 joint positions of the dexterous hand with dimensions (B, 21); a vector of the target joint movements with dimensions (B, 21); a three-dimensional tensor of the encoded language instructions with dimensions (B, L, D); and a timestamp vector of time synchronization with dimensions (B).

[0029] S3: The visual encoder extracts image features, the language encoder parses instructions, and the joint states are fused and then input into the motion head network. The training adopts a double buffering mechanism, which splices online and offline samples into training batches according to a preset ratio, and optimizes the model parameters by jointly using the imitation loss and the judgment results based on the reward classifier. The visual encoder is a hybrid visual encoder, consisting of two encoders: a first visual encoder and a second visual encoder. The first visual encoder is a visual Transformer model based on self-supervised learning, which is responsible for extracting global semantic features of the input image and outputting the first feature sequence with dimensions (B, N1, D1), where B is the batch size, N1 is the number of patches (256), and D1 is the feature dimension (768). The second visual encoder is the backbone network of the single-stage object detection network. It is responsible for extracting local spatial features of the input image and outputting a second feature map with dimensions (B, C, H, W), where C is the number of channels (256), and H and W are the height and width of the feature map. The first feature sequence is processed by dimensional projection of the first scale alignment module to output the first aligned feature; The second feature map is processed by adaptive average pooling and sequence rearrangement through the second scale alignment module to output the second alignment feature. The first alignment feature and the second alignment feature have the same spatial size and feature dimension (B, 256, 512), where B is the batch size, 256 is the number of tokens in a 16x16 grid, and 512 is the feature dimension. Using a multi-scale adaptive cross-attention mechanism, the first alignment feature is used as the query vector Q, and the second alignment feature is used as the key vector K and value vector V. Attention weights are calculated and attention-enhanced features are generated. The corresponding vector calculation formulas are as follows:

[0030] Where Wq, Wk, and Wv are learnable projection matrix coefficients, Fglobal is the first alignment feature, and Flocal is the second alignment feature; The formula for calculating attention weights is as follows:

[0031] Where L is the number of multi-scale layers, the learnable weights for the i-th scale, and B is the relative position bias code matrix. This is the bias scaling factor. Let be the learnable scaling matrix for the i-th scale. The projection matrix of the value at the i-th scale, M is the dynamic mask matrix, τ(t) is the adjustment function, and Gi is the gate matrix of the i-th matrix; After calculating the attention weights, attention-enhanced features are generated, involving the following formulas:

[0032] in, For learnable residuals; The cross-attention mechanism adopts a multi-head attention-based implementation, with 8 attention heads, and each attention head is uniformly configured with 64 feature dimensions; The first alignment feature and attention enhancement feature are then input into the gating network to calculate the fusion weights. Through multimodal feature concatenation and nonlinear transformation, the gating weights used for feature fusion are adaptively calculated. The hidden layer features in the middle are:

[0033] Among them, Linear compresses the splicing features (B, 256, 1152) to 512 dimensions; The gating function formula for introducing modal feature differentiation and residual connection is as follows:

[0034] Where Sigmoid is the activation function, and Linearres(Fenhanced) is the attention feature residual branch. Compressing 512 dimensions to 1 dimension; The generated gating weights are used to dynamically weight and fuse the attention-enhanced features and global features to obtain the final fused features. The formulas involved are as follows:

[0035] Where ⊙ represents element-wise multiplication (to achieve dynamic weighting of image features by gating weights); When the gate value is close to 1, it indicates that the scene is complex and requires precise localization, and the fusion features adopt more local spatial features. When the gate is close to 0, it indicates that the scene is simple or the global understanding is sufficient, and the fused features retain more global semantic features. When the gate value is between 0.3 and 0.7, it indicates that a balance between global and local information is needed. Subsequently, the first alignment feature and the attention enhancement feature are weighted and fused according to the fusion weight to obtain the fused visual feature; The feature fusion layer consists of a feedforward neural network, residual connections, and a normalization layer; The language encoder is built on a pre-trained text model. It first segments the input Task natural language instructions into words, and then converts the text tags into low-dimensional dense language embeddings through the input embedding layer. The encoder network then parses the semantic information of the instructions to generate a semantically consistent sequence of language features. The joint state mapping unit receives real-time joint data from the dexterous hand, and the linear projection layer maps the data to a hidden dimension that matches visual and linguistic features, forming standardized joint state features. The dual-buffering training mechanism employs an online cyclic first-in-first-out buffer and an offline dataset buffer, with batch sampling mixed according to a preset ratio; The online buffer is generated by the intervention of the teleoperated glove when the dexterous hand is driven by model inference, while the offline dataset buffer consists of the pre-collected dexterous hand grasping dataset.

[0036] S4. The model inference produces target action information, which is then sequentially fused, velocity limited, dead zone suppressed, and multi-step Bezier interpolation to generate a smooth, low-jitter control sequence that is sent to the dexterous hand for execution. The reward classifier outputs the success probability based on vision and joint state, which serves as a weighted signal for training and runtime. During the model inference phase, the hybrid visual encoder extracts features from multiple camera images, the language encoder parses the semantics of the task instructions, and, combined with the joint states via the action head network, plans the target joint angle sequence. This transforms discrete inference outputs into smooth control commands that conform to the dynamic constraints of dexterous hands; In temporal fusion planning, by introducing temporal consistency constraints, the current planning perspective is... The angle executed in the previous control cycle The fusion planning formula based on index-weighted fusion is:

[0037] Where λ is the timing coefficient; This mechanism is equivalent to a first-order low-pass filter, which effectively suppresses inter-frame jumps in the model planning output instructions and ensures the continuity of the motion trajectory. In speed limiting planning, the fused planning angle Calculate the angle change joint by joint ,like Then cut to the maximum permissible angular velocity. The formula for limiting the amplitude is:

[0038] Ensure that the planned trajectory meets the speed capability limits of the dexterous hand joints, and prevent control instability and vibration caused by excessive acceleration and mechanical shock; at the same time If the value is less than the dead zone threshold, it is determined to be a minor disturbance and the current control command is skipped to avoid frequent triggering of motor movement and reduce control jitter. In Bezier trajectory interpolation planning, traditional uniform time segmentation can lead to sudden velocity changes in the middle of the interpolation, creating a logical gap with velocity limiting planning. Therefore, an innovative iterative optimization method based on velocity constraints is introduced to specifically address the above problems. At the initial time point, , (k=0,1,...,N−1, N is the number of interpolation steps), where Corresponding trajectory starting point , Corresponding trajectory endpoint Only intermediate nodes participate in iterative optimization; For the intermediate nodes of the i-th iteration Based on whether the instantaneous angular velocity satisfies Adjustments are made to obtain the time node of the (i+1)th iteration:

[0039] in The corresponding instantaneous angular velocity , The convergence threshold is calculated using the following formula:

[0040] in, , , , For Bessel control nodes; To avoid timing errors, use Monotonicity constraint formula limitations; Finally, it satisfies the convergence formula The iteration process is terminated. Non-uniform time node sequence Substitute the values ​​into the Bessel interpolation formula to generate a compliant trajectory; The compliant trajectory is sent to the dexterous hand execution terminal through the library interface, including control parameters such as position control commands, motor speed, motor current, left and right hand indicators and control mode (0: force-position hybrid control mode, 1: speed control mode); The parallel-running reward classifier extracts features from multiple camera images based on a hybrid visual encoder, combines these features with the current joint state input to a binary classifier, and outputs the success probability. ; in, To reward the original output of the binary classification head output layer in the classifier, For activation functions; During the training phase, the success probability is used to adjust the reward signal. This guides the learning of high-success-rate control sequences for motion planning strategies. in, Fixed reward items, Dynamic reward coefficient, Final reward signal; During the execution of strategy reasoning, when The system triggers a manual intervention alarm, which serves as a weighted signal for motion planning quality monitoring and anomaly detection.

[0041] S5. The deployment thread issues control commands through a circular buffer aligned with the timestamp; it monitors reward and jitter metrics in real time, triggers a rollback trajectory and allows manual intervention when thresholds are exceeded, and writes the manually intervened data to an offline buffer for subsequent training.

[0042] The deployment thread uses a fixed-capacity circular buffer as the time-series storage for action A and timestamp T, with the write pointer incrementing cyclically modulo the capacity; during instruction query and issuance, the query time is used as the reference. Choose the benchmark The smallest element is used to output the corresponding action. If the minimum difference exceeds the timestamp tolerance δ, it is set to empty to avoid timing out-of-bounds errors. The control loop retrieves instructions and their timestamps (A, T) from the queue and reads the current time. And perform time delay constraint verification, when The instruction is discarded if it is not in use; otherwise, it is sent to the execution end through the predetermined interface, thereby suppressing the interference of expired or mismatched instructions on closed-loop control. In abnormal threshold triggering, success probability Below the threshold At the same time, manual intervention is allowed and the pre-intervention status, remote operation actions, post-intervention status, timestamps and trigger reasons are recorded throughout the process. The data is synchronously written into online and offline buffers according to the intervention identifier and periodically solidified into a dataset for subsequent training and calibration of the closed loop. Through the collaborative design of time alignment, latency constraints, intervention, and data accumulation, the risk of issuing expired instructions and joint jitter can be reduced to a certain extent, the stability of deployment can be improved, and a data closed loop of continuous iterative optimization can be formed.

[0043] The hardware device used in this invention is: Ruiyan Intelligent Control RY-H1 Dexterous Hand (16 degrees of freedom, 21 joints).

[0044] Similarly, the present invention is also compatible with other dexterous hand devices with 16 degrees of freedom or less.

[0045] The technical architecture of this invention has high scalability. Through modular adaptation and hierarchical control, the existing dexterous hand control model can be extended to a composite robot system that combines a dexterous hand and a robotic arm, thereby achieving fully autonomous control of dexterous operation tasks. This extended solution is not only compatible with existing technical architectures, but also adaptable to different types of robotic arms, providing a universal composite robot control solution for fields such as industrial automation and service robots, further enhancing the robot's task coverage and level of autonomy.

[0046] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for dexterous hand motion planning and control based on a visual language action model, characterized in that, The method includes: S1. Collect images from multiple cameras, voice task commands, finger joint positions and timestamps, and perform time-series synchronization and consistency verification on the multi-source data; S2. Perform dynamic adaptive exponential moving average filtering on the joint sequence and calculate the difference between adjacent frames, image cropping and color normalization. Reconstruct the processing results into structured features that fit the input of the adaptation model. S3. The visual encoder extracts image features, the language encoder parses instructions, and the joint states are fused and then input into the action head network. The training adopts a double buffering mechanism, which splices online and offline samples into training batches according to a preset ratio. The model parameters are jointly optimized by imitation loss and the judgment results based on the reward classifier. S4. The model infers the target action information and performs temporal fusion, velocity limiting, dead zone suppression and multi-step Bezier interpolation in sequence to generate a smooth and low-jitter control sequence, which is then sent to the dexterous hand for execution. The reward classifier outputs the success probability based on vision and joint state, which serves as a weighted signal for training and runtime. S5: The deployment thread issues control commands through a circular buffer aligned with the timestamp, monitors reward and jitter metrics in real time, triggers a rollback trajectory when thresholds are exceeded and allows manual intervention, and writes the manually intervened data to an offline buffer for subsequent training.

2. The method for dexterous hand motion planning and control based on visual language action model according to claim 1, characterized in that, In S1, real-time video data is generated from images from multiple cameras, consisting of multiple frames from each camera.

3. The method for dexterous hand motion planning and control based on visual language action model according to claim 1, characterized in that, The formula for the adaptive exponential moving average filtering algorithm in S2 is as follows: in, No. t The smoothing coefficient is dynamically adjusted in real time. Based on the smoothing coefficient, For the first t Constant joint tremor signals No. t Instantaneous angular velocity of the joint at any given moment. To adjust the weights, for t The filtered joint position output value at each moment for t The raw joint position input values ​​collected at all times, for t The filtered output value at time -1.

4. The method for dexterous hand motion planning and control based on visual language action model according to claim 1, characterized in that, The visual encoder in S3 includes a first visual encoder and a second visual encoder; The first visual encoder is a self-supervised learning-based visual Transformer model, which is responsible for extracting global semantic features of the input image and outputting the first feature sequence. Among them, the second visual encoder is the backbone network of the single-stage object detection network, which is responsible for extracting local spatial features of the input image and outputting the second feature map; The first feature sequence is processed by dimensional projection through the first scale alignment module to output the first aligned feature; the second feature map is processed by adaptive average pooling and sequence rearrangement through the second scale alignment module to output the second aligned feature; so that the two have the same spatial size and feature dimension.

5. The method for dexterous hand motion planning and control based on visual language action model according to claim 4, characterized in that, S3 then uses a multi-scale adaptive cross-attention mechanism, taking the first alignment feature as the query vector Q and the second alignment feature as the key vector K and value vector V, to calculate attention weights and generate attention-enhanced features. The formula for calculating attention weights is as follows: in, L For multi-scale layers, the first i Learnable weights at each scale B Relative position bias matrix, This is the bias scaling factor. For the first i A learnable scaling matrix at each scale, No. i The projection matrix of values ​​at each scale, M For dynamic mask matrix, τ (t) Regulation function, G i No. i A gating matrix of matrices.

6. The method for dexterous hand motion planning and control based on visual language action model according to claim 5, characterized in that, After calculating the attention weights, S3 generates attention-enhanced features, involving the following formula: in, The residual term is then learned; subsequently, the first alignment feature and attention enhancement feature are input into the gating network to calculate the fusion weights; through multimodal feature concatenation and nonlinear transformation, the gating weights used for feature fusion are adaptively calculated, and the intermediate hidden layer features are: Among them, Linear compresses the splicing features to 512 dimensions; The gating function formula for modal feature differentiation and residual connection is introduced as follows: in, Sigmoid For activation function, Linear res ( F enhanced ) represents the attention feature residual branch. Compress the feature map into a low dimension; The generated gating weights are used to dynamically weight and fuse the attention-enhanced features and global features to obtain the final fused features. The formula is as follows: Where ⊙ represents element-wise multiplication; Subsequently, based on the fusion weights, the first alignment feature and the attention enhancement feature are weighted and fused to obtain the fused visual feature.

7. The method for dexterous hand motion planning and control based on visual language action model according to claim 1, characterized in that, The adaptive Bezier interpolation optimization in S4 specifically involves: at the initial time node... (k=0,1,...) N -1, N (where the number of interpolation steps is...) Corresponding trajectory starting point , Corresponding trajectory endpoint Only intermediate nodes participate in iterative optimization; for the th i The intermediate node of the next iteration Based on whether the instantaneous angular velocity satisfies Adjustments were made to obtain the first... i +1 iteration time node in The corresponding instantaneous angular velocity , The convergence threshold is calculated using the following formula: in, , , , For Bessel control nodes; To avoid timing errors, use Monotonicity constraint formula restrictions; ultimately satisfies convergence formula Terminate the iteration process; convert the non-uniform time node sequence Substitute the values ​​into the Bessel interpolation formula to generate a compliant trajectory; The compliance trajectory is sent to the dexterous hand execution terminal through the library interface.

8. The method for dexterous hand motion planning and control based on visual language action model according to claim 1, characterized in that, In S5, the deployment thread uses a fixed-capacity circular buffer as the time-sequential storage for action A and timestamp T, and the write pointer increments cyclically modulo the capacity; during instruction query and issuance, the query time is used as the reference. Choose the benchmark Find the smallest element and output the corresponding action accordingly. If the smallest difference exceeds the timestamp tolerance... δ If the value is empty, it should be set to null to avoid timing out-of-bounds errors; the control loop retrieves instructions and their timestamps (A, T) from the queue and reads the current time. And perform time delay constraint verification, when The instruction is discarded if it is not in use; otherwise, it is sent to the execution end through the predetermined interface, thereby suppressing the interference of expired or mismatched instructions on closed-loop control. In abnormal threshold triggering, success probability Below the threshold During this process, manual intervention is permitted, and the pre-intervention status, remote operation actions, post-intervention status, timestamps, and triggering reasons are recorded throughout. These are synchronously written to online and offline buffers according to the intervention identifier and periodically solidified into a dataset for subsequent training and calibration of the closed loop.

Images