Action correction method and system for cooperative operation of multi-fingered dexterous hand and mechanical arm

Abstract

The invention discloses a motion correction method and system for cooperative operation of a multi-fingered dexterous hand and a mechanical arm, and the method comprises the steps: collecting motion data of the cooperative operation of the multi-fingered dexterous hand and the mechanical arm, and building a training data set covering various types of tasks; training by adopting a robot visual imitation learning framework to obtain a basic strategy; a lightweight residual error strategy is introduced on the basis of a basic strategy, a control signal is adjusted through online reinforcement learning, and fine-grained correction is carried out on the contact stage while the global smoothness is kept; through an interactive attention residual error weight network, a residual error strategy can adaptively pay attention to errors of different stages, adaptive fusion of track segment-observation features is realized in a time sequence dimension, and dimension-by-dimension residual error weights completely aligned with a joint angle are output in an action dimension. And action correction of cooperative operation of the multi-fingered dexterous hand and the mechanical arm is completed. According to the method, differential correction of strategies in different operation stages can be achieved, and coordination correction is conducted on the mechanical arm and the multi-fingered dexterous hand on the fine granularity level.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot control technology, specifically relating to a method and system for motion correction in the collaborative operation of a multi-finger dexterous hand and a robotic arm. Background Technology

[0002] Dexterous manipulation in robots is widely used in manufacturing, medical, service, and precision assembly fields. Its core objective is to achieve stable, precise, and efficient maneuvering capabilities similar to humans through the coordinated movements of multi-fingered dexterous hands and robotic arms. However, existing robot control methods generally rely on a single policy learning framework, making it difficult to simultaneously ensure stability and flexibility for long-term tasks. Due to the high-dimensional motion space and complex contact dynamics, a single policy often struggles to cover diverse operational scenarios and is prone to performance degradation when faced with differences in object shape, positional changes, and environmental disturbances.

[0003] In existing research, residual strategies have been increasingly introduced to mitigate the accumulation of errors in the base policy during inference. Some methods achieve rapid policy adjustment by directly superimposing lightweight corrections at the output; however, these methods neglect the varying requirements of residual magnitudes at different time points in the task, potentially leading to under-correction or over-compensation in long-term tasks. Other methods introduce uniformly scaled residual factors onto baseline actions to improve operational accuracy, but their performance is limited in high-degree-of-freedom dexterity tasks due to the lack of differentiated processing for different degrees of freedom. Furthermore, some methods adjust residuals for finger joints and distal forces, which can improve operational performance under specific conditions, but often rely on high-quality demonstration data and strict action alignment, resulting in insufficient generalization ability in complex and varied scenarios.

[0004] In recent years, some technologies have attempted to improve the precision of dexterous manipulation by combining residual correction and online reinforcement learning. However, the core idea of ​​these existing technologies is to dynamically optimize the basic policy using residual compensation to enhance the robot's adaptability in complex environments. Some solutions have shown that real-time compensation during execution can indeed improve policy stability. However, due to the lack of joint modeling of explicit temporal dependencies and spatial features in the task, these methods still struggle to guarantee sufficient accuracy and robustness in complex, long-term, multi-stage manipulation tasks, thus failing to meet the dual requirements of generalization and stability in practical applications. Summary of the Invention

[0005] The purpose of this invention is to address the problems in the prior art by providing a motion correction method and system for the collaborative operation of a multi-finger dexterous hand and a robotic arm. By introducing trajectory segmentation and temporal information modeling, the residual weights are dynamically adjusted in a stage-sensitive manner, thereby achieving differentiated correction of the strategy at different operation stages. It can output adaptive weights consistent with the motion space dimension, and coordinate and correct the motion of the robotic arm and the multi-finger dexterous hand at a fine-grained level.

[0006] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a motion correction method for collaborative operation between a multi-finger dexterous hand and a robotic arm is provided, including: Collect motion data of multi-finger dexterity hand and robotic arm cooperating to establish a training dataset covering multiple types of tasks; The basic strategy is obtained by training a robot vision imitation learning framework based on the established training dataset; A lightweight residual strategy is introduced on the basis of the basic strategy. The control signal is adjusted through online reinforcement learning, so as to achieve fine-grained correction of the contact stage while maintaining global smoothness. By using an interactive attention residual weight network, the residual strategy can adaptively focus on errors at different stages. This not only achieves adaptive fusion of trajectory segment and observation features in the temporal dimension, but also outputs dimension-wise residual weights that are perfectly aligned with joint angles in the action dimension, thus completing the action correction for the collaborative operation of the multi-finger dexterous hand and the robotic arm.

[0007] As a preferred approach, in each task, the basic strategy generates the overall motion trajectory, while the residual strategy dynamically adjusts the residual weights of different dimensions based on the motion trajectory category and multi-step observation characteristics, thereby coordinating and correcting the movements of the multi-finger dexterous hand and the robotic arm.

[0008] As a preferred embodiment, the robot visual imitation learning framework adopts a 3D Diffusion Policy framework that combines 3D visual representation with a diffusion strategy.

[0009] As a preferred approach, in the step of training the basic policy based on the established training dataset using a robot vision imitation learning framework, a consistency model is introduced during the training phase to distill the original multi-step sampling 3DDiffusion Policy framework into a single-step prediction. Let the clean action sequence be In terms of noise scale n Noisy samples were obtained The calculation expression is as follows:

[0010] In the formula, , This is the cumulative retention factor; Approximately noise-free samples are obtained from the teacher model using a K-step ordinary differential equation ODE solver. :

[0011] Received and Features are consistent across low-dimensional manifolds. To minimize cross-scale noise prediction discrepancies, a consistency network is used. Noise prediction model based on teacher strategies Learn the following loss functions:

[0012] This ensures The predictions are consistent across different noise levels, and the action distribution is constrained to a physically feasible low-dimensional manifold; the inference phase involves only single-step computation, directly based on observations. To obtain high-quality actions, the calculation expression is as follows:

[0013] In the formula, This is the observation state encoder.

[0014] As a preferred approach, in the step of training the basic policy using a robot vision imitation learning framework based on the established training dataset, the multilayer perceptron (MLP) encoder of the 3D Diffusion Policy framework is replaced with a pyramid convolutional structure, thereby simultaneously capturing the local and global geometric features of the point cloud, enabling the basic policy to reduce inference complexity while maintaining action quality.

[0015] As a preferred embodiment, the interactive attention residual weight network embeds the category of the most recent H-step trajectory segment. Compared with current observation features Fusion generates residual weights with the same dimension as the original action sequence through a multi-head cross-attention mechanism. The calculation expression is as follows:

[0016] In the formula, Residual weights It can be independently adjusted in each dimension of the motion, making the residual The amplification is achieved during the motion phase and the consistency is maintained during the skill phase; thus, the dexterous hand and the robotic arm can perform joint fine-tuning within the same joint space, as shown in the following calculation expression:

[0017] in, This indicates element-wise multiplication.

[0018] As a preferred approach, an asymptotic coefficient is also introduced to avoid perturbations caused by residual non-convergence in the early stages of training. The participation of the residual strategy is controlled by using joint fine-tuning with asymptotic coefficient P as the probability. Otherwise, only use steady control. Meanwhile, the probability is linearly increased with the number of task executions to achieve a smooth transition; the residual policy is optimized by the proximal policy PPO, and a weight regularization term is added to the objective function to suppress ineffective gains in the free space.

[0019] Secondly, a motion correction system for collaborative operation of a multi-finger dexterous hand and a robotic arm is provided, including: The training dataset creation module is used to collect motion data of multi-finger dexterity hand and robotic arm collaborative operation to create a training dataset covering multiple types of tasks; The basic policy training module is used to train basic policies based on the established training dataset using a robot vision imitation learning framework. The residual policy introduction module is used to introduce a lightweight residual policy on top of the basic policy. By adjusting the control signal through online reinforcement learning, it can achieve fine-grained correction of the contact phase while maintaining global smoothness. The residual weight adjustment module is used to enable the residual strategy to adaptively focus on the error at different stages through the interactive attention residual weight network. It not only achieves adaptive fusion of trajectory segment-observation features in the temporal dimension, but also outputs dimension-wise residual weights that are fully aligned with the joint angles in the action dimension, thus completing the action correction of the collaborative operation of the multi-finger dexterous hand and the robotic arm.

[0020] Thirdly, an electronic device is provided, including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the motion correction method for the coordinated operation of a multi-fingered dexterous hand and a robotic arm as described in the first aspect.

[0021] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing at least one instruction, which, when executed by a processor, implements the motion correction method for the coordinated operation of a multi-finger dexterous hand and a robotic arm as described in the first aspect.

[0022] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects: To address the problem that existing residual correction methods are limited and cannot meet the accuracy and stability requirements of high-degree-of-freedom dexterity operations, this invention proposes a motion correction method for collaborative operation between a multi-fingered dexterous hand and a robotic arm. By collecting motion data from the collaborative operation of the multi-fingered dexterous hand and the robotic arm and establishing a training dataset covering multiple tasks, a rich and comprehensive data foundation is provided for subsequent policy training. This effectively overcomes the limitations of single-data-source models and provides strong support for high-precision operations. Based on the established training dataset, a basic policy is trained using a robot vision imitation learning framework. A lightweight residual policy is then introduced on top of this basic policy. This composite policy system design maintains the smoothness of global motion during operation while enabling fine-grained correction of the contact phase through online reinforcement learning. This combined global and local optimization effectively solves the problem of traditional methods struggling to balance overall stability and local accuracy in complex operations, significantly improving the accuracy and stability of the operation. Meanwhile, the method of this invention, through an interactive attention residual weight network, enables the residual strategy to adaptively focus on errors at different stages. This network not only achieves adaptive fusion of trajectory segments and observation features in the temporal dimension, dynamically adjusting the correction strategy according to the characteristics of different stages, but also outputs dimension-wise residual weights perfectly aligned with joint angles in the action dimension, achieving precise correction of the joint movements of the multi-fingered dexterous hand and the robotic arm. This adaptive error correction mechanism greatly improves the flexibility and adaptability of the strategy, enabling it to better cope with various complex operation scenarios. By introducing trajectory segmentation and temporal information modeling, the method of this invention dynamically adjusts the residual weights in a stage-sensitive manner. Unlike traditional methods that rely on a single scaling factor, this method can output adaptive weights consistent with the action space dimension, coordinating and correcting the movements of the robotic arm and the multi-fingered dexterous hand at a fine-grained level. This differentiated correction method allows the strategy to be precisely adjusted according to the characteristics of different operation stages, improving the accuracy and robustness of the operation. The method of this invention also significantly enhances the generalization ability of the strategy in complex scenarios and long-term tasks, and can stably adapt to changes in different types of tasks and operating environments. It can be widely applied to robot dexterity operation tasks in manufacturing, assembly and other fields, providing a more efficient and reliable solution for industrial automation production, and has broad market application prospects and significant economic benefits.

[0023] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 The principle architecture diagram of introducing a residual strategy for fine-grained correction on the basis of the basic strategy in this embodiment of the invention; Figure 2 The design principle architecture diagram of the motion correction method for the collaborative operation of a multi-finger dexterous hand and a robotic arm in this invention is shown in the embodiment of the invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] To address the challenges of insufficient accuracy in long-term tasks and limited robustness in complex scenarios for robot dexterity operations, this invention proposes a motion correction method for collaborative operation between a multi-finger dexterity hand and a robotic arm. Targeting robot dexterity tasks, the proposed technical solution mainly involves introducing an adaptive residual strategy to perform fine-grained motion correction while providing a global motion reference based on a basic strategy, thereby achieving refined execution of dexterity tasks.

[0028] Please see Figure 1 At the training level, this invention first establishes a dataset covering multiple task types through joint data collection from a robotic arm and a dexterous hand. The basic strategy is trained on this dataset to acquire preliminary generalization ability for the tasks. Subsequently, during execution, the residual strategy is adaptively fine-tuned using temporal features and spatial information to compensate for the deficiencies of the basic strategy in high-dimensional action spaces in a stage-sensitive manner. Through this design, the robot can maintain efficient and stable operational performance in complex environments and long-term tasks. At the execution level, this invention can support a variety of typical dexterous operation tasks, including insertion (task 1), grasping (task 2), rotation (task 3), and transport (task 4). In each task, the basic strategy is responsible for generating the overall motion trajectory, while the residual strategy dynamically adjusts the residual weights of different dimensions according to the trajectory category and multi-step observation features, thereby coordinating and correcting the movements of the robotic arm and the multi-finger dexterous hand to ensure the stability and accuracy of the operation.

[0029] Please see Figure 2 The motion correction method for the coordinated operation of a multi-fingered dexterous hand and a robotic arm according to embodiments of the present invention includes: S1. Collect motion data of multi-finger dexterity hand and robotic arm working together to establish a training dataset covering multiple tasks; S2. Based on the established training dataset, the basic strategy is trained using a robot vision imitation learning framework; S3. A lightweight residual policy is introduced on the basis of the basic policy. The control signal is adjusted through online reinforcement learning to achieve fine-grained correction of the contact stage while maintaining global smoothness. S4. By using an interactive attention residual weight network, the residual strategy can adaptively focus on the errors at different stages. This not only achieves adaptive fusion of trajectory segment and observation features in the temporal dimension, but also outputs dimension-wise residual weights that are perfectly aligned with the joint angles in the action dimension, thus completing the action correction for the collaborative operation of the multi-finger dexterous hand and the robotic arm.

[0030] In one possible implementation, the robot visual imitation learning framework described in step S2 employs a 3D Diffusion Policy framework that combines 3D visual representation with a diffusion strategy. The 3D Diffusion Policy (3D-DP) framework is a robot visual imitation learning framework that combines 3D visual representation with a diffusion strategy. Its core lies in generating robust and generalizable action policies using compact 3D visual features and a diffusion model, suitable for complex robot operation tasks. The 3D-DP framework uses a depth camera to acquire depth information of the scene and extracts compact 3D visual features from sparse point clouds through an efficient point encoder. This 3D visual representation can capture the geometric structure of the scene, providing rich spatial information for action generation. The diffusion strategy is a type of conditional action generation model that gradually generates action sequences that conform to the target distribution by simulating the diffusion and denoising steps in the data generation process. In the 3D-DP framework, the diffusion strategy is used to map 3D visual features to the robot action space, realizing the conversion from vision to action. The basic policy trained using the 3D Diffusion Policy framework is... Although 3D Diffusion Policy can generate high-quality action sequences, it requires further processing under high-dimensional point clouds and complex interactions. In subsequent iterations, the inference overhead is too large, leading to delays in action generation and unstable contact, which is particularly serious in long-term hand-arm collaborative tasks.

[0031] To address this, embodiments of the present invention introduce a consistency model during the training phase, distilling the original multi-step sampling 3DDiffusion Policy framework into a single-step prediction, thereby significantly reducing inference complexity and improving real-time performance.

[0032] Let the clean action sequence be In terms of noise scale n Noisy samples were obtained The calculation expression is as follows:

[0033] In the formula, , This is the cumulative retention factor; Approximately noise-free samples are obtained from the teacher model using a K-step ordinary differential equation ODE solver. :

[0034] A K-step ODE solver is a tool for solving ordinary differential equations (ODEs) with high accuracy or adaptive step size by calculating multiple intermediate slopes in each step using numerical integration methods (such as the Runge-Kutta method). Its core lies in using a weighted average of multi-step slopes to approximate the true solution, thereby improving accuracy while maintaining efficiency. As can be seen from the consistency model, the expression obtained above... and Features are consistent across low-dimensional manifolds; therefore, to minimize cross-scale noise prediction discrepancies, a consistency network is needed. Noise prediction model for teacher strategies is needed. Learn the following loss functions:

[0035] This ensures The predictions are consistent across different noise levels, and the action distribution is constrained to a physically feasible low-dimensional manifold; the inference phase involves only single-step computation, directly based on observations. To obtain high-quality actions, the calculation expression is as follows:

[0036] In the formula, This is the observation state encoder.

[0037] Furthermore, in implementation, this invention replaces the Multilayer Perceptron (MLP) encoder in the 3D Diffusion Policy framework with a pyramid convolutional structure to simultaneously capture local and global geometric features of the point cloud, thereby obtaining a more hierarchical and robust representation. An MLP encoder typically consists of an input layer, one or more hidden layers, and an output layer. The input layer receives raw data or features and passes them to the next layer. The hidden layer is the intermediate layer connecting the input and output layers; each hidden layer contains multiple neurons (nodes), each neuron is connected to all nodes in the previous layer, and outputs a weighted sum of values ​​processed by an activation function. The output layer generates the final feature representation, which can be used for subsequent classification, regression, or other tasks. The pyramid convolutional structure, on the other hand, is a deep learning convolutional architecture designed to mimic the visual principle of a pyramid. Through multi-scale feature extraction and hierarchical information fusion, it significantly improves the model's perception capability and computational efficiency in complex scenes. The core inspiration for the pyramid convolutional structure comes from the multi-scale image processing mechanism of the human visual system. It simulates the information abstraction process of a pyramid from the bottom layer (details) to the top layer (semantics) by constructing multi-level convolutional kernels of different scales, enabling the model to simultaneously capture local details and global contextual information. This invention's embodiments, through this design, enable the basic strategy... It can significantly reduce reasoning complexity while maintaining action quality, providing solid support for the real-time execution of dexterous operations.

[0038] In one possible implementation, although the basic strategy of uniform distillation acceleration via step S2... It can generate smooth control in most cases. However, its training relies on offline data and lacks crucial hand-object contact information from online interactions. In real-world implementation, this deficiency often leads to the inability to promptly correct out-of-distribution states, resulting in a gradual accumulation of prediction bias. To compensate for this deficiency, step S3 of this embodiment introduces a lightweight residual strategy. By adjusting the control signal through online reinforcement learning, fine-grained corrections can be made to the contact phase while maintaining global smoothness.

[0039] In one possible implementation, to enable the residual strategy to adaptively focus on errors at different stages, step S4 of this embodiment designs an interactive attention residual weight network, which embeds the category of the most recent H-step trajectory segment. Compared with current observation features Fusion generates residual weights with the same dimension as the original action sequence through a multi-head cross-attention mechanism. The calculation expression is as follows:

[0040] Unlike uniform scaling, in the formula, Residual weights It can be independently adjusted in each dimension of the motion, making the residual The amplification is achieved during the motion phase and the consistency is maintained during the skill phase; thus, the dexterous hand and the robotic arm can perform joint fine-tuning within the same joint space, as shown in the following calculation expression:

[0041] in, This indicates element-wise multiplication.

[0042] Furthermore, in order to avoid perturbations caused by residual non-convergence in the early stages of training, this embodiment of the invention introduces an asymptotic coefficient. The participation of the residual strategy is controlled by using joint fine-tuning with asymptotic coefficient P as the probability. Otherwise, only use steady control. Meanwhile, the probability is linearly increased with the number of task executions to achieve a smooth transition; the residual policy is optimized by the proximal policy PPO, and a weight regularization term is added to the objective function to suppress ineffective gains in the free space.

[0043] The core of PPO optimization lies in balancing training stability and efficiency by limiting the policy update magnitude. Its optimization strategies encompass objective function design, sampling strategies, parameter tuning, network structure optimization, and training process improvements. Specifically: (1) Regarding the objective function design, PPO limits the magnitude of policy updates by designing a special objective function to prevent excessively large update steps from causing training instability. The objective function of PPO usually includes two parts: a policy gradient term with clipping constraints and a value function error term. Clipping techniques are used to limit the range of policy ratio changes and prevent excessively large update steps. (2) Regarding the sampling strategy, PPO uses importance sampling techniques to make full use of the data generated by the old policy. By calculating the weight ratio between the old and new policies, PPO can use old data to estimate the expected return under the new policy. This strategy not only improves data utilization but also reduces the computational cost of resampling. (3) Regarding parameter tuning and optimization, the learning rate is one of the key parameters affecting the training effect of PPO. An excessively large learning rate may lead to training instability, while an excessively small learning rate may lead to low training efficiency. Therefore, it is necessary to adjust the learning rate according to the specific task to find the best balance point. The clipping parameter is used to control the magnitude of policy updates. In practical applications, the value of the clipping parameter can be adjusted to balance training stability and efficiency. In addition, the accuracy of the advantage function estimation has an important impact on the training effect of PPO. Commonly used advantage function estimation methods include temporal difference (TD) estimation and generalized advantage estimation (GAE). GAE, by combining the temporal difference method and discount factor, can achieve a smooth estimation of the advantage at each time step, thereby improving training stability. (4) Regarding network structure optimization, separating the policy network and the value network can avoid mutual influence between the two, thereby improving training performance. The policy network is responsible for outputting action probabilities, while the value network is responsible for evaluating state values. In addition, increasing the depth and width of the network can improve the expressive power of the model, but may also increase the training difficulty and computational cost. Therefore, it is necessary to select an appropriate network structure according to the specific task. (5) Regarding training process optimization, PPO allows the same data to be used for policy updates multiple times in each iteration. This strategy not only improves data utilization but also speeds up training. In addition, during training, when the policy performance no longer improves or reaches the preset number of iterations, training can be stopped in advance to avoid overfitting and wasting computational resources.

[0044] Another embodiment of the present invention provides a motion correction system for collaborative operation of a multi-finger dexterous hand and a robotic arm, comprising: The training dataset creation module is used to collect motion data of multi-finger dexterity hand and robotic arm collaborative operation to create a training dataset covering multiple types of tasks; The basic policy training module is used to train basic policies based on the established training dataset using a robot vision imitation learning framework. The residual policy introduction module is used to introduce a lightweight residual policy on top of the basic policy. By adjusting the control signal through online reinforcement learning, it can achieve fine-grained correction of the contact phase while maintaining global smoothness. The residual weight adjustment module is used to enable the residual strategy to adaptively focus on the error at different stages through the interactive attention residual weight network. It not only achieves adaptive fusion of trajectory segment-observation features in the temporal dimension, but also outputs dimension-wise residual weights that are fully aligned with the joint angles in the action dimension, thus completing the action correction of the collaborative operation of the multi-finger dexterous hand and the robotic arm.

[0045] In one possible implementation, the basic policy training module of this embodiment of the invention uses a 3D Diffusion Policy framework that combines 3D visual representation and diffusion strategy to train the basic policy.

[0046] Furthermore, the action correction system of this embodiment introduces a consistency model during the training phase, distilling the original multi-step sampling 3D Diffusion Policy framework into single-step prediction. Let the clean action sequence be In terms of noise scale n Noisy samples were obtained The calculation expression is as follows:

[0047] In the formula, , This is the cumulative retention factor; Approximately noise-free samples are obtained from the teacher model using a K-step ordinary differential equation ODE solver. :

[0048] Received and Features are consistent across low-dimensional manifolds. To minimize cross-scale noise prediction discrepancies, a consistency network is used. Noise prediction model based on teacher strategies Learn the following loss functions:

[0049] This ensures The predictions are consistent across different noise levels, and the action distribution is constrained to a physically feasible low-dimensional manifold; the inference phase involves only single-step computation, directly based on observations. To obtain high-quality actions, the calculation expression is as follows:

[0050] In the formula, This is the observation state encoder.

[0051] Furthermore, in this embodiment of the invention, the multilayer perceptron (MLP) encoder of the 3D Diffusion Policy framework is replaced with a pyramid convolutional structure, thereby simultaneously capturing the local and global geometric features of the point cloud, enabling the basic policy to reduce inference complexity while maintaining action quality.

[0052] In one possible implementation, when the residual weight adjustment module of this embodiment enables the residual strategy to adaptively focus on errors at different stages through an interactive attention residual weight network, the interactive attention residual weight network embeds the category of the most recent H-step trajectory segment. Compared with current observation features Fusion generates residual weights with the same dimension as the original action sequence through a multi-head cross-attention mechanism. The calculation expression is as follows:

[0053] In the formula, Residual weights It can be independently adjusted in each dimension of the motion, making the residual The amplification is achieved during the motion phase and the consistency is maintained during the skill phase; thus, the dexterous hand and the robotic arm can perform joint fine-tuning within the same joint space, as shown in the following calculation expression:

[0054] in, This indicates element-wise multiplication.

[0055] Meanwhile, to avoid perturbations caused by residual non-convergence in the early stages of training, this embodiment of the invention introduces an asymptotic coefficient. The participation of the residual strategy is controlled by using joint fine-tuning with asymptotic coefficient P as the probability. Otherwise, only use steady control. Meanwhile, the probability is linearly increased with the number of task executions to achieve a smooth transition; the residual policy is optimized by the proximal policy PPO, and a weight regularization term is added to the objective function to suppress ineffective gains in the free space.

[0056] Ultimately, this invention not only achieves adaptive fusion of trajectory segments and observed features in the temporal dimension, but also outputs dimension-wise residual weights perfectly aligned with joint angles in the action dimension, enabling joint fine-tuning of the multi-finger dexterous hand and robotic arm in a high-dimensional action space. This design improves the flexibility and accuracy of the strategy during critical contact phases and significantly enhances its applicability in real-world environments, allowing the robot to perform fine-grained dexterous operations with lower inference latency.

[0057] Another embodiment of the present invention provides an electronic device including a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the motion correction method for the coordinated operation of the multi-finger dexterous hand and the robotic arm.

[0058] Another embodiment of the present invention provides a computer-readable storage medium storing at least one instruction, which, when executed by a processor, implements the motion correction method for the collaborative operation of the multi-finger dexterous hand and the robotic arm.

[0059] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For ease of explanation, the above content only shows the parts related to the embodiments of the present invention; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in storage devices formed by various electronic devices, enabling the execution process described in the method of the embodiments of the present invention.

[0060] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for correcting the motion of a multi-fingered hand in cooperation with a robot arm, characterized by, The method comprises the steps of: Collecting motion data of multi-fingered hand and robot arm cooperative operation, and establishing a training data set covering multiple types of tasks; Training a basic strategy based on the established training data set using a robot visual imitation learning framework; Introducing a lightweight residual strategy on the basis of the strategy, adjusting the control signal through online reinforcement learning, and achieving fine-grained correction of the contact phase while maintaining global smoothness; Through the interactive attention residual weight network, the residual strategy can adaptively focus on the errors of different stages, not only achieving adaptive fusion of trajectory segments-observation features in the time dimension, but also outputting per-dimension residual weights that are completely aligned with joint angles in the action dimension, thus completing the motion correction of multi-fingered hand and robot arm cooperative operation.

2. The motion correction method for cooperative operation of a multi-fingered hand and a robot arm according to claim 1, characterized in that, In each task, the basic strategy generates the overall motion trajectory, and the residual strategy dynamically adjusts the residual weights of different dimensions according to the motion trajectory category and multi-step observation features, thereby coordinating and correcting the motion of multi-fingered hand and robot arm.

3. The motion correction method for cooperative operation of a multi-fingered hand and a robot arm according to claim 1, characterized in that, The robot visual imitation learning framework uses a 3D Diffusion Policy framework that combines 3D visual representation and diffusion strategy.

4. The method according to claim 3, wherein In the step of training a basic strategy based on the established training data set using a robot visual imitation learning framework, a consistency model is introduced in the training stage to distill the original multi-step sampling 3D Diffusion Policy framework into single-step prediction; Let the clean sequence of actions be At noise scale n The noisy samples are obtained The computation expression is as follows: In the formula, , is the cumulative retention factor; approximate noise-free samples by the teacher model through a K-step ordinary differential equation (ODE) solver : obtained and The characteristics on low-dimensional manifold are consistent, in order to minimize the cross-scale noise prediction difference, the consistency network From the noise prediction model of the teacher strategy The following loss function is learned: This ensures Predicts consistently across different noise levels and constrains the action distribution to a low-dimensional manifold of physically feasible actions; the inference phase only requires a single-step computation directly from the observation Obtains high-quality actions, computed by the expression In the formula, is an observation state encoder.

5. The method according to claim 4, wherein In the step of training a basic strategy based on the established training data set using a robot visual imitation learning framework, the multi-layer perception MLP encoder of the 3D Diffusion Policy framework is replaced by a pyramid convolution structure, thereby simultaneously capturing local and global geometric features of the point cloud, enabling the basic strategy to reduce inference complexity while maintaining motion quality.

6. The motion correction method for cooperative operation of a multi-fingered hand and a robot arm according to claim 4, characterized in that, The interaction attention residual weight network embeds the recent H-step trajectory segment class with the current observation feature Fusion, generate residual weights with the same dimension as the original action sequence through multi-head cross attention mechanism The calculation expression is as follows: In the formula, , residual weight Each dimension of action can be independently adjusted, so that the residual In the motion segment, it is amplified, and in the skill segment, it is consistent; thus, the dexterous hand and the mechanical arm can jointly fine-tune in the same joint space, and the calculation expression is as follows: wherein denotes element-wise multiplication.

7. The method according to claim 6, wherein Also include the introduction of progressive coefficient to avoid disturbance caused by residual non-convergence in early training Control the participation of residual strategy: use joint fine-tuning with progressive coefficient P as probability , otherwise only use stationary control , while linearly increasing the probability with the number of task executions to achieve a smooth transition; optimize the residual strategy through proximal policy optimization (PPO), and add a weight regularization term in the objective function to suppress invalid gains in free space.

8. A motion correction system for a multi-fingered hand operating in cooperation with a robot arm, characterized by The method comprises the steps of: A training data set establishment module is used to collect motion data of multi-fingered hand and robot arm cooperative operation, and establish a training data set covering multiple types of tasks; A basic strategy training module is used to train a basic strategy based on the established training data set using a robot visual imitation learning framework; A residual strategy introduction module is used to introduce a lightweight residual strategy on the basis of the strategy, adjust the control signal through online reinforcement learning, and achieve fine-grained correction of the contact phase while maintaining global smoothness; A residual weight adjustment module is used to make the residual strategy adaptively focus on the errors of different stages through an interactive attention residual weight network, not only achieving adaptive fusion of trajectory segments-observation features in the time dimension, but also outputting per-dimension residual weights that are completely aligned with joint angles in the action dimension, thus completing the motion correction of multi-fingered hand and robot arm cooperative operation.

9. An electronic device, comprising: A processor and a memory are included, and the processor is used to execute a computer program stored in the memory to implement the motion correction method of multi-fingered hand and robot arm cooperative operation as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one instruction, and the at least one instruction is executed by the processor to implement the action correction method of the multi-fingered hand and the mechanical arm cooperative operation as claimed in any one of claims 1 to 7.

Images