Dexterous hand control method, device, intelligent agent and storage medium

Abstract

The embodiment of the present application provides a dexterous hand control method, device, intelligent agent and storage medium, the method includes acquiring sensor data, preprocessing the sensor data, obtaining input information, the input information includes environmental information and state information of the dexterous hand, inputting the input information into the cerebellum model, obtaining corresponding control instructions, the cerebellum model includes a trajectory planning model and a motion controller, the trajectory planning model is used to output the corresponding trajectory planning result according to the input information, the motion controller is used to convert the trajectory planning result into the corresponding control instruction, the motion controller is obtained by combining the cerebellum model neural network with the whole body dynamics model based on the model predictive control algorithm, and the actuator module of the dexterous hand is driven and controlled according to the control instruction. The dexterous hand control method provided by the embodiment of the present application can improve the flexibility and accuracy of control.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of humanoid robot dexterous hands, and in particular to a dexterous hand control method, device, intelligent body and storage medium. Background Art

[0002] With the rapid advancement of science and technology, robotics is developing at an unprecedented speed. As an outstanding representative in this field, dexterous hands have impressive complex and delicate operation capabilities. However, this high degree of freedom also brings unprecedented control challenges, especially in mission scenarios that require rapid response and high coordination. Traditional control strategies, although they perform well in certain scenarios, are often limited by complex mathematical models and fixed algorithm frameworks, making it difficult to flexibly respond to changing operating environments.

[0003] In the related art, traditional control strategies, such as proportional, integral and differential (PID) control, classical adaptive control, open-loop control, etc., can be used to control the dexterous hand.

[0004] However, in the process of realizing the present application, the inventors found that there are at least the following problems in the prior art: for dexterous hands with high degrees of freedom, the above-mentioned method lacks accuracy and flexibility in task scenarios requiring rapid response and high coordination. Summary of the invention

[0005] The present application provides a dexterous hand control method, device, intelligent agent and storage medium to improve the accuracy and flexibility of dexterous hand control.

[0006] In a first aspect, the present application provides a dexterous hand control method, comprising:

[0007] Get sensor data;

[0008] Preprocessing the sensor data to obtain input information; the input information includes environmental information and state information of the dexterous hand;

[0009] The input information is input into the cerebellum model to obtain corresponding control instructions; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output the corresponding trajectory planning result according to the input information, and the motion controller is used to convert the trajectory planning result into the corresponding control instruction; the motion controller is obtained by combining the cerebellum model neural network with the whole body dynamics model based on the model predictive control algorithm;

[0010] The actuator module of the dexterous hand is driven and controlled according to the control instruction.

[0011] In one possible design, the environmental information includes at least one of the following parameters of an object in the real environment where the dexterous hand is located: position, shape, and size; the state information includes at least one of the following parameters: position, strength, speed, and acceleration of the fingers of the dexterous hand.

[0012] In a possible design, before inputting the input information into the cerebellum model to obtain the corresponding control instruction, the method further includes:

[0013] Construct a physical model based on the physical structure of the dexterous hand and the physical structure of the robotic arm;

[0014] Construct the cerebellum neural network to be trained and the trajectory planning model to be trained;

[0015] Obtain a training sample set;

[0016] Based on the training sample set and the physical model, the cerebellum neural network to be trained and the trajectory planning model to be trained are trained to obtain the cerebellum model neural network and the trajectory planning model;

[0017] The cerebellum model neural network is combined with a whole-body dynamics model based on a model predictive control algorithm to obtain a motion controller, and the cerebellum model is determined according to the motion controller and the trajectory planning model.

[0018] In one possible design, the model-based predictive control algorithm combines the cerebellum model neural network with the whole body dynamics model to obtain a motion controller, including:

[0019] The inverse kinematics module and the inverse dynamics module are connected in series and then connected in parallel with the cerebellum model neural network to obtain a motion controller; wherein the inverse dynamics module is also used to be connected in series with the forward dynamics module and the forward kinematics module in sequence, the forward dynamics module is used to feedback output to the inverse dynamics module, and the forward kinematics module is used to feedback output to the inverse kinematics module and the cerebellum model neural network.

[0020] In one possible design, the model-based predictive control algorithm combines the cerebellum model neural network with the whole body dynamics model to obtain a motion controller, including:

[0021] The cerebellum model neural network is connected in series with the positive dynamics module and the positive kinematics module in sequence to obtain a motion controller; wherein the positive kinematics module is used to feedback output to the cerebellum model neural network.

[0022] In one possible design, the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7.

[0023] In a possible design, obtaining a training sample set includes:

[0024] The dexterous hand and the robotic arm are controlled by the teleoperation system to perform grasping and manipulation tasks in multiple scenarios, and sensor data is collected during the task execution process; the task objects in different scenarios are different or the operation environment is different; different task objects refer to the task objects having at least one different parameter: shape, material, position;

[0025] The sensor data is preprocessed to obtain a training sample set.

[0026] In a possible design, the training sample set includes multiple groups of image sequences and corresponding expected trajectories, and multiple groups of trajectories and corresponding expected torques; based on the training sample set and the physical model, the cerebellum neural network to be trained and the trajectory planning model to be trained are trained to obtain the cerebellum model neural network and the trajectory planning model, including:

[0027] Based on a first loss function and a back propagation algorithm, the cerebellum neural network to be trained is trained according to multiple groups of trajectories and corresponding expected torques to obtain the cerebellum model neural network; the first loss function includes the error between the actual torque output by the cerebellum neural network to be trained and the expected torque;

[0028] Based on the second loss function and the back propagation algorithm, the trajectory planning model to be trained is trained according to multiple groups of the image sequences and the corresponding expected trajectories to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

[0029] In a possible design, the training sample set includes multiple groups of trajectories and corresponding expected torques, and multiple groups of image sequences and corresponding expected torques; the training of the cerebellum neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellum model neural network and the trajectory planning model includes:

[0030] Based on a first loss function and a back propagation algorithm, the cerebellum neural network to be trained is trained according to multiple groups of trajectories and corresponding expected torques to obtain an initial cerebellum model neural network; the first loss function includes an error between an actual torque output by the cerebellum neural network to be trained and the expected torque;

[0031] Based on the second loss function and the back-propagation algorithm, the trajectory planning model to be trained and the initial cerebellum model neural network are jointly trained according to multiple groups of the image sequences and the corresponding expected torques to obtain the trajectory planning model and the cerebellum model neural network; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

[0032] In a second aspect, the present application provides a dexterous hand control device, comprising:

[0033] An acquisition module, used to acquire sensor data;

[0034] A determination module, used to pre-process the sensor data to obtain input information; the input information includes environmental information and state information of the dexterous hand;

[0035] A processing module is used to input the input information into the cerebellum model to obtain corresponding control instructions; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output the corresponding trajectory planning result according to the input information, and the motion controller is used to convert the trajectory planning result into the corresponding control instruction; the motion controller is obtained by combining the cerebellum model neural network with the whole body dynamics model based on the model predictive control algorithm;

[0036] A drive control module is used to drive and control the actuator module of the dexterous hand according to the control instruction.

[0037] In a third aspect, the present application provides a dexterous hand control device, comprising: at least one processor and a memory;

[0038] The memory stores computer-executable instructions;

[0039] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect and various possible designs of the first aspect.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect is implemented.

[0041] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect and various possible designs of the first aspect.

[0042] The present application provides a dexterous hand control method, device, intelligent agent and storage medium. The method includes acquiring sensor data, preprocessing the sensor data, obtaining input information, the input information including environmental information and state information of the dexterous hand, inputting the input information into a cerebellum model, obtaining corresponding control instructions, the cerebellum model including a trajectory planning model and a motion controller, the trajectory planning model is used to output corresponding trajectory planning results according to the input information, the motion controller is used to convert the trajectory planning results into corresponding control instructions, the motion controller is obtained by combining the cerebellum model neural network with the whole body dynamics model based on a model predictive control algorithm, and the actuator module of the dexterous hand is driven and controlled according to the control instructions.

[0043] The dexterous hand control method provided in the present application, by introducing the cerebellum mechanism and combining the cerebellum model neural network with the whole-body dynamics model for model predictive control, can improve the flexibility and accuracy of control, enable the dexterous hand to perform various complex tasks more accurately, help the dexterous hand to quickly adapt to different operating environments and task requirements, reduce dependence on precise mathematical models, and thus simplify the design and implementation of the control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1 A schematic diagram of a scenario of a dexterous hand control method provided in an embodiment of the present disclosure;

[0046] Figure 2 A schematic diagram of a flow chart of a dexterous hand control method provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a flow chart of a training method for a cerebellum model provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of a trajectory planning model to be trained provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of a neural network to be trained for a cerebellum model provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of the model predictive control based on the whole body dynamics model provided in the embodiment of the present application;

[0051] Figure 7 Schematic diagram of the structure of the cerebellum model provided in the embodiment of the present application Figure 1 ;

[0052] Figure 8 Schematic diagram of the structure of the cerebellum model provided in the embodiment of the present application Figure 2 ;

[0053] Fig. 9 Schematic diagram of the structure of the cerebellum model provided in the embodiment of the present application Figure 3 ;

[0054] Fig.10 A schematic diagram of the structure of a dexterous hand control device provided in an embodiment of the present application;

[0055] Fig.11 A schematic diagram of the hardware structure of the dexterous hand control device provided in an embodiment of the present application.

[0056] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] It should be noted that the dexterous hand control method provided in the present application can be used in the field of humanoid robot technology, and can also be used in any field other than the field of humanoid robot technology. The application field of the dexterous hand control method provided in the present application is not limited.

[0059] With the rapid advancement of science and technology, robotics is developing at an unprecedented speed, and high-degree-of-freedom dexterous hands, as an outstanding representative in this field, have impressive complex and delicate operating capabilities. However, this high degree of freedom also brings unprecedented control challenges, especially in mission scenarios that require rapid response and high coordination. Traditional control strategies, although they perform well in certain scenarios, are often limited by complex mathematical models and fixed algorithm frameworks, making it difficult to flexibly respond to changing operating environments.

[0060] In order to solve the above technical problems, the inventors of the present application have found that the cerebellum, as a key part responsible for movement coordination in an organism, exhibits excellent learning and adaptive capabilities. It can adjust the movement strategy in a short time to accurately adapt to changes in the external environment. Therefore, the learning mechanism of the cerebellum can be integrated into the field of robot control, especially the control of high-degree-of-freedom dexterous hands. In the control of high-degree-of-freedom dexterous hands, the introduction of the cerebellum learning mechanism can bring many advantages. First, it can improve the flexibility and accuracy of control, so that the dexterous hands can perform various complex tasks more accurately. Secondly, since the cerebellum learning mechanism has a strong adaptive ability, it can help the dexterous hands quickly adapt to different operating environments and task requirements. Finally, this control strategy is also expected to reduce the dependence on accurate mathematical models, thereby simplifying the design and implementation of the control system. In addition, in order to ensure accuracy, the cerebellum model neural network can be combined with the whole-body dynamics model for model predictive control (MPC) to further improve the control accuracy. Based on this, the embodiment of the present application provides a dexterous hand control method.

[0061] Figure 1 Schematic diagram of a scenario of the dexterous hand control method provided by an embodiment of the present disclosure. Figure 1 As shown, the dexterous hand control system includes an intelligent body and a cerebellum model. The intelligent body may include a sensor module and an actuator module, and the cerebellum model may include a trajectory planning model and a motion controller.

[0062] In the specific implementation process, the sensor module collects sensor data, the control device obtains sensor data, pre-processes the sensor data, and obtains input information; the input information includes environmental information and state information of the dexterous hand; the input information is input into the cerebellum model to obtain corresponding control instructions; in the process of the cerebellum model processing the input information, the trajectory planning model outputs the corresponding trajectory planning result according to the input information, and the motion controller converts the trajectory planning result into the corresponding control instruction, wherein the motion controller is obtained by combining the cerebellum model neural network with the whole body dynamics model based on the model predictive control algorithm. The control device outputs the control instruction (including the joint torque of the dexterous hand and the mechanical arm) to the actuator module, and the actuator module drives the dexterous hand and the mechanical arm according to the control instruction to complete tasks such as grasping. The dexterous hand control method provided in the embodiment of the present application can improve the flexibility and accuracy of the control by introducing the cerebellum mechanism and combining the cerebellum model neural network with the whole body dynamics model for model predictive control, so that the dexterous hand can perform various complex tasks more accurately, help the dexterous hand quickly adapt to different operating environments and task requirements, and reduce the dependence on accurate mathematical models, thereby simplifying the design and implementation of the control system.

[0063] It should be noted that Figure 1 The scene diagram shown is only an example. The dexterous hand control method and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0064] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0065] Figure 2 This is a flow chart of the dexterous hand control method provided in the embodiment of the present application. Figure 2 As shown, the method includes:

[0066] 201. Obtain sensor data.

[0067] The execution subject of this embodiment may be an intelligent entity such as a humanoid robot.

[0068] In this embodiment, the sensor data may include environmental information (such as object position, shape, size) collected by sensors such as position sensors, visual sensors, force sensors, and speed sensors, as well as key data such as finger position, force, and speed.

[0069] 202. Preprocess the sensor data to obtain input information; the input information includes environmental information and state information of the dexterous hand.

[0070] Specifically, after obtaining the sensor data, the sensor data may be pre-processed by at least one of filtering, denoising, normalization, data cleaning, data segmentation, etc. to obtain input information.

[0071] Exemplarily, during the filtering process, different types of filters such as low-pass filters, high-pass filters, and band-pass filters can be selected according to the characteristics and requirements of the data to remove high-frequency noise and interference in the data. During the denoising process, methods such as mean filtering, median filtering, and Kalman filtering can be used for denoising to reduce noise and outliers in the data. During the normalization process, linear normalization, nonlinear normalization, and other methods can be used to convert the data into values ​​within a specific range (e.g., 0 to 1 or -1 to 1) for subsequent analysis and processing. During the data cleaning process, duplicate data can be removed, missing data can be processed, and erroneous data can be corrected, wherein duplicate data can be identified and deleted by comparing data values ​​and timestamps; missing data can be filled by interpolation, regression, and other methods; erroneous data can be corrected by setting thresholds or using other methods. In the data segmentation process, data segmentation can be performed according to conditions such as timestamps, task types, and object types for more detailed analysis and processing.

[0072] In some embodiments, the environmental information includes at least one of the following parameters of an object in the real environment where the dexterous hand is located: position, shape, and size; the state information includes at least one of the following parameters: position, strength, speed, and acceleration of the fingers of the dexterous hand.

[0073] 203. Input the input information into the cerebellum model to obtain corresponding control instructions; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output the corresponding trajectory planning results according to the input information, and the motion controller is used to convert the trajectory planning results into corresponding control instructions; the motion controller is obtained by combining the cerebellum model neural network with the whole body dynamics model based on a model predictive control algorithm.

[0074] Specifically, the environmental information and dexterous hand state information extracted from the sensor data are input into the trained cerebellum model. The trajectory planning model outputs the corresponding trajectory planning results according to the input information, and the motion controller converts the trajectory planning results into control instructions through adaptive linear mapping.

[0075] 204. Drive and control the actuator module of the dexterous hand according to the control instruction.

[0076] Specifically, after obtaining the control command, the control command is sent to the actuator module of the dexterous hand to drive the dexterous hand to complete tasks such as grasping and manipulation. In this process, motion planning and coordination are carried out, including planning the motion trajectory of the dexterous hand and the coordinated movements of the fingers, to ensure that the dexterous hand can complete the task according to the predetermined path and movement. Using real-time feedback and online adjustment mechanisms, the control strategy is continuously optimized to adapt to changes in different environments and task requirements.

[0077] The dexterous hand control method provided in this embodiment, by introducing the cerebellum mechanism and combining the cerebellum model neural network with the whole-body dynamics model for model predictive control, can improve the flexibility and accuracy of control, enable the dexterous hand to perform various complex tasks more accurately, help the dexterous hand to quickly adapt to different operating environments and task requirements, reduce dependence on precise mathematical models, and thus simplify the design and implementation of the control system.

[0078] Figure 3 Schematic diagram of the flow chart of the training method of the cerebellum model provided in the embodiment of the present application. Figure 3 As shown, the method includes:

[0079] 301. Construct a physical model based on the physical structure of the dexterous hand and the physical structure of the robotic arm.

[0080] Specifically, according to the actual physical structure of the dexterous hand and the robotic arm, a three-dimensional model is constructed as a physical model, which may include the geometric dimensions and physical properties of the fingers, joints, drive mechanisms, etc. The kinematic model and dynamic model of the dexterous hand and the robotic arm are established using the kinematic equations and dynamic equations. These models describe the motion laws and force conditions of the dexterous hand in different postures.

[0081] Exemplarily, a kinematic model based on DH can be constructed, wherein the forward kinematics formula of the robotic arm and the dexterous hand can be:

[0082] (1)

[0083] in, is the input joint angle, The output position of the robotic arm and dexterous hand.

[0084] The inverse kinematics formula for the robotic arm and hand is:

[0085] (2)

[0086] in, is the output joint angle, is the input pose of the robotic arm and dexterous hand.

[0087] A Lagrangian-based dynamic model can be constructed: Among them, the positive dynamic model of the robotic arm and dexterous hand is:

[0088] (3)

[0089] in, is the input driving torque, The output joint angles of the robotic arm and dexterous hand;

[0090] The inverse dynamics model of the robotic arm and dexterous hand is:

[0091] (4)

[0092] in, is the output driving torque, are the input joint angles of the robotic arm and dexterous hand.

[0093] 302. Construct a cerebellum neural network to be trained and a trajectory planning model to be trained.

[0094] Specifically, the trajectory planning model to be trained uses an imitation learning method to extract trajectory features from the operation data of humans or other dexterous hands, and encodes and decodes the trajectory based on deep learning techniques (such as Convolutional Neural Network (CNN) and the attention-based sequence model Transformer) to generate trajectories suitable for different tasks. CNN can be used to extract the spatial features of the trajectory, while Transformer can capture the temporal dependency of the trajectory.

[0095] In some embodiments, the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7. The trajectory planning system combines the advantages of CNN and Transformer, and can effectively extract information from image sequences and predict future trajectories of robotic arms and dexterous hands. The system has good scalability and flexibility, and can be adjusted and optimized according to specific application scenarios.

[0096] Specifically, the input of the trajectory planning model to be trained includes image sequences (visual information collected by visual sensors) and the corresponding positions of the robotic arm and dexterous hand (state information collected by position sensors, etc.), and random variables. The output of the trajectory planning model to be trained includes the predicted trajectory sequence and random variables.

[0097] For example, Figure 4 As shown, the input of the trajectory planning system can be defined as an image sequence , x n is the nth image, n is a positive integer greater than 1, Image information x i ∈R w×h×d , Represents the width, height and depth of the image (for example, the size can be 3, 480, 630 respectively). The output of the trajectory planning system can be defined as trajectory sequence information ,t n is the nth trajectory, and n is a positive integer greater than 1. Each trajectory contains the end position of the robot arm (e.g., 7 dimensions) and the bending angle of the dexterous hand (e.g., 15 dimensions). The trajectory planning system can adopt the Mobile ALHOA algorithm, and the backbone network is composed of CNN and Transformer. CNN adopts ResNet18, Transformer Encoder adopts a 4-layer self-attention network, and Transformer Deconder adopts a 7-layer cross-attention network.

[0098] The cerebellar neural network to be trained may include an input layer, an intermediate layer (such as a cerebellar cortex simulation layer), and an output layer. The intermediate layer may contain multiple cerebellar units, each of which processes the input nonlinearly through sparse connections and local generalization capabilities.

[0099] For example, Figure 5 As shown in the figure, the input of the cerebellum neural network to be trained is trajectory information, and the output is information such as joint torque. The cerebellum neural network to be trained includes a virtual associative space (AC) and a physical storage space (AP). Through the conceptual mapping and actual mapping of these two spaces, nonlinear mapping is achieved based on local approximation and associative functions. Among them, the virtual associative space (AC) is a mapping of the input space, which is used to store the quantized encoded information of the input state. After quantization encoding, the input vector is mapped to multiple storage units in AC. There is partial overlap between these storage units, and similar inputs will stimulate more overlapping units in AC. The physical storage space (AP) stores the weights of network training. Each storage unit in AC corresponds to a weight in AP. The output of the network is the sum of the weights of all stimulated storage units in AP. Compared with traditional control algorithms, control algorithms based on cerebellum models have faster efficiency and accuracy.

[0100] 303. Obtain a training sample set.

[0101] Specifically, the teleoperation system can be used to control the dexterous hand and the robotic arm to perform grasping and manipulation tasks in different scenarios (such as objects of different shapes, materials, postures and different operating environments). Environmental information (such as object position, shape, size), finger position, force, speed and other sensor data are collected in real time. The collected data is preprocessed (such as filtering, denoising, normalization, etc.) to improve data quality and usability.

[0102] In some embodiments, in order to ensure the authenticity and richness of training data and improve training accuracy, specially collected data in different scenarios can be used as training samples. Specifically, the acquisition of training sample sets can include: controlling the dexterous hand and the robotic arm to perform grasping and operation tasks in multiple scenarios through a teleoperation system, and collecting sensor data during the execution of the tasks; different task objects or different operating environments in different scenarios; different task objects refer to task objects with different at least one of the following parameters: shape, material, position; preprocessing the sensor data to obtain a training sample set.

[0103] 304. Based on the training sample set and the physical model, train the cerebellum neural network to be trained and the trajectory planning model to be trained to obtain the cerebellum model neural network and the trajectory planning model.

[0104] Specifically, during the training process, the sensor data collected by teleoperation can be input into the trajectory planning model to be trained, and the CNN and transformer of the model can be used to generate predicted trajectory features. Then, the generated predicted trajectory features are used as input, combined with environmental information and dexterous hand state information, and input into the motion controller model. The motion controller model (cerebellum neural network to be trained, whole body dynamics model and model predictive control) outputs corresponding control instructions according to the input information, and these instructions are converted into signals that can be understood by the dexterous hand actuator through adaptive linear mapping. After obtaining the corresponding control instructions, the error between the actual results (such as the motion trajectory and force of the dexterous hand, etc.) obtained based on the physical model and the expected results is used as the loss function, and the weight of the model is updated through the back propagation algorithm. According to the convergence and performance of the algorithm, the model is optimized and adjusted, including adjusting the learning rate, increasing the number of iterations, and modifying the network structure (such as increasing or decreasing the number of cerebellar units, changing the connection mode, etc.).

[0105] In some embodiments, in order to improve the training efficiency, the cerebellum neural network to be trained and the trajectory planning model to be trained can be trained separately. Specifically, the training sample set includes multiple groups of image sequences and corresponding expected trajectories, as well as multiple groups of trajectories and corresponding expected torques; the training of the cerebellum neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellum model neural network and the trajectory planning model can include: based on the first loss function and the back propagation algorithm, according to the multiple groups of trajectories and the corresponding expected torques, the cerebellum neural network to be trained is trained to obtain the cerebellum model neural network; the first loss function includes the error between the actual torque output by the cerebellum neural network to be trained and the expected torque; based on the second loss function and the back propagation algorithm, according to the multiple groups of the image sequences and the corresponding expected trajectories, the trajectory planning model to be trained is trained to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

[0106] For example, the simulation environment MuJoCo can be used for training. The model training can use a 12G RTX 3080 GPU. The Smooth L1 loss function (Huber loss function) and AdamW optimizer are used. The expression of the loss function is as follows:

[0107] (5)

[0108] Among them, Loss is the loss function, T pred is the model prediction value, such as the expected torque or expected trajectory, T true is a real value, such as actual torque or actual trajectory.

[0109] During the training process, you can first train the trajectory planning model to be trained for 100 cycles, with 32 sets of image sequences and corresponding trajectories in each batch; then train the cerebellum neural network to be trained, with 128 trajectories and corresponding moments in each batch. In each training iteration, generate state-action-reward sequences from the MuJoCo environment. Use these sequences to update the corresponding model. Calculate the loss based on the corresponding loss function and use the AdamW optimizer to update the model's weights. Check the performance of the model regularly and adjust hyperparameters such as learning rate and weight decay when necessary.

[0110] In some embodiments, in order to improve the training accuracy while ensuring the training efficiency, the cerebellum model may be initially trained, and then the initial cerebellum model neural network obtained by the initial training and the trajectory planning model to be trained may be jointly trained after the initial training. Specifically, the training sample set includes multiple groups of trajectories and corresponding expected torques, as well as multiple groups of image sequences and corresponding expected torques; the training of the cerebellum neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellum model neural network and the trajectory planning model may include: based on the first loss function and the back propagation algorithm, the cerebellum neural network to be trained is trained according to multiple groups of trajectories and corresponding expected torques to obtain the initial cerebellum model neural network; the first loss function includes the error between the actual torque output by the cerebellum neural network to be trained and the expected torque; based on the second loss function and the back propagation algorithm, the trajectory planning model to be trained and the initial cerebellum model neural network are jointly trained according to multiple groups of the image sequences and the corresponding expected torques to obtain the trajectory planning model and the cerebellum model neural network; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

[0111] For example, the simulation environment MuJoCo can be used for training. The model training can use a 12G RTX 3080 GPU. The Smooth L1 loss function (Huber loss function, such as the loss function shown in expression (5)) and the AdamW optimizer are used.

[0112] During the training process, the cerebellum neural network to be trained can be trained first, with 128 trajectories and corresponding moments input per batch. In each training iteration, a state-action-reward sequence is generated from the MuJoCo environment. The loss is calculated based on the corresponding loss function, and the weights of the model are updated using the AdamW optimizer. Check the performance of the model regularly and adjust hyperparameters such as learning rate and weight decay when necessary. Then jointly train the training trajectory planning model to be trained and the cerebellum neural network to be trained for 100 cycles, with 32 sets of image sequences and corresponding moments input per batch. In each training iteration, a state-action-reward sequence is generated from the MuJoCo environment. The loss is calculated based on the corresponding loss function, and the weights of the model are updated using the AdamW optimizer. Check the performance of the model regularly and adjust hyperparameters such as learning rate and weight decay when necessary.

[0113] 305. Combining the cerebellum model neural network with the whole-body dynamics model based on a model predictive control algorithm to obtain a motion controller, and determining the cerebellum model according to the motion controller and the trajectory planning model.

[0114] Specifically, in order to further improve accuracy, the cerebellum model neural network can be combined with model predictive control (MPC) and the whole-body dynamics model to construct a motion controller. The MPC method is used to predict future states and optimize control strategies, while the whole-body dynamics model provides an accurate description of the movement of the dexterous hand. The optimal control strategy is solved through optimization algorithms (such as gradient descent or genetic algorithms) to achieve precise motion control. At the same time, the control strategy is adjusted and optimized online using the rapid learning and adaptive capabilities of the cerebellum model to adapt to changes in different environments and task requirements.

[0115] In some embodiments, there are multiple ways to combine the cerebellar model neural network with the whole-body dynamics model based on the model predictive control algorithm. In one achievable method, the two models can be combined in parallel to improve the calculation accuracy. Specifically, the method of combining the cerebellar model neural network with the whole-body dynamics model based on the model predictive control algorithm to obtain a motion controller can include: connecting the inverse kinematics module and the inverse dynamics module in series and then connecting them in parallel with the cerebellar model neural network to obtain a motion controller; wherein the inverse dynamics module is also used to connect the forward dynamics module and the forward kinematics module in series in sequence, the forward dynamics module is used to feedback output to the inverse dynamics module, and the forward kinematics module is used to feedback output to the inverse kinematics module and the cerebellar model neural network. In this embodiment, the cerebellum model neural network is combined in parallel with the whole-body dynamics model, and the information about the actual motion state output by the forward kinematics module (such as joint angles, the position of the end effector, etc.) is fed back to the cerebellum model neural network, so that the cerebellum model neural network can adjust the internal weights and parameters to correct the motion trajectory, adapt to the dynamic characteristics, etc., so as to more accurately predict and generate control signals, improve control accuracy, enhance robustness, optimize motion planning, and the cerebellum model and inverse dynamics output can be combined in parallel to improve accuracy.

[0116] In another possible implementation, the two models can be connected in series to improve the operation efficiency. Specifically, the method of combining the cerebellar model neural network with the whole-body dynamics model based on the model predictive control algorithm to obtain a motion controller may include: connecting the cerebellar model neural network with the positive dynamics module and the positive kinematics module in series to obtain a motion controller; wherein the positive kinematics module is used to feedback the output to the cerebellar model neural network. In this embodiment, by connecting the cerebellar model neural network with the whole-body dynamics model in series, the information about the actual motion state (such as joint angles, the position of the end effector, etc.) output by the positive kinematics module is fed back to the cerebellar model neural network, so that the cerebellar model neural network can adjust the internal weights and parameters to correct the motion trajectory, adapt to the dynamic characteristics, etc., so as to more accurately predict and generate control signals, improve control accuracy, enhance robustness, optimize motion planning, and adopt a series combination method to simplify the model structure and improve operation efficiency.

[0117] For example, Figure 6 As shown, the model predictive control algorithm based on whole-body dynamics solves the control command through inverse kinematics and dynamics, and adjusts the control command through forward kinematics and dynamics feedback to keep it consistent with the desired trajectory.

[0118] For example, Figure 7 and 8 As shown, by combining the cerebellar model with MPC based on whole-body dynamics, the accuracy of control can be improved.

[0119] like Figure 7 As shown, the cerebellum model neural network can be connected in parallel with the MPC based on whole-body dynamics. Specifically, while the cerebellum model neural network generates control instructions according to the predicted trajectory output by the trajectory planning model, the control instructions can be generated by the inverse kinematics module and the inverse dynamics module, and the two control instructions are fused to obtain the final control instructions. At the same time, the accuracy of the control instructions can be further improved based on the feedback adjustment of positive dynamics and positive kinematics.

[0120] like Figure 8 As shown, the cerebellar model neural network can be connected in series with the MPC based on whole-body dynamics. Specifically, after the cerebellar model neural network generates control instructions according to the predicted trajectory output by the trajectory planning model, the accuracy of the control instructions can be further improved through feedback regulation of positive dynamics and positive kinematics.

[0121] In some embodiments, Fig. 9As shown, in order to simplify the model and improve the computing efficiency, the cerebellum model neural network can also be used as a motion controller, and then a cerebellum model is constructed based on the trajectory planning model and the cerebellum model neural network. The predicted trajectory output by the trajectory planning model is processed by the cerebellum model neural network to obtain the control instruction.

[0122] The training method of the cerebellum model provided in this embodiment obtains a motion controller by combining the cerebellum model neural network with an MPC based on a whole-body dynamics model, and obtains a cerebellum model based on the motion controller and a trajectory planning model, thereby introducing the cerebellum mechanism into the control of dexterous hands. This can improve the flexibility and accuracy of control, enable dexterous hands to perform various complex tasks more accurately, help dexterous hands to quickly adapt to different operating environments and task requirements, and reduce dependence on precise mathematical models, thereby simplifying the design and implementation of control systems.

[0123] Fig.10 This is a schematic diagram of the structure of the dexterous hand control device provided in the embodiment of the present application. Fig.10 As shown, the dexterous hand control device 100 includes: an acquisition module 1001 , a determination module 1002 , a processing module 1003 and a drive control module 1004 .

[0124] An acquisition module 1001 is used to acquire sensor data;

[0125] A determination module 1002 is used to pre-process the sensor data to obtain input information; the input information includes environmental information and state information of the dexterous hand;

[0126] The processing module 1003 is used to input the input information into the cerebellum model to obtain corresponding control instructions; the cerebellum model includes a trajectory planning model and a motion controller; the trajectory planning model is used to output the corresponding trajectory planning result according to the input information, and the motion controller is used to convert the trajectory planning result into the corresponding control instruction; the motion controller is obtained by combining the cerebellum model neural network with the whole body dynamics model based on the model predictive control algorithm;

[0127] The drive control module 1004 is used to drive and control the actuator module of the dexterous hand according to the control instruction.

[0128] The dexterous hand control device provided in the embodiment of the present application can perceive the changes in the external environment and the complexity of the operation task in real time by simulating the efficient learning and adaptive ability of the cerebellum, and dynamically adjust and optimize the control strategy accordingly. This ability significantly improves the adaptability and flexibility of the dexterous hand in a complex and changeable environment, enabling it to flexibly respond to various challenges. After introducing the cerebellum learning framework, the system can more accurately match the operating environment and task requirements. Not only does it improve the operating accuracy of the dexterous hand, but it also significantly enhances its grasping stability to ensure the smooth completion of the operation task. In addition, the rapid response capability of the cerebellum learning mechanism enables the system to quickly process data and make decisions after receiving sensor data, and generate control instructions. This rapid response capability is particularly important for tasks that require urgent adjustment or respond to emergencies, and can significantly improve overall operating efficiency.

[0129] In some embodiments, the environmental information includes at least one of the following parameters of an object in the real environment where the dexterous hand is located: position, shape, and size; the state information includes at least one of the following parameters: position, strength, speed, and acceleration of the fingers of the dexterous hand.

[0130] In some embodiments, the device 100 also includes a training module (not shown): constructing a physical model based on the physical structure of the dexterous hand and the physical structure of the robotic arm; constructing a cerebellum neural network to be trained and a trajectory planning model to be trained; obtaining a training sample set; based on the training sample set and the physical model, training the cerebellum neural network to be trained and the trajectory planning model to be trained to obtain the cerebellum model neural network and the trajectory planning model; combining the cerebellum model neural network with the whole-body dynamics model based on a model predictive control algorithm to obtain a motion controller, and determining the cerebellum model according to the motion controller and the trajectory planning model.

[0131] In some embodiments, the training module is specifically used to: connect the inverse kinematics module and the inverse dynamics module in series and then connect them in parallel with the cerebellum model neural network to obtain a motion controller; wherein the inverse dynamics module is also used to connect the forward dynamics module and the forward kinematics module in series in sequence, the forward dynamics module is used to feedback output to the inverse dynamics module, and the forward kinematics module is used to feedback output to the inverse kinematics module and the cerebellum model neural network.

[0132] In some embodiments, the training module is specifically used to: connect the cerebellum model neural network in series with the positive dynamics module and the positive kinematics module in sequence to obtain a motion controller; wherein the positive kinematics module is used to feedback output to the cerebellum model neural network.

[0133] In some embodiments, the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7.

[0134] In some embodiments, the training module is specifically used to: control the dexterous hand and the robotic arm to perform grasping and manipulation tasks in multiple scenarios through a remote operation system, and collect sensor data during the task execution; the task objects in different scenarios are different or the operating environment is different; different task objects refer to at least one of the following parameters of the task objects being different: shape, material, position; pre-process the sensor data to obtain a training sample set.

[0135] In some embodiments, the training sample set includes multiple groups of image sequences and corresponding expected trajectories, and multiple groups of trajectories and corresponding expected torques; the training module is specifically used to: based on a first loss function and a back-propagation algorithm, train the cerebellum neural network to be trained according to the multiple groups of trajectories and the corresponding expected torques to obtain the cerebellum model neural network; the first loss function includes the error between the actual torque output by the cerebellum neural network to be trained and the expected torque; based on a second loss function and a back-propagation algorithm, train the trajectory planning model to be trained according to the multiple groups of image sequences and the corresponding expected trajectories to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

[0136] In some embodiments, the training module is specifically used for: the training sample set includes multiple groups of trajectories and corresponding expected torques, and multiple groups of image sequences and corresponding expected torques; the training module is specifically used for: based on a first loss function and a back-propagation algorithm, the cerebellum neural network to be trained is trained according to multiple groups of trajectories and corresponding expected torques to obtain an initial cerebellum model neural network; the first loss function includes the error between the actual torque output by the cerebellum neural network to be trained and the expected torque; based on a second loss function and a back-propagation algorithm, the trajectory planning model to be trained and the initial cerebellum model neural network are jointly trained according to multiple groups of image sequences and corresponding expected torques to obtain the trajectory planning model and the cerebellum model neural network; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

[0137] The dexterous hand control device provided in the embodiment of the present application can be used to execute the above-mentioned method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0138] Fig.11 A schematic diagram of the hardware structure of the dexterous hand control device provided in an embodiment of the present application.

[0139] Device 110 may include one or more of the following components: a processing component 1101 , a memory 1102 , a power component 1103 , a multimedia component 1104 , an audio component 1105 , an input / output (I / O) interface 1106 , a sensor component 1107 , and a communication component 1108 .

[0140] The processing component 1101 generally controls the overall operation of the device 110, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1101 may include one or more processors 1109 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1101 may include one or more modules to facilitate the interaction between the processing component 1101 and other components. For example, the processing component 1101 may include a multimedia module to facilitate the interaction between the multimedia component 1104 and the processing component 1101.

[0141] The memory 1102 is configured to store various types of data to support operations on the device 110. Examples of such data include instructions for any application or method operating on the device 110, contact data, phone book data, messages, pictures, videos, etc. The memory 1102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] The power supply component 1103 provides power to the various components of the device 110. The power supply component 1103 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 110.

[0143] The multimedia component 1104 includes a screen that provides an output interface between the device 110 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1104 includes a front camera and / or a rear camera. When the device 110 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0144] The audio component 1105 is configured to output and / or input audio signals. For example, the audio component 1105 includes a microphone (MIC), and when the device 110 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1102 or sent via the communication component 1108. In some embodiments, the audio component 1105 also includes a speaker for outputting audio signals.

[0145] I / O interface 1106 provides an interface between processing component 1101 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0146] The sensor assembly 1107 includes one or more sensors for providing various aspects of status assessment for the device 110. For example, the sensor assembly 1107 can detect the open / closed state of the device 110, the relative positioning of components, such as the display and keypad of the device 110, and the sensor assembly 1107 can also detect the position change of the device 110 or a component of the device 110, the presence or absence of user contact with the device 110, the orientation or acceleration / deceleration of the device 110, and the temperature change of the device 110. The sensor assembly 1107 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1107 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1107 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0147] The communication component 1108 is configured to facilitate wired or wireless communication between the device 110 and other devices. The device 110 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1108 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1108 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0148] In an exemplary embodiment, the device 110 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0149] An embodiment of the present application also provides an intelligent agent, including a dexterous hand, an arm, a sensor, an actuator, and a dexterous hand control device as described in the above embodiment.

[0150] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1102 including instructions, and the instructions can be executed by a processor 1109 of the device 110 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0151] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium may be any available medium that can be accessed by a general or special-purpose computer.

[0152] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0153] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0154] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the dexterous hand control method performed by the dexterous hand control device as described above.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A dexterous hand control method, characterized in that: include: Based on the model predictive control algorithm, the cerebellum model neural network is combined with the whole body dynamics model to obtain a motion controller, and the cerebellum model is determined according to the motion controller and the trajectory planning model; The model-based predictive control algorithm combines the cerebellum model neural network with the whole-body dynamics model to obtain a motion controller, and determines the cerebellum model according to the motion controller and the trajectory planning model, including: connecting an inverse kinematics module and an inverse dynamics module in series and then connecting them in parallel with the cerebellum model neural network to obtain a motion controller; wherein the inverse dynamics module is also used to be connected in series with a forward dynamics module and a forward kinematics module in sequence, the forward dynamics module is used to feed back and output to the inverse dynamics module, and the forward kinematics module is used to feed back and output to the inverse kinematics module and the cerebellum model neural network; Get sensor data; Preprocessing the sensor data to obtain input information; the input information includes environmental information and state information of the dexterous hand; the environmental information includes information of the operating environment and at least one of the following parameters of an object in the operating environment: position, shape, and size; The input information is input into the cerebellum model to obtain corresponding control instructions; the cerebellum model includes the trajectory planning model and the motion controller; the trajectory planning model is used to output the corresponding trajectory planning result according to the input information, and the motion controller is used to convert the trajectory planning result into the corresponding control instruction; The actuator module of the dexterous hand is driven and controlled according to the control instruction.

2. The method according to claim 1, characterized in that The state information includes at least one of the following parameters: the position, strength, speed, and acceleration of the fingers of the dexterous hand; the model-based predictive control algorithm combines the cerebellum model neural network with the whole body dynamics model to obtain a motion controller, and before determining the cerebellum model according to the motion controller and the trajectory planning model, it also includes: Construct a physical model based on the physical structure of the dexterous hand and the physical structure of the robotic arm; Constructing a cerebellum neural network to be trained and a trajectory planning model to be trained; the backbone network of the trajectory planning model to be trained includes a convolutional neural network and an attention network; the convolutional neural network is a residual network; the attention network includes a self-attention network with a first preset number of layers and a cross-attention network with a second preset number of layers; the first preset number of layers is greater than or equal to 4, and the second preset number of layers is greater than or equal to 7; Obtain a training sample set; Based on the training sample set and the physical model, the cerebellum neural network to be trained and the trajectory planning model to be trained are trained to obtain the cerebellum model neural network and the trajectory planning model.

3. The method according to claim 2, characterized in that The step of obtaining a training sample set includes: The dexterous hand and the robotic arm are controlled by the teleoperation system to perform grasping and manipulation tasks in multiple scenarios, and sensor data is collected during the task execution process; the task objects in different scenarios are different or the operation environment is different; different task objects refer to the task objects having at least one different parameter: shape, material, position; The sensor data is preprocessed to obtain a training sample set.

4. The method according to any one of claims 2 to 3, characterized in that: The training sample set includes multiple groups of image sequences and corresponding expected trajectories, and multiple groups of trajectories and corresponding expected torques; the training of the cerebellum neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellum model neural network and the trajectory planning model includes: Based on a first loss function and a back propagation algorithm, the cerebellum neural network to be trained is trained according to multiple groups of trajectories and corresponding expected torques to obtain the cerebellum model neural network; the first loss function includes the error between the actual torque output by the cerebellum neural network to be trained and the expected torque; Based on the second loss function and the back propagation algorithm, the trajectory planning model to be trained is trained according to multiple groups of the image sequences and the corresponding expected trajectories to obtain the trajectory planning model; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

5. The method according to any one of claims 2-3, characterized in that: The training sample set includes multiple groups of trajectories and corresponding expected torques, and multiple groups of image sequences and corresponding expected torques; the training of the cerebellum neural network to be trained and the trajectory planning model to be trained based on the training sample set and the physical model to obtain the cerebellum model neural network and the trajectory planning model includes: Based on a first loss function and a back propagation algorithm, the cerebellum neural network to be trained is trained according to multiple groups of trajectories and corresponding expected torques to obtain an initial cerebellum model neural network; the first loss function includes an error between an actual torque output by the cerebellum neural network to be trained and the expected torque; Based on the second loss function and the back-propagation algorithm, the trajectory planning model to be trained and the initial cerebellum model neural network are jointly trained according to multiple groups of the image sequences and the corresponding expected torques to obtain the trajectory planning model and the cerebellum model neural network; the second loss function includes the error between the actual trajectory output by the trajectory planning model to be trained and the expected trajectory.

6. A dexterous hand control device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the dexterous hand control method according to any one of claims 1 to 5.

7. An intelligent agent, characterized in that: It comprises a dexterous hand, an arm, a sensor, an actuator and the dexterous hand control device as claimed in claim 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the dexterous hand control method according to any one of claims 1 to 5 is implemented.

Images