Multi-degree-of-freedom bionic manipulator control method

Abstract

The application relates to the technical field of robots, and provides a multi-degree-of-freedom bionic manipulator control method. The method first establishes a kinematics model and a dynamics relationship of the bionic manipulator, and on this basis, a multi-modal training and real-time control method for a 20-degree-of-freedom bionic manipulator is provided. The method comprehensively utilizes three heterogeneous sensing modalities of inertial measurement, computer vision and surface electromyography to synchronously perceive the hand movement intention of a human operator, and realizes fusion regression of multi-source information based on a deep neural network to output target control instructions of joints of the manipulator. In the training stage, an inertial measurement system is used to provide high-precision joint angle true values; in the deployment stage, vision and surface electromyography are mainly used as network inputs, and real-time closed-loop control is realized in combination with feedback at the manipulator end, so that high-precision control of the multi-degree-of-freedom bionic manipulator is realized.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a control method for a multi-degree-of-freedom bionic manipulator. Background Technology

[0002] The human hand is a highly dexterous organ with complex movements; the lengths of the five fingers' joints and their bending angles are all different. Through the coordinated action of these five fingers, lightweight objects can be grasped. Existing robotic hands are mostly used in industrial automation, but they lag significantly behind human hands in terms of dexterity, compliance, and multi-degree-of-freedom control, making it difficult to meet the demands of complex operations and human-machine collaboration.

[0003] Therefore, developing a flexible and precisely controlled multi-degree-of-freedom bionic manipulator is of great significance. Summary of the Invention

[0004] In view of this, embodiments of this application provide a control method for a multi-degree-of-freedom bionic manipulator to solve the technical problem of how to achieve precise control of a multi-degree-of-freedom bionic manipulator.

[0005] A first aspect of this application provides a method for controlling a multi-degree-of-freedom bionic manipulator, comprising:

[0006] Establish the kinematic equations of a multi-degree-of-freedom bionic manipulator, and build a pre-trained pattern recognition model based on the kinematic equations;

[0007] Acquire historical control data; historical control data includes at least historical control targets and historical sensor detection data corresponding to each historical control target.

[0008] The pre-trained pattern recognition model is trained using historical control data to obtain the trained pattern recognition model.

[0009] The real-time control target is obtained, and the control command is determined based on the trained pattern recognition model. The real-time control target may include user control commands or real-time sensor detection data.

[0010] A multi-degree-of-freedom bionic robotic arm is controlled based on control commands.

[0011] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment first establishes the kinematic model and dynamic relationship of the bionic manipulator, and on this basis, proposes a multimodal training and real-time control method for a 20-DOF bionic manipulator. This method comprehensively utilizes three heterogeneous sensing modalities—inertial measurement, computer vision, and surface electromyography—to synchronously perceive the hand movement intentions of the human operator, and uses a deep neural network to achieve the fusion and regression of multi-source information, outputting target control commands for each joint of the manipulator. During the training phase, an inertial measurement system provides high-precision true values ​​of joint angles; during the deployment phase, vision and surface electromyography are mainly used as network inputs, combined with feedback from the manipulator end to achieve real-time closed-loop control, thereby realizing high-precision control of the multi-DOF bionic manipulator. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a multi-degree-of-freedom bionic manipulator control method provided in an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of the structure of the robotic arm controlled in the embodiments of this application.

[0015] Figure 3 This is a schematic diagram of the coordinate system of the finger joint linkage provided in the embodiments of this application.

[0016] Figure 4 This is a schematic diagram of the dual-input branch deep regression network structure provided in the embodiments of this application.

[0017] Figure 5 This is a flowchart illustrating another multi-degree-of-freedom bionic manipulator control method provided in this application embodiment. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0019] The following describes in detail, with reference to the accompanying drawings, a multi-degree-of-freedom bionic manipulator control method according to an embodiment of this application.

[0020] Figure 1 This is a flowchart illustrating a multi-degree-of-freedom bionic robotic arm control method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0021] In step S101, the kinematic equations of the multi-degree-of-freedom bionic manipulator are established, and a pre-trained pattern recognition model is constructed based on the kinematic equations.

[0022] In step S102, historical control data is acquired.

[0023] The historical control data includes at least the historical control targets and the historical sensor detection data corresponding to each historical control target.

[0024] In step S103, a pre-trained pattern recognition model is trained using historical control data to obtain a trained pattern recognition model.

[0025] In step S104, the real-time control target is obtained, and the control command is determined based on the real-time control target using the trained pattern recognition model.

[0026] The real-time control targets include user control commands or real-time sensor detection data.

[0027] In step S105, the multi-degree-of-freedom bionic manipulator is controlled based on control commands.

[0028] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain processing capabilities.

[0029] In some embodiments of this application, the kinematic equations of a multi-degree-of-freedom bionic manipulator can be established first, and a pre-trained pattern recognition model can be constructed based on the kinematic equations. Then, historical control data can be acquired. Next, the historical control data can be used to train the pre-trained pattern recognition model to obtain the trained pattern recognition model.

[0030] Furthermore, real-time control targets can be acquired, and a trained pattern recognition model can be used to determine control commands based on these targets. Finally, the multi-degree-of-freedom bionic manipulator can be controlled based on these commands.

[0031] According to the technical solution provided in the embodiments of this application, a kinematic model and dynamic relationship of the bionic manipulator are first established. Based on this, a multimodal training and real-time control method for a 20-DOF bionic manipulator is proposed. This method comprehensively utilizes three heterogeneous sensing modalities—inertial measurement, computer vision, and surface electromyography—to synchronously perceive the hand movement intentions of the human operator. It then uses a deep neural network to achieve the fusion and regression of multi-source information, outputting target control commands for each joint of the manipulator. During the training phase, an inertial measurement system provides high-precision true values ​​of joint angles. During the deployment phase, vision and surface electromyography are mainly used as network inputs, combined with feedback from the manipulator end to achieve real-time closed-loop control, thereby realizing high-precision control of the multi-DOF bionic manipulator.

[0032] Figure 2 This is a schematic diagram of the structure of the robotic arm controlled in the embodiments of this application. Figure 2 As shown, the multi-degree-of-freedom bionic manipulator used in this application embodiment includes at least a manipulator skeleton, which includes at least a base, multiple bionic fingers, and an end effector; wherein each bionic finger includes at least one end effector. A rotating joint and One link; among which It is a positive integer; the end effector is located at the fingertip of the bionic finger.

[0033] In some implementations, establishing the kinematic equations of a multi-degree-of-freedom bionic manipulator may include:

[0034] The structure of the finger is equivalent to a rotating joint-linkage model;

[0035] The DH matrix parameter method is used to perform matrix transformation on adjacent rotary joints to obtain the coordinate transformation matrix of each rotary joint. The DH matrix, short for Denavit-Hartenberg Matrix, is a robot kinematics modeling method that describes the spatial relationship between adjacent links by establishing coordinate systems in each link and deriving the pose of the end effector, which is equivalent to the base coordinate system.

[0036] A local coordinate system is established for each link, and the relationship between the local coordinate systems is described by the coordinate transformation matrix. The mathematical mapping between the end effector of each bionic finger, each rotation joint and the base is determined.

[0037] The kinematic equations of the bionic robotic hand can be determined based on at least mathematical mapping.

[0038] In some embodiments of this application, determining the mathematical mapping between the end effector of each bionic finger, each rotary joint, and the base may include:

[0039] The total transformation matrix for determining the mathematical mapping between any point on the finger and the base is: ;wherein, any point on the finger includes the end effector of the finger and any point where the rotation joint is located; Let be the homogeneous transformation matrix between any rotating joint and the base. Indicates the base; Let be the homogeneous transformation matrix between the first rotational joint of the finger and the base. For the first finger Rotate joints and fingers Homogeneous transformation matrix between rotational joints for abbreviation, , The total number of joints that rotate between the base and the fingers;

[0040] The three-dimensional components of the (j+1)th rotational joint of the finger in the finger joint link coordinate system are determined as follows: , , Each rotating joint is fixed to the z-axis and rotates in the right-hand helical direction;

[0041] Determine the first The homogeneous transformation matrix is ;in, Represents the cosine function. Represents the sine function. This represents the rotation angle about the z-axis. Indicates the joint offset. Indicates the length of the link, angle It represents the angle between two adjacent z-axis;

[0042] In response to determining n=4, the link parameters of the multi-degree-of-freedom bionic manipulator are substituted into the first... The transformation matrix yields the homogeneous transformation matrix of the fingertip coordinate system relative to the base coordinate system. ;in, for , for , for , for , , , , , and All are link lengths.

[0043] Figure 3 This is a schematic diagram of the coordinate system of the finger joint linkage provided in an embodiment of this application. (Reference) Figure 3 The above homogeneous transformation matrix can be constructed.

[0044] In some embodiments of this application, determining the mathematical mapping between the end effector of each bionic finger, each rotary joint, and the base may further include:

[0045] Determine the articulation and fingertip poses of the multi-degree-of-freedom bionic robotic hand. ;in, For the joint and fingertip poses of a multi-degree-of-freedom bionic robotic hand, , and These are the unit direction vectors of the x-axis of the terminal coordinate system in the base coordinate system. , and , respectively, are the unit direction vectors of the y-axis of the terminal coordinate system in the base coordinate system. , and These are the unit direction vectors of the z-axis of the end coordinate system in the base coordinate system. , and These are the coordinates of the end position, Represents the rotation matrix;

[0046] The poses of each end effector are obtained, and the angles of each rotational joint are determined based on the poses of the corresponding bionic end effectors and the inverse homogeneous transformation matrix:

[0047] ;

[0048] ;

[0049] ; ; ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] in, The angle of the first rotational joint, The angle of the second rotation joint. The angle of the third rotation joint. The angle of the fourth rotation joint. The projection length of the end point in the plane of rotation of the base. , To transform the spatial problem into the inverse solution process of a planar two-linkage problem, the coordinates of the wrist point are required; This is an intermediate value calculated for the geometric relationship of a planar two-link system. The wrist point, in robotics, is a key geometric point used to decompose a 6-DOF spatial problem into a planar localization problem and a spatial orientation problem. Analogous to a finger as a robotic arm, the wrist point can be determined using the above method.

[0055] In other words, the angles of each joint can be calculated in reverse from the pose of any fingertip. Furthermore, the angles of the finger joints can be calculated based on the real-time changes in the fingertip pose.

[0056] In some embodiments of this application, determining the forward kinematic equations of the bionic manipulator may include:

[0057] Based on Lagrange function modeling, the torque required to drive each rotational joint in the bionic finger is obtained;

[0058] Based on mathematical mapping and torque, a high-order polynomial fitting method is used to plan the trajectory of the desired pose of each joint.

[0059] The kinematic equations of the bionic robotic hand were determined based on the planning results.

[0060] The torque required to drive each rotating joint in the bionic finger is:

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] in, , , and These represent the torques required for the first to fourth rotating joints, respectively. , , and These are the gravity terms for the first to fourth rotating joints, respectively. , , and These are the centrifugal force terms for the first to fourth rotating joints, respectively. , , and These are the Coriolis force terms for the first to fourth rotational joints, respectively. Indicates the number of rotational joints. Indicates the first The velocity vector of each rotating joint Indicates the first The velocity vector of each rotating joint Indicates the first The acceleration vector of each rotating joint.

[0066] In some implementations, using a high-order polynomial fitting method to plan the trajectory of the desired pose of each joint may include:

[0067] Establish the polynomial general formula for the change of joint angle over time:

[0068] ;in, Joint angle over time The function, For undetermined coefficients, It is a time matrix;

[0069] The planned trajectory of a single joint is divided into three time intervals, and piecewise fitting is performed using fourth-order polynomials, cubic polynomials, and fourth-order polynomials respectively to obtain piecewise continuous functions. ;in, Joint angle over time Piecewise continuous functions , and These are the joint angle trajectory functions for the starting segment, transition segment, and ending segment, respectively. to , to , to These are the undetermined coefficients of the corresponding piecewise polynomial; to Plan time nodes for the trajectory;

[0070] Taking the first derivative of the piecewise continuous function, we get ;in, The joint angular velocity with respect to time The function;

[0071] Taking the second derivative of the piecewise continuous function, we get ;in, The joint angular acceleration with respect to time The function;

[0072] Based on the preset starting pose, intermediate transition pose, and ending pose, and combined with the velocity and acceleration constraints at each time point, the undetermined coefficients are solved. to , to , to This ensures that the joint angle, angular velocity, and angular acceleration remain continuous at the segment connection points, thereby obtaining a smooth joint trajectory.

[0073] In a preferred embodiment, the trajectory boundary conditions can be set as follows: at time... , , , At each joint angle, the preset key postures are passed. , , , At the segment connection point and At the initial moment, the position, velocity, and acceleration are continuous; and termination time Furthermore, the angular velocity and / or angular acceleration can be set to zero to reduce start-up shock and end-of-cycle vibration.

[0074] Accordingly, the constraint relationship can be written as follows: ; ;

[0075] Additional boundary conditions can be written as ; ; and .

[0076] By using the above-mentioned piecewise high-order polynomial trajectory planning method, we can ensure that the finger joint movement process is smooth and continuous, and meet the requirements of bionic grasping action for speed and acceleration smoothness.

[0077] In some embodiments of this application, determining the kinematic equations of the bionic robotic hand based on the planning results may include:

[0078] Obtain the time node parameters, and construct a piecewise polynomial coefficients based on the time node parameters to solve the equation. ;in, For boundary condition vectors, The coefficient matrix is ​​composed of the time parameters of each segment. Let be the coefficient vector of the polynomial to be determined;

[0079] The first The local time variable for each motion segment is represented as follows: Let the first Local initial time of each motion segment It is 0, and the local termination time for each motion segment is given. Then the boundary condition vector, coefficient matrix, and coefficient vector to be determined are:

[0080] ;

[0081] in, , and These represent the durations of the first, second, and third segments of the trajectory, respectively. , , and These are the target angles of the joints at critical moments; , , and These are the target angular velocities at the start and end times, respectively. , , and These are the target angular accelerations at the start and end times, respectively.

[0082] Define the generalized coordinates of the simulated robotic arm as follows: The dynamic equations are obtained. ;in, The inertia matrix, For Coriolis force and centrifugal force terms, For gravity, Let be the driving torque vector of each rotary joint. Let Jacobian matrix be the contact point. Let be the contact force vector, representing the combination of normal and tangential forces at all contact points; , for The real space represents the equivalent generalized force after the contact force is mapped to the joint space; the contact force vector... As an optimization variable or as an input of external forces measured by simulation / sensors.

[0083] In one example, in simulation environments such as Webots, MuJoCo, and Gazebo, the physics engine can directly output contact reactions based on the contact collision model, friction model, and constraint solution results, and assemble them into... In real-world environments, contact forces are directly measured by placing torque sensors, flexible pressure arrays, or tactile sensors at the tips or joints of mechanical fingers, and then uniformly represented in the contact coordinate system or base coordinate system through coordinate transformation.

[0084] In some embodiments of this application, constructing a hierarchical quadratic programming optimization model for multi-finger cooperative control based on kinematic equations and dynamic equations may include:

[0085] For the The root of the bionic finger is defined as the Cartesian space acceleration error at its distal end. ;in, For the first Jacobian matrix of linear velocity at the tip of a biomimetic finger; This is the derivative of the linear velocity Jacobian matrix with respect to time; For the first The expected Cartesian acceleration at the tip of a bionic finger;

[0086] Within each control cycle, construct a quadratic programming optimization problem: ;in, The number of bionic fingers participating in collaborative control, This is the joint driving torque vector calculated from the dynamic equations. For the first The task weight matrix of the bionic finger. Energy regularization coefficient; Energy regularization coefficient It is a constant greater than 0, used to balance trajectory tracking accuracy and joint drive torque consumption;

[0087] Preferably, The settings are based on the actuator's rated torque, trajectory tracking error tolerance, and contact stability requirements, and can be obtained through simulation calibration or experimental tuning. Preferably, Take it as a diagonal nonnegative matrix or a diagonal positive definite matrix, when the... When the bionic finger is the primary task finger or is in a contact state, its corresponding weight matrix is ​​increased. ;

[0088] Construct hierarchical constraints; hierarchical constraints should include at least safety constraints, contact stability constraints, primary task finger trajectory constraints, and auxiliary or secondary task constraints in descending order of priority.

[0089] Solving for the optimal joint acceleration based on a quadratic programming objective function and hierarchical constraints. With contact force This generates joint control input during the multi-finger collaborative grasping process.

[0090] In some implementations, security constraints may include:

[0091] Dynamic constraints .

[0092] Actuator torque constraint conditions ;in, and These are the minimum and maximum driving torque vectors that each actuator is allowed to output, respectively.

[0093] Collision constraints ,in and It can be any two different bionic fingers, any two different links, a bionic finger and the palm, a bionic finger and a target object, or a bionic finger and an external environmental obstacle. for and The minimum spacing function; A preset safety distance threshold is set. Minimum spacing function. Used to characterize the current generalized coordinates Below, object With object The minimum geometric distance between them.

[0094] And, the current step linearization constraints ;in To obtain the gradient in generalized coordinates, To control the cycle.

[0095] In some embodiments, the spatial pose of each link can first be calculated using forward kinematics based on the current generalized coordinates; then, each link can be simplified into a geometric model such as a line segment, cylinder, sphere, capsule, or bounding box; the minimum distance between two geometric models can be approximated based on discrete sampling points, and this minimum distance can be defined as the minimum spacing function. .

[0096] In a preferred embodiment, the connecting rod and connecting rod Each is approximated as a centerline segment and and the corresponding equivalent radius and Then the minimum spacing function can be expressed as:

[0097] ;in, and Determined by the current joint configuration through forward kinematics. and These represent the equivalent envelope radii of the corresponding links.

[0098] In other embodiments, objects can also be obtained directly based on the collision detection module or distance query interface provided by the physics simulation engine MuJoCo. With object The minimum distance between them, and use it as the minimum spacing function. .

[0099] The contact state constraint is that, in response to determining that the target bionic finger is not in contact with an object, the contact force of the target bionic finger is zero; the target bionic finger is any bionic finger.

[0100] The above steps established the kinematic model and dynamic relationships of the bionic manipulator. Based on this, this application proposes a multimodal training and real-time control method for a 20-DOF bionic manipulator. This method integrates three heterogeneous sensing modalities—inertial measurement, computer vision, and surface electromyography—to synchronously perceive the operator's hand movement intentions. It then uses a deep neural network to fuse and regress multi-source information, outputting target control commands for each joint of the manipulator. During the training phase, the inertial measurement system provides high-precision true values ​​of joint angles; during the deployment phase, vision and surface electromyography are primarily used as network inputs, combined with feedback from the manipulator end to achieve real-time closed-loop control.

[0101] The pre-trained pattern recognition model used in the embodiments of this application may include a dual-input branch deep regression network. Figure 4 This is a schematic diagram of the dual-input branch deep regression network structure provided in an embodiment of this application. Figure 4 As shown, the dual-input branch deep regression network includes a visual input branch, a surface electromyography input branch, a fusion and regression layer, a training phase supervision and optimization module, and a deployment phase feedback and explanation module.

[0102] This dual-input branch deep regression network uses visual keypoint features and surface electromyography features as two parallel inputs. It completes high-dimensional representation learning through its independent feature extraction branches, and then outputs 20-dimensional robot joint angle prediction results through feature concatenation and multi-layer fully connected regression layers. Figure 4 The paper also illustrates the supervision and optimization path during the training phase, as well as the role of inertial feedback at the robotic end during the deployment phase.

[0103] The visual input branch includes visual keypoint input P. vis The system consists of a visual keypoint input unit, a depthwise separable convolutional module, and a first fully connected mapping layer. The visual keypoint input unit is used to input visual keypoint features; the depthwise separable convolutional module is used to extract hand spatial geometric features; and the first fully connected mapping layer is used to further map the convolutional features into visual embedding features.

[0104] Surface electromyography input branch includes electromyographic feature input F emg The system consists of a unit, a depthwise separable convolutional module, and a second fully connected mapping layer. The electromyography (EMG) feature input unit is used to input EMG feature vectors. The depthwise separable convolutional module is used to extract feature representations reflecting muscle activation states. The second fully connected mapping layer is used to further compress EMG features and map them into EMG embedding features.

[0105] The fusion and regression layers include feature splicing and fusion [Z] vis ;Z emgThe system comprises a feature embedding unit, a multilayer fully connected regression unit, a joint angle prediction output unit, and an output description unit; a feature splicing and fusion unit is used to splice visual embedding features and electromyographic embedding features to form a joint fused feature; a multilayer fully connected regression unit is used to perform nonlinear mapping and regression on the fused joint features; a joint angle prediction output unit is used to output the predicted joint angle; and an output description unit is used to indicate that the output predicted joint angle corresponds to the joint representation of the active joint angle and lateral swing angle of each of the five fingers.

[0106] The training phase supervision and optimization module includes an Inertial Measurement Unit (IMU) supervision label input unit, a loss function calculation unit, and a parameter optimization unit; IMU supervision labels. The input unit provides the true values ​​of joint angles; the loss function calculation unit constructs the mean squared error loss based on the difference between the predicted and true joint angles, as well as the joint range of motion constraint term; the parameter optimization unit iteratively updates the network parameters using the Adam optimizer and selects the optimal model parameters based on the validation set results. The pose data returned by the IMU can serve as baseline data for finger poses during the training phase for supervised training; during the deployment phase, it can be used as closed-loop feedback to determine whether the action execution is accurate.

[0107] A preferred loss function can be expressed as: ;in, For loss function, Where n is the batch sample number, and n is the sample number. The predicted joint angle output for the nth sample. For the IMU supervision label of the nth sample, It can be denoted as MSE; For range constraint weighting coefficients, This is a penalty term for the joint angle range of motion. The network is preferably trained using the Adam optimizer, and the optimal model parameters are selected through a validation set.

[0108] The deployment phase feedback explanation module includes a robot end feedback correction explanation unit and an input boundary explanation unit. The robot end feedback correction explanation unit is used to explain that the execution effect can be corrected by using the robot end MU closed-loop feedback during the deployment phase. The input boundary explanation unit is used to explain that the inertial feedback information during the deployment phase is not necessarily used as the forward input branch of the neural network.

[0109] In other words, the dual-input branch deep regression network provided in this application uses visual keypoint features and electromyographic (EMG) feature vectors as two inputs. The visual keypoint features are processed by a depthwise separable convolutional module and a fully connected mapping layer in the visual input branch to extract spatial geometric features, forming visual embedding features. The EMG feature vectors are processed by a depthwise separable convolutional module and a fully connected mapping layer in the surface EMG input branch to extract muscle activation features, forming EMG embedding features. Subsequently, the output features of the two branches are concatenated and fused, and then processed through multiple fully connected regression layers to output predicted joint angles. , It is a 20-dimensional real number space.

[0110] In a preferred embodiment, the 20-dimensional prediction output (i.e., the predicted key angles) corresponds to the active joint angles and lateral swing angles of each of the five fingers. During the network training phase, the ground truth joint angles obtained from IMU calculations are used as supervision labels, and the network is optimized using a mean squared error loss function combined with joint range of motion constraints. Preferably, the Adam optimizer is used for parameter updates, and the optimal model parameters are selected through a validation set.

[0111] Furthermore, during the deployment phase, the network still uses visual and electromyographic features as forward inputs; the inertial information from the robotic arm is mainly used for closed-loop feedback correction to help improve the control accuracy and system stability of the robotic arm, but it is not necessarily used as the forward input mode of the dual-input branch deep regression network. Thus, this invention achieves deep fusion of visual and electromyographic information, while balancing training supervision accuracy and deployment control robustness.

[0112] Figure 5 This is a flowchart illustrating another multi-degree-of-freedom bionic manipulator control method provided in an embodiment of this application. Figure 5 As shown, the overall control flow can be divided into two main parts according to the system runtime sequence: the training phase and the real-time closed-loop control phase. The training phase mainly completes multimodal data acquisition, feature extraction, label generation, dataset construction, network training, and performance evaluation; the real-time closed-loop control phase mainly completes model parameter loading, real-time multimodal input construction, joint angle prediction output, servo drive, and closed-loop feedback correction.

[0113] The training phase module includes the following sub-modules: a multimodal data acquisition sub-module, used to simultaneously acquire data from a monocular RGB camera, surface electromyography (EMG) electrode array signals, and hand inertial sensor array data; and a feature extraction and label generation sub-module, used to extract key hand features from visual images. Preprocessing of surface electromyography signals and extraction of electromyographic features Simultaneously, the true values ​​of joint angles are obtained based on inertial attitude calculation. The multimodal time synchronization and resampling submodule is used to align and resample visual features, electromyographic features, and inertial ground truth labels under a unified time reference. The total number of training samples is [number missing]. When, the dataset can be represented as ,in, Indicates the first indivual , Indicates the first indivual , Indicates the first indivual The labeled dataset construction submodule is used to form a dataset... For sample input, with The training dataset is for supervised labeling; the dual-branch deep regression network training submodule is used to learn network parameters using the Adam optimizer under the constraints of mean squared error loss and joint range of motion to obtain the optimal model parameters; the performance evaluation submodule is used to evaluate the angle error, analyze the prediction consistency, and test the generalization performance of the trained model.

[0114] The performance evaluation submodule further includes: an angle error evaluation unit, used to evaluate the deviation between the predicted joint angle and the true joint angle using metrics such as Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE); a prediction consistency analysis unit, used to evaluate the degree of matching between the model output and the true label using correlation coefficient, goodness of fit, or trend consistency metrics; and a generalization performance testing unit, used to test the robustness and transferability of the model under cross-object, cross-time period, or cross-action conditions.

[0115] The output of the training phase consists of optimal model parameters, which are passed to the real-time closed-loop control phase via the parameter loading path extending from left to right in the diagram, for subsequent online forward inference. The real-time closed-loop control phase module includes the following sub-modules: a real-time input acquisition sub-module, used to simultaneously acquire images from a monocular red-green-blue (RGB) camera in front of the operator and electromyographic signals from the forearm muscle surface during the deployment phase; and a real-time feature extraction sub-module, used to extract visual features in real time. and electromyographic characteristics The multimodal input construction submodule combines real-time visual features and electromyographic features to form the network input vector; the trained network loading and forward inference submodule calls the optimal model parameters obtained during the training phase to perform forward inference on the multimodal input vector; and the prediction output submodule outputs the predicted joint angles. The servo control submodule generates control commands based on predicted joint angles and drives the 20-DOF bionic manipulator through proportional-derivative-position control. The execution and feedback submodule acquires actual joint angle information from inertial sensors located at each finger joint after the manipulator performs a grasping or manipulating action, and generates feedback on the actual joint angles. The closed-loop correction submodule is used to return the actual joint angle feedback to the servo controller to achieve real-time correction during the control process.

[0116] The method provided in this application includes a training phase and a real-time closed-loop control phase. During the training phase, a monocular RGB camera, a surface electromyography (SEMG) electrode array, and a hand inertial sensor array acquire visual data, electromyographic signals, and inertial posture data, respectively. After feature extraction and posture calculation, visual features are obtained. Electromyographic characteristics True value of joint angle Subsequently, time synchronization and resampling were performed on the data from each modality to construct a system based on... For input, with We use a labeled dataset and train a bi-branch deep regression network based on it to obtain the optimal model parameters.

[0117] In a preferred embodiment, the training phase also includes a performance evaluation module for performing angle error evaluation, prediction consistency analysis, and generalization performance testing. This performance evaluation module allows for a comprehensive evaluation of the trained network's performance across multiple dimensions, including accuracy, consistency, and robustness, providing a basis for model selection and subsequent deployment.

[0118] Furthermore, the optimal model parameters obtained during the training phase are input to the forward inference module of the real-time closed-loop control phase via the parameter loading path. In the real-time closed-loop control phase, the system acquires visual and electromyographic features in real time, constructs a multimodal input vector, and obtains the predicted joint angles through forward inference by the trained network. The data is then sent to the servo controller to drive the 20-DOF bionic manipulator to perform grasping or manipulating actions. Inertial sensors at the manipulator's end further collect actual joint angle information and feed it back to the controller, forming a closed-loop control circuit, thereby improving control accuracy and dynamic stability.

[0119] The technical solution provided in this application embodiment can achieve precise, robust, and low-latency real-time control of a 20-DOF bionic manipulator, and is suitable for application scenarios such as intelligent prostheses, remote teleoperation, and dexterous operation of humanoid robots.

[0120] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0121] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0122] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A control method for a multi-degree-of-freedom bionic robotic arm, characterized in that, include: The kinematic equations of the multi-degree-of-freedom bionic manipulator are established, and a pre-trained pattern recognition model is constructed based on the kinematic equations. Acquire historical control data; The historical control data includes at least historical control targets and historical sensor detection data corresponding to each historical control target; The pre-trained pattern recognition model is trained using the historical control data to obtain the trained pattern recognition model. A real-time control target is obtained, and a control command is determined based on the trained pattern recognition model using the real-time control target; wherein, the real-time control target includes user control commands or real-time sensor detection data; The multi-degree-of-freedom bionic manipulator is controlled based on the control commands.

2. The multi-degree-of-freedom bionic manipulator control method according to claim 1, characterized in that, The multi-degree-of-freedom bionic manipulator includes at least a manipulator skeleton, which includes at least a base, multiple bionic fingers, and an end effector; wherein each bionic finger includes at least one end effector. One rotating joint and One link; The value is a positive integer; the end effector is located at the fingertip of the bionic finger; The kinematic equations of the multi-degree-of-freedom bionic manipulator are established, including: The structure of the finger is equivalent to a rotating joint-linkage model; The DH matrix parameter method is used to perform matrix transformation on adjacent rotational joints to obtain the coordinate transformation matrix of each rotational joint; A local coordinate system is established for each link, and the relationship between the local coordinate systems is described using the coordinate transformation matrix. The mathematical mapping between the end effector of each bionic finger, each rotation joint, and the base is determined. The kinematic equations of the bionic manipulator are determined at least based on the mathematical mapping.

3. The multi-degree-of-freedom bionic manipulator control method according to claim 2, characterized in that, Determine the mathematical mapping between the end effector of each bionic finger, each rotational joint, and the base, including: The total mathematical transformation between any point on the finger and the base is then determined as follows: ;wherein, any point on the finger includes the end effector of the finger and any point where the rotation joint is located; Let be the homogeneous transformation matrix between any rotating joint and the base. Indicates the base; Let be the second transformation matrix between the first rotational joint of the finger and the base. For the first finger Rotate joints and fingers Homogeneous transformation matrix between rotational joints for abbreviation, , The total number of joints that rotate between the base and the fingers; The three-dimensional components of the (j+1)th rotational joint of the finger in the finger joint link coordinate system are determined as follows: , , Each rotating joint is fixed to the z-axis and rotates in the right-hand helical direction; Determine the first The homogeneous transformation matrix is ;in, Represents the cosine function. Represents the sine function. This represents the rotation angle about the z-axis. Indicates the joint offset. Indicates the length of the link, angle It represents the angle between two adjacent z-axis; In response to determining n=4, the link parameters of the multi-degree-of-freedom bionic manipulator are substituted into the first... The transformation matrix yields the homogeneous transformation matrix of the fingertip coordinate system relative to the base coordinate system. ;in, for , for , for , for , , , , , and All are link lengths.

4. The multi-degree-of-freedom bionic manipulator control method according to claim 3, characterized in that, Determining the mathematical mapping between the end effector of each bionic finger, each rotational joint, and the base also includes: The articulated fingertips of the multi-degree-of-freedom bionic manipulator are positioned as follows: ;in, The joint fingertips of the multi-degree-of-freedom bionic manipulator are in pose. , and These are the unit direction vectors of the x-axis of the terminal coordinate system in the base coordinate system. , and These are the unit direction vectors of the y-axis of the terminal coordinate system in the base coordinate system. , and These are the unit direction vectors of the z-axis of the end coordinate system in the base coordinate system. , and These are the coordinates of the end position, The rotation matrix is ​​represented by the inverse transformation matrix. By multiplying both sides of the above equation by the inverse transformation matrix, the specified joint variables can be separated, thereby obtaining the corresponding angles of each joint variable. The poses of each end effector are obtained, and the angles of each rotational joint are determined based on the poses of the corresponding bionic end effectors and the inverse homogeneous transformation matrix: ； ； ； ； ； ； ； ； ； in, The angle of the first rotational joint, The angle of the second rotation joint. The angle of the third rotation joint. The angle of the fourth rotation joint. The projection length of the end point in the plane of rotation of the base. , To transform the spatial problem into the inverse solution process of a planar two-linkage problem, the coordinates of the wrist point are required; Intermediate values ​​are used to calculate the geometric relationships of two planar connecting rods.

5. The multi-degree-of-freedom bionic manipulator control method according to claim 4, characterized in that, Determine the forward kinematic equations of the bionic robotic hand, including: Based on Lagrange function modeling, the torque required to drive each rotational joint in the bionic finger is obtained; Based on the mathematical mapping and the torque, a high-order polynomial fitting method is used to plan the trajectory of the desired pose of each joint. The kinematic equations of the bionic robotic hand were determined based on the planning results. The torque required to drive each rotating joint in the bionic finger is: ； ； ； ； in, , , and These represent the torques required for the first to fourth rotating joints, respectively. , , and These are the gravity terms for the first to fourth rotating joints, respectively. , , and These are the centrifugal force terms for the first to fourth rotating joints, respectively. , , and These are the Coriolis force terms for the first to fourth rotational joints, respectively. Indicates the number of rotational joints. Indicates the first The velocity vector of each rotating joint Indicates the first The velocity vector of each rotating joint Indicates the first The acceleration vector of each rotating joint.

6. The multi-degree-of-freedom bionic manipulator control method according to claim 5, characterized in that, Trajectory planning for the desired pose of each joint is performed using a high-order polynomial fitting method, including: Establish the polynomial general formula for the change of joint angle over time: ;in, Joint angle over time The function, For undetermined coefficients, It is a time matrix; The planned trajectory of a single joint is divided into three time intervals, and piecewise fitting is performed using fourth-order polynomials, cubic polynomials, and fourth-order polynomials respectively to obtain piecewise continuous functions. ;in, Joint angle over time Piecewise continuous functions , and These are the joint angle trajectory functions for the starting segment, transition segment, and ending segment, respectively. to , to , to These are the undetermined coefficients of the corresponding piecewise polynomial; to Plan time nodes for the trajectory; Taking the first derivative of the piecewise continuous function, we get ;in, The joint angular velocity with respect to time The function; Taking the second derivative of the piecewise continuous function, we get ;in, The joint angular acceleration with respect to time The function; Based on the preset starting pose, intermediate transition pose, and ending pose, and combined with the velocity and acceleration constraints at each time point, the undetermined coefficients are solved. to , to , to This ensures that the joint angle, angular velocity, and angular acceleration remain continuous at the segment connection points, thereby obtaining a smooth joint trajectory.

7. The multi-degree-of-freedom bionic manipulator control method according to claim 6, characterized in that, Based on the planning results, the kinematic equations of the multi-degree-of-freedom bionic manipulator are determined, including: Obtain the time node parameters, and construct a piecewise polynomial coefficient equation based on the time node parameters. ;in, For boundary condition vectors, The coefficient matrix is ​​composed of the time parameters of each segment. Let be the coefficient vector of the polynomial to be determined; The first The local time variable for each motion segment is represented as follows: Let the first Local initial time of each motion segment It is 0, and the local termination time for each motion segment is given. Then the boundary condition vector, coefficient matrix, and coefficient vector to be determined are: ； in, , and These represent the durations of the first, second, and third segments of the trajectory, respectively. , , and These are the target angles of the joints at critical moments; , , and These are the target angular velocities at the start and end times, respectively. , , and These are the target angular accelerations at the start and end times, respectively. The generalized coordinates of the simulated robotic arm are defined as follows: , for In a real space, the dynamic equations are obtained. ;in, The inertia matrix, For Coriolis force and centrifugal force terms, For gravity, Let be the driving torque vector of each rotary joint. Let Jacobian matrix be the contact point. Let be the contact force vector, representing the combination of normal and tangential forces at all contact points; The contact force is the equivalent generalized force mapped to joint space; the contact force vector. As an optimization variable or as an input of external forces measured by simulation / sensors.

8. The multi-degree-of-freedom bionic manipulator control method according to claim 7, characterized in that, Based on the aforementioned kinematic and dynamic equations, a hierarchical quadratic programming optimization model for multi-finger cooperative control is constructed, including: For the For a bionic finger, the Cartesian space acceleration error at its distal end is defined as: ;in, For the first Jacobian matrix of linear velocity at the tip of a biomimetic finger; The derivative of the linear velocity Jacobian matrix with respect to time is given. For the first The expected Cartesian acceleration at the tip of a bionic finger; Within each control cycle, construct a quadratic programming optimization problem: ;in, The number of bionic fingers participating in collaborative control, This is the joint driving torque vector calculated from the dynamic equations. For the first The task weight matrix of the bionic finger. The energy regularization coefficient; It is a constant greater than 0, used to balance trajectory tracking accuracy and joint drive torque consumption; Construct hierarchical constraints; the hierarchical constraints include at least safety constraints, contact stability constraints, primary task finger trajectory constraints, and auxiliary or secondary task constraints, in descending order of priority. Based on the quadratic programming objective function and the hierarchical constraints, the optimal joint acceleration is solved. With contact force This generates joint control input during the multi-finger collaborative grasping process.

9. The multi-degree-of-freedom bionic manipulator control method according to claim 7, characterized in that, The security constraints include: Dynamic constraints ; Actuator torque constraint conditions ;in, and These are the minimum and maximum driving torque vectors that each actuator is allowed to output, respectively. Collision constraints ,in and This could be any two different bionic fingers, any two different links, a bionic finger and the palm, a bionic finger and a target object, or a bionic finger and an external environmental obstacle. for and The minimum spacing function; The preset safety distance threshold; the minimum spacing function Used to characterize the current generalized coordinates Below, object With object The minimum geometric distance between them; And, the linearization constraints for the current step: ;in To obtain the gradient in generalized coordinates, To control the cycle; The contact state constraint condition is that, in response to determining that the target bionic finger is not in contact with an object, the contact force of the target bionic finger is zero; the target bionic finger can be any bionic finger.

10. The multi-degree-of-freedom bionic manipulator control method according to claim 1, characterized in that, The pre-trained pattern recognition model includes a dual-input branch deep regression network; The dual-input branch deep regression network includes a visual input branch, a surface electromyography input branch, a fusion and regression layer, a training phase supervision and optimization module, and a deployment phase feedback module. The visual input branch includes a visual keypoint input unit, a depthwise separable convolutional module, and a first fully connected mapping layer; the visual keypoint input unit is used to input visual keypoint features; the depthwise separable convolutional module is used to extract hand spatial geometric structure features; the first fully connected mapping layer is used to further map the convolutional features into visual embedding features; The surface electromyography (EMG) input branch includes an EMG feature input unit, a depthwise separable convolutional module, and a second fully connected mapping layer. The EMG feature input unit is used to input EMG feature vectors. The depthwise separable convolutional module is used to extract feature representations reflecting muscle activation states. The second fully connected mapping layer is used to further compress the EMG features and map them into EMG embedded features. The fusion and regression layer includes a feature splicing and fusion unit, a multi-layer fully connected regression unit, a joint angle prediction output unit, and an output description unit. The feature splicing and fusion unit is used to splice visual embedded features and electromyographic embedded features to form a joint fused feature. The multi-layer fully connected regression unit is used to perform nonlinear mapping and regression on the fused joint feature. The joint angle prediction output unit is used to output the predicted joint angle. The output description unit is used to indicate that the output predicted joint angle corresponds to the joint representation of the active joint angle and lateral swing angle of each of the five fingers. The training phase supervision and optimization module includes an inertial navigation IMU supervision label input unit, a loss function calculation unit, and a parameter optimization unit. The IMU supervision label input unit is used to provide the true value of the joint angle. The loss function calculation unit is used to construct the mean squared error loss and the joint range of motion constraint term based on the difference between the predicted joint angle and the true joint angle. The parameter optimization unit is used to iteratively update the network parameters using the Adam optimizer and select the optimal model parameters based on the validation set results. The deployment phase feedback explanation module includes a robot end feedback correction explanation unit and an input boundary explanation unit; the robot end feedback correction explanation unit is used to explain that the execution effect can be corrected by using the closed-loop feedback of the robot end inertial measurement unit (IMU) during the deployment phase; the input boundary explanation unit is used to explain that the inertial feedback information during the deployment phase is not necessarily used as the forward input branch of the neural network.

Images