A method and system for controlling a robot joint based on a motion capture device

Abstract

The application discloses a kind of robot joint control method and system based on motion capture device, belong to robot control technical field.The existing robot joint control scheme, lack effective human-computer cooperation and natural interaction mechanism, leading to robot flexibility is poor, cannot naturally express the intention of operator.A kind of robot joint control method based on motion capture device of the application, by constructing motion capture module, data processing module, coordinate system transformation module, solving optimization module, the rotation posture data of human joint collected using motion capture device is converted into the target joint control information required for robot execution, realize robot joint control based on motion capture device, so that human action can be mapped to robot efficiently and accurately, so that robot flexibility is good, can effectively handle sudden action or complex collaborative operation, so that the intention of operator can be naturally expressed, realize human-computer cooperation and natural interaction.

Description

TECHNICAL FIELD

[0001] The application relates to a robot joint control method and system based on a motion capture device, and belongs to the technical field of robot control. BACKGROUND

[0002] A multi-objective optimization method is disclosed in Chinese literature (Qiang Yiling, Wu Di, Hou Guowei, et al. A multi-objective optimization method for inverse kinematics of redundant manipulator [J]. Mechanical Science and Technology, 2024, 43 (6): 934-942), which combines the parameterized joint method and improves the multi-objective evolutionary algorithm MOEA / D to form the IM-IK algorithm (Improved MOEA / D-inverse kinematics) to solve the inverse kinematics problem of the redundant manipulator. The adaptive interval search strategy is added to the multi-objective evolutionary algorithm MOEA / D framework to improve the local search ability of the algorithm and improve the solving speed. The method can guarantee the continuity of the joint motion of the manipulator and effectively avoid the joint limit while meeting the accuracy requirements of the end effector pose of the manipulator.

[0003] The above scheme can realize intelligent motion control of the multi-degree-of-freedom arm of the engineering machinery, but lacks effective human-machine cooperation and natural interaction mechanism, and does not involve how to efficiently and accurately map human actions to robots, resulting in poor flexibility of the robot, difficulty in handling sudden actions or complex cooperative operations, and inability to naturally express the intention of the operator, making it difficult to achieve high-precision synchronous control of the robot arms and the manipulator.

[0004] Further, the scheme and the existing robot data acquisition scheme lack efficient capture and accurate restoration of natural human actions, resulting in insufficient human-machine cooperation training data and limiting the development of intelligent systems.

[0005] The information disclosed in the background technology is only used to understand the background of the inventive concept, and therefore it can include information that does not constitute prior art. SUMMARY

[0006] In view of the above problems or one of the above problems, a first object of the present application is to provide a robot joint control method and system based on a motion capture device, by constructing a motion capture module, a data processing module, a coordinate system transformation module, and a solving optimization module, the rotation posture data of the human joints collected by the motion capture device is converted into target joint control information required for the robot to execute, so as to realize robot joint control based on the motion capture device, so that human actions can be efficiently and accurately mapped to the robot, so that the robot has good flexibility and can effectively handle sudden actions or complex cooperative operations, so that the operator's intention can be naturally expressed, human-machine cooperation and natural interaction are realized, and high-precision synchronous control of the robot arms and the mechanical hand is facilitated; meanwhile, efficient and accurate data acquisition can be realized, so that human natural actions can be efficiently captured and accurately restored.

[0007] In view of the above problems or one of the above problems, a second object of the present application is to provide a robot joint control method and system based on a motion capture device, which can realize high-fidelity and low-delay mapping of human upper limb and hand actions, effectively improve the control flexibility and precision of the robot in complex operation tasks, and can realize efficient capture and accurate restoration of human natural actions, provide real training data for the robot, and has significant practical value and engineering promotion potential.

[0008] To achieve one of the above objects, the first technical solution of the present application is:

[0009] A robot joint control method based on a motion capture device, comprising the following steps:

[0010] Step one, receiving the rotation posture data of the human joints collected by the motion capture device through the motion capture module created in advance;

[0011] Step two, converting the rotation posture data into a unified rotation matrix using the data processing module created in advance, and combining the human structure information to calculate the global pose of each joint to obtain the pose information;

[0012] Step three, converting the pose information from the motion capture device coordinate system to the robot operating system coordinate system and the robot body coordinate system in sequence using the coordinate system transformation module created in advance to obtain the pose standard information of the robot;

[0013] Step four, using the solving optimization module created in advance, based on the pose standard information, performing pose inverse solution on the human joints to obtain the target joint control information required for the robot to execute, and realizing robot joint control based on the motion capture device.

[0014] The application can efficiently and accurately map human actions to robots, so that the robot has good flexibility and can effectively handle sudden actions or complex collaborative operations; thus, the operator's intention can be naturally expressed, human-robot collaboration and natural interaction can be realized, and high-precision synchronous control of robot arms and mechanical hands can be facilitated.

[0015] The application can not only efficiently collect data on real robots, but also accurately control robots in a simulation environment to achieve efficient data collection, so that natural human actions can be efficiently captured and accurately restored.

[0016] Further, the application uses a motion capture device to collect real-time rotation posture data of human joints and converts it into structured target joint control information, so as to efficiently and accurately collect and map human action data. The application can provide high-quality training data for robot control strategy learning, verification and optimization, and has the advantages of natural human-computer interaction and strong system expansion. The application can not only be used to control real robots to efficiently collect data and accurately restore natural human actions, but also can directly control robots in a simulation environment to collect data synchronously, thereby providing a solid data foundation for intelligent behavior modeling under complex tasks, human-robot collaboration and virtual-real fusion research.

[0017] As a preferred technical measure:

[0018] Step one, the method for receiving the rotation posture data of human joints collected by the motion capture device through the pre-created motion capture module is as follows:

[0019] A data transmission channel for connecting the motion capture module and the motion capture device is constructed through a network or Bluetooth communication;

[0020] The upper limb and hand posture data of the operator are collected by the motion capture device to obtain motion capture data;

[0021] The motion capture data, including local rotation information of each joint of the operator, are received through the data transmission channel;

[0022] A mapping table and a data sharing array are constructed based on the types of human joints;

[0023] Based on the mapping table, the local rotation information of each joint is stored in the data sharing array, and a joint index is set according to the storage position of the local rotation information, so as to obtain the rotation posture data.

[0024] As a preferred technical measure:

[0025] Step two, using the pre-created data processing module, convert the rotation pose data into a unified rotation matrix, and combine the human body structure information to calculate the global pose of each joint, the method of obtaining the pose information is as follows:

[0026] Based on the data sharing array, according to the joint index, extract the local rotation information of each joint from the rotation pose data;

[0027] Convert the local rotation information into quaternion form to obtain the pose quaternion;

[0028] According to the pose quaternion, the rotation matrix is calculated;

[0029] According to the topological structure of the human upper limb skeleton, select key control joints, including but not limited to the hip, multi-segment spine, shoulder, upper arm, forearm and hand;

[0030] Based on the rotation matrix and the key control joints, the global pose of each joint of the human body is calculated using forward kinematics, which at least includes the hand end pose and the finger end pose;

[0031] Transform the hand end pose into the local coordinate system with the hip as the origin to obtain the arm pose information;

[0032] Transform the finger end pose into the local coordinate system with the wrist as the origin to obtain the finger pose information;

[0033] Summarize the arm pose information and the finger pose information to obtain the pose information.

[0034] As a preferred technical measure:

[0035] The method of converting the local rotation information into quaternion form to obtain the pose quaternion is as follows:

[0036] Step 11: Based on the conversion relationship between degrees and radians, process the local rotation information to calculate the pose radian data;

[0037] Step 12: According to the cosine and sine calculation formula, process the pose radian data to calculate the cosine and sine values of each axis;

[0038] Step 13: Based on the cosine and sine values of each axis and the rotation sequence of each axis, calculate the pose quaternion.

[0039] As a preferred technical measure:

[0040] The method of calculating the global pose of each joint of the human body based on the rotation matrix and the key control joints using forward kinematics is as follows:

[0041] Step 21, obtain a rotation matrix, which at least includes a global rotation matrix of the hip joint and a local rotation matrix of the spinal joint;

[0042] Step 22, multiply the global rotation matrix with the local rotation matrix to obtain a global rotation matrix of the spinal joint;

[0043] Step 23, according to the global rotation matrix of the spinal joint, and in combination with the topological relationship of the key control joint, a global rotation matrix of the arm end is calculated to complete the rotation construction of the whole upper limb chain;

[0044] Step 24: on the basis of obtaining the global rotation matrix, relying on forward kinematics, in combination with the spatial offset vector between joints, the global pose of each joint is calculated.

[0045] As a preferred technical measure:

[0046] The arm end pose is transformed into a local coordinate system with the hip as the origin to obtain the arm pose information as follows:

[0047] According to the global position difference between the arm and the hip, the offset amount of the arm joint to the hip joint in the global coordinate system is obtained;

[0048] Based on the offset amount, in combination with the inverse transformation of the hip rotation matrix, the position information of the arm in the hip coordinate system is obtained;

[0049] The arm rotation matrix is multiplied by the inverse matrix of the hip rotation matrix to obtain the arm pose matrix of the arm in the hip coordinate system;

[0050] The position information of the arm and the arm pose matrix are combined to obtain the arm pose information, and the transformation of the arm end pose is completed.

[0051] As a preferred technical measure:

[0052] Step three, using the coordinate transformation module created in advance, the pose information is converted from the motion capture device coordinate system to the robot operating system coordinate system and the robot body coordinate system in sequence to obtain the standard pose information of the robot as follows:

[0053] According to the difference between the motion capture device coordinate system and the robot operating system coordinate system, a transformation matrix between the robot operating system coordinate system and the motion capture device coordinate system is constructed;

[0054] Based on the transformation matrix, the pose information output by the motion capture device is converted into the arm end pose in the robot operating system coordinate system;

[0055] The robot body coordinate system is a local coordinate system, which includes a left arm coordinate system and a right arm coordinate system;

[0056] According to the difference between the robot operating system coordinate system and the left arm coordinate system, a first rotation transformation matrix and a second rotation transformation matrix are constructed.

[0057] According to the difference between the robot operating system coordinate system and the right arm coordinate system, a first rotation transformation matrix and a second rotation transformation matrix are constructed.

[0058] Based on the first rotation transformation matrix, the second rotation transformation matrix, the first rotation transformation matrix and the second rotation transformation matrix, and the end-of-arm pose under the robot operating system coordinate system, the end-of-arm pose under the robot body coordinate system is calculated.

[0059] Taking the wrist joint as the reference origin, the finger end pose coordinate system is converted, and appropriate scaling is performed in combination with the finger length to obtain the finger end pose under the robot body coordinate system.

[0060] The end-of-arm pose under the robot body coordinate system and the finger end pose are coupled to obtain the pose standard information of the robot.

[0061] Further, the motion capture device coordinate system is a left-hand coordinate system, with the X-axis to the right, the Y-axis upward, and the Z-axis forward. The robot operating system coordinate system is a right-hand coordinate system, with the X-axis forward, the Y-axis to the right, and the Z-axis upward.

[0062] As a preferred technical measure:

[0063] Step four, using the previously created solving optimization module, based on the pose standard information, the human joint is inversely solved to obtain the target joint control information required by the robot execution, and the method is as follows:

[0064] Obtain the pose standard information, which includes the end-of-arm pose under the robot body coordinate system and the finger end pose;

[0065] Convert the end-of-arm pose into Euclidean group data;

[0066] Taking the current robot state as the initial value and the target pose as the optimization target, a target function is constructed;

[0067] Based on the Euclidean group data, the target function is solved to obtain the joint angle solution that satisfies the constraint condition;

[0068] Obtain the degree of freedom information of the robot fingers and the current posture of the fingers;

[0069] According to the degree of freedom information, determine the finger joint limiting condition and the inter-finger coordination constraint information;

[0070] Based on the finger end pose, the current posture of the fingers, the joint limiting condition, and the inter-finger coordination constraint information, the five-finger angle control information is solved.

[0071] The joint angle solution and the five-finger angle control information are coupled to obtain target joint control information required by the robot to execute.

[0072] As a preferred technical measure:

[0073] The control issuing module is further included.

[0074] The control issuing module is used to generate control instructions and control the joint movement of the robot, and the method is as follows:

[0075] According to the target joint control information, control instructions required by the robot to execute are generated.

[0076] The control commands are sent to the robot at a frequency of 500 Hz through a communication protocol to control the robot in real time.

[0077] When the robot communication is interrupted or abnormal, the control instructions of the previous frame are used to maintain the continuity of the robot action.

[0078] When no new control instructions are received for more than a set time, the robot is controlled to enter a safe state.

[0079] The control issuing module is used to not only interact with a real robot, but also interact with a simulated robot, so that the data acquisition efficiency can be significantly improved, and multi-scene data acquisition can be realized, the diversity of data is improved, and the training data of the robot is enriched.

[0080] To achieve one of the above purposes, a second technical solution of the present application is:

[0081] A robot joint control system based on a motion capture device includes a motion capture module, a data processing module, and a solving and optimization module.

[0082] The motion capture module is used to receive the rotation posture data of the human joints collected by the motion capture device and store the rotation posture data in the shared memory.

[0083] The data processing module is used to call the rotation posture data from the shared memory, convert the rotation posture data into a unified rotation matrix, and calculate the global pose of each joint in combination with the human structure information to obtain the pose information.

[0084] The solving and optimization module is used to perform posture back analysis on the human joints based on the pose standard information to obtain the target joint control information required by the robot to execute.

[0085] The motion capture module, the data processing module and the solving optimization module are arranged, and based on the low-delay communication mechanism of shared memory, high-fidelity and low-delay mapping of human upper limb and hand action can be realized, real-time control of the robot is facilitated, the rotation attitude data conversion mechanism and the coordinate system conversion mechanism are combined, the posture consistency of man and machine is maintained, and the control precision can be effectively improved, natural human action reproduction and remote control can be realized, the joint control intuitiveness of the robot is good, efficient and accurate data acquisition can be realized, real training data of the robot is provided, and therefore the robot data acquisition and actual deployment efficiency can be significantly improved, and the engineering value is high.

[0086] To achieve one of the above purposes, a third technical solution of the present application is:

[0087] A server comprises:

[0088] One or more processors;

[0089] A storage device for storing one or more programs;

[0090] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned motion capture device-based robot joint control method.

[0091] Compared with the prior art, the present application has the following beneficial effects:

[0092] The present application can efficiently and accurately map human action to the robot by constructing the motion capture module, the data processing module, the coordinate system conversion module and the solving optimization module, converting the rotation attitude data of the human joint collected by the motion capture device into the target joint control information required by the robot, realizing motion capture device-based robot joint control, so that the robot has good flexibility and can effectively handle sudden actions or complex cooperative operations, and thus the operator's intention can be naturally expressed, human-machine cooperation and natural interaction can be realized, and high-precision synchronous control of the robot arms and the mechanical hand is facilitated.

[0093] Further, the present application can realize high-fidelity and low-delay mapping of human upper limb and hand action, effectively improve the control flexibility and precision of the robot in complex operation tasks, and can realize efficient and accurate data acquisition, so that natural human action can be efficiently captured and accurately restored, real training data of the robot is provided, and the present application has significant practical value and engineering promotion potential.

[0094] Further, the application uses a motion capture device to collect real-time rotation posture data of human joints, and converts the data into structured target joint control information, so that efficient and accurate human motion data collection and mapping are realized. The method can provide high-quality training data for learning, verification and optimization of robot control strategies, and has the advantages of natural human-computer interaction and strong system expansion. The application can not only be used to control a real robot to collect data efficiently, but also can be used to directly control a robot in a simulation environment to collect data synchronously, thereby providing a solid data foundation for intelligent behavior modeling, human-robot collaboration and virtual-real fusion research under complex tasks. BRIEF DESCRIPTION OF DRAWINGS

[0095] Figure 1 FIG. 1 is a flowchart of a robot joint control method based on a motion capture device according to an embodiment of the application;

[0096] Figure 2 FIG. 2 is a structural block diagram of a robot joint control system based on a motion capture device according to an embodiment of the application. DETAILED DESCRIPTION

[0097] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts should fall within the scope of protection of the present application. The present application covers any alternative, modification, equivalent method and solution defined by the claims within the spirit and scope of the present application.

[0098] As shown in FIG. 1, a specific embodiment of the robot joint control method based on a motion capture device according to the application is as follows: Figure 1

[0099] A robot joint control method based on a motion capture device includes the following steps:

[0100] Step 1: receiving rotation posture data of human joints collected by a motion capture device through a motion capture module created in advance;

[0101] Step 2: converting the rotation posture data into a uniform rotation matrix using a data processing module created in advance, and combining human structure information to calculate the global pose of each joint to obtain pose information;

[0102] Step 3: converting the pose information from the motion capture device coordinate system to the robot operating system coordinate system and the robot body coordinate system in sequence using a coordinate system conversion module created in advance to obtain standard pose information of the robot; ​

[0103] Step four, using the pre-created solving optimization module, based on the pose standard information, the posture of the human body joints is inversely solved, the target joint control information required by the robot is obtained, and the robot joint control based on the motion capture device is realized.

[0104] As shown in the figure, one specific embodiment of the robot joint control system based on the motion capture device of the application is as follows: Figure 2

[0105] A robot joint control system based on a motion capture device, comprising a motion capture device, a motion capture module, a data processing module, a solving optimization module, and a control issuing module.

[0106] The motion capture device is used to collect the posture data of the upper limbs and hands of the operator. The motion capture module first establishes a connection with the data sending end of the motion capture device through the communication protocol TCP / UDP or Bluetooth, then receives and decodes the data packet (motion capture data) to obtain real data, and finally writes the decoded real data into shared memory (data sharing array) for use by the data processing module.

[0107] The data processing module includes a data format conversion module, a pose calculation module, and a coordinate system transformation module. The data format conversion module obtains motion capture data from shared memory and processes the data generated by different motion capture devices into a unified data format. The pose calculation module reconstructs the end pose information of the upper limbs and hands of the human body based on the local pose data of each joint provided by the motion capture system, providing input for subsequent coordinate transformation and inverse kinematics solving. The coordinate system transformation module maps the human body pose from the coordinate system defined by the motion capture device to the robot body coordinate system.

[0108] The solving optimization module inversely solves the joint angles based on the target end pose, and the control issuing module sends the solved joint angles to the robot for execution through the communication protocol DDS. The use of DDS communication design has strong universality, making the application not only interact with real robots, but also interact with simulated robots, thereby significantly improving data acquisition efficiency and realizing multi-scene data acquisition, improving data diversity, and thus enriching the training data of the robot.

[0109] In this embodiment, the data format conversion module can process different data formats (such as quaternions, rotation matrices, Euler angles) generated by different motion capture devices into the form of rotation matrices. The processing process is as follows:

[0110] First, convert the data into quaternion form, then calculate the rotation matrix based on the quaternion. The process of calculating the quaternion is as follows:

[0111] ​Step 1: Convert the received angle information (angle) into radian information (rad) by the following formula:

[0112]

[0113] Step 2: Calculate the half-angle cosine value (c) and half-angle sine value (s) of each axis using the calculated radian information (rad) by the following formula:

[0114]

[0115]

[0116] Step 3: Calculate the quaternion components according to the rotation order (XYZ, XZY, ZXY, ZYX, YZX, YXZ) When calculating the quaternion components, different rotation orders are used. Since this embodiment uses the XYZ rotation order, only the calculation formula for the XYZ rotation order is listed below:

[0117]

[0118]

[0119]

[0120]

[0121] where X represents the X-axis of the coordinate system, Y represents the Y-axis of the coordinate system, and Z represents the Z-axis of the coordinate system; represents the x component of the quaternion, represents the y component of the quaternion, represents the z component of the quaternion, represents the value of the scalar part; represents the half-angle cosine value around the X-axis, represents the half-angle cosine value around the Y-axis, represents the half-angle cosine value around the Z-axis; represents the half-angle sine value around the X-axis, represents the half-angle sine value around the Y-axis, represents the half-angle sine value around the Z-axis.

[0122] Step 4: After the calculation is completed, return the adjusted quaternion After obtaining the quaternion, the corresponding rotation matrix can be obtained by the following formula: :

[0123]

[0124] In this embodiment, the pose calculation module calculates the global pose of each joint of the human body by forward kinematics, and then transforms the pose of the arm end into a local coordinate system with the hip as the origin, and transforms the pose of the finger end into a local coordinate system with the wrist as the origin.

[0125] The calculation process of the arm end pose is as follows:

[0126] Step 1: Set the joint index and offset required to be used, the joint index refers to the position of the required joint in the data sharing array, and the offset refers to the distance between the joints.

[0127] Step 2: According to the topological structure of the human upper limb skeleton, select key control joints, including but not limited to the hip, spine (multiple segments), shoulder, upper arm, forearm and hand, etc. Each joint corresponds to a unique label or index in the motion capture data, and the system aligns it with the original data through a mapping table and uniformly converts it into a rotation matrix or quaternion format to obtain the local rotation matrix of each joint. In this way, the system can be compatible with the output structure of different motion capture devices, and ensure that all joint rotation poses participating in the calculation have consistency and comparability.

[0128] Step 3: The motion capture device usually provides local rotation information of each joint, that is, the rotation of the joint relative to its parent node. In order to obtain the global rotation matrix of each joint in the world coordinate system, the system performs matrix cascading calculation according to the hierarchical structure of the human skeleton to obtain the global rotation matrix. The specific method is as follows:

[0129] The global rotation matrix of the spine joint can be obtained by multiplying the global rotation matrix of the hip and the local rotation of the spine. This process is recursively expanded in a way that propagates from the root node (such as the hip joint hips) level by level, until the end of the hand, completing the rotation construction of the entire upper limb chain.

[0130] Taking the calculation of the spine joint as an example, the global rotation matrix relative to the hip joint hips needs to be calculated. The calculation formula is as follows:

[0131]

[0132] Where, is the global rotation matrix of the spine joint, is the global rotation matrix of the hip joint hips, is the local rotation matrix of the spine joint.

[0133] Through similar calculation, the global rotation matrix of the arm end can be finally obtained.

[0134] Step 4: Calculate the global pose of each joint based on the offset of each joint. Based on the obtained global rotation matrix, the system combines the known human body structure parameters (i.e. the spatial offset vector between joints) to calculate the global position of each joint. This process relies on the principle of forward kinematics: the position of a child joint is determined by the parent joint position, parent joint rotation, and relative offset.

[0135] The system sets the offset value between each pair of adjacent joints based on the standard human model or measured calibration data, and gradually superimposes the transformation to finally obtain the global pose information (position + rotation) of the upper limb end (such as left / right wrist).

[0136] In this embodiment, the formula for calculating the spine joint position is as follows:

[0137]

[0138] Where, is the global position of the spine joint; is the global position of the hips joint, generally the origin; is the global rotation matrix of the hips joint; is the offset of the spine joint relative to the hips joint, which needs to be measured; similarly, by calculating level by level, the global position of the arm end can be finally calculated .

[0139] Step 5: To adapt to the robot control framework, the system needs to further convert the pose of the arm end to the local coordinate system with the hip as the origin, thereby eliminating the influence of the overall displacement of the human body on the action mapping. The conversion process includes relative offset calculation of position vector and coordinate system transformation of rotation matrix. The position transformation method is as follows:

[0140] According to the global position difference between the hand and the hip, combined with the inverse transformation of the hip rotation, the position of the hand in the hip coordinate system is obtained.

[0141] The posture transformation method is as follows:

[0142] The hand rotation matrix is multiplied by the inverse matrix of the hip rotation to obtain its posture in the hip coordinate system.

[0143] For finger joints, the system uses a similar process, but the reference origin is the wrist joint. Due to the structural differences between the length of the human finger and the robot dexterous hand, the system introduces a scaling factor in the end posture conversion process to improve the consistency and practicality of the action mapping. The calculation formula is as follows:

[0144]

[0145]

[0146]

[0147] wherein, is the offset of the hand joint to the hips joint in the global coordinate system; is the position of the hand joint in the hip coordinate system; is the rotation of the hand joint in the hip coordinate system.

[0148] The specific value of the scaling factor is the ratio of the length of the human finger to the length of the robot finger.

[0149] Step 6: Return the position and pose of the arm end, i.e. feed back the and calculated above to the subsequent module.

[0150] In this embodiment, the calculation of the finger end pose and the calculation of the arm are basically the same, except that the length of the finger needs to be considered and the rotation matrix of the finger needs to be converted to the local coordinate system with the wrist as the origin. Since there is a certain gap between the length of the human finger and the length of the robot dexterous hand, the real length of the configurator's finger and the length of the robot dexterous hand need to be measured during calculation in order to perform scaling processing.

[0151] In this embodiment, in order to realize the effective docking of motion capture data and the robot system, the coordinate system conversion module is used to convert the original data from the coordinate system used by the motion capture device to the control coordinate system used by the robot body. Since there are differences in axis definition and chirality between the two coordinate systems, multi-stage conversion is necessary.

[0152] The motion capture device Unity usually uses a left-handed coordinate system, with the X axis to the right, the Y axis upward, and the Z axis forward. The robot operating system ROS and most robot platforms use a right-handed coordinate system, with the X axis forward, the Y axis right, and the Z axis upward. The robot body Unitree has its own defined local coordinate system, with the axis direction varying due to the left and right arms.

[0153] In this embodiment, the method for converting the end pose coordinate system of the arm by the coordinate system conversion module is as follows:

[0154] First, the coordinate system of the motion capture device Unity is converted to the coordinate system of the robot operating system ROS, and the process is as follows:

[0155] The pose data output by the motion capture system is converted into pose data in the robot operating system (ROS) right-hand coordinate system through a predefined rotation transformation matrix. The pose data includes position and corresponding attitude. To facilitate operation, the position and attitude are converted separately in this embodiment. The expression for position conversion is as follows:

[0156]

[0157] wherein is the position of the arm end in the robot operating system (ROS) coordinate system, is the position of the arm end in the motion capture device Unity coordinate system, is the transformation matrix between the robot operating system (ROS) coordinate system and the motion capture device Unity coordinate system, and the expression is as follows:

[0158]

[0159] The expression for attitude conversion is as follows:

[0160]

[0161] wherein is the attitude of the arm end in the robot operating system (ROS) coordinate system, is the processed attitude of the arm end in the motion capture device Unity coordinate system, is the transformation matrix between the robot operating system (ROS) coordinate system and the motion capture device Unity coordinate system, and the expression is as follows:

[0162]

[0163] Then, the robot operating system (ROS) coordinate system is converted into the robot (Robot) coordinate system, and the process is as follows:

[0164] Considering that the left and right arm coordinate axes of the robot body are defined differently, the system uses independent rotation transformation matrices for the left and right arms to ensure that the converted end attitude meets the robot control input specifications, and the expression is as follows:

[0165]

[0166] wherein, is the end attitude of the arm in the Unitree body coordinate system, and are both transformation matrices.

[0167] For the left arm coordinate system, the expression for the transformation matrix is as follows:

[0168]

[0169]

[0170] For the right arm coordinate system, the expression of the transformation matrix is as follows:

[0171]

[0172]

[0173] In this embodiment, the method of the coordinate system conversion module for converting the finger tip position coordinate system is as follows:

[0174] The coordinate system conversion process of the finger is similar to that of the arm, but the reference origin is the wrist joint. Due to the differences in structure and degrees of freedom between the robot dexterous hand and the human hand, appropriate scaling is required in the conversion process in combination with the actual finger length to maintain the consistency of the motion.

[0175] The system sets a transformation matrix for the left and right hands respectively to ensure that the end position of each finger can be accurately mapped into the control space of the robot finger. The calculation formula is as follows:

[0176]

[0177] wherein is the end position of the finger in the robot Unitree body coordinate system, is the transformation matrix, is the end position of the finger in the Unity coordinate system of the motion capture device.

[0178] The expression of the left hand transformation matrix is as follows:

[0179]

[0180] The expression of the right hand transformation matrix is as follows:

[0181]

[0182] Therefore, when the left and right arms and the fingers of the left and right hands are converted in the body coordinate system, their transformation matrices are different, thereby further improving the control accuracy of the robot.

[0183] In this embodiment, to realize the accurate tracking of the robot joints to the target motion of the human body, the solving and optimization module performs inverse kinematics (IK) solving based on the end position information, and processes the joint angle calculation of the arm and the finger respectively.

[0184] The arm part has 7 degrees of freedom, and the system implements inverse kinematics IK solving based on the library Pinocchio for robot dynamics calculation and the open-source software tool CasADi for dynamic optimization.

[0185] The library Pinocchio for robot dynamics calculation can construct a robot motion model to generate kinematic constraints of each joint, and the open-source software tool CasADi for dynamic optimization can define an objective function and optimization variables to perform numerical optimization calculation.

[0186] The calculation process of the arm joint angle is as follows:

[0187] First, the target pose (position and rotation matrix) of the arm end is converted into a special Euclidean group SE(3) format;

[0188] Then, an optimization problem is constructed, taking the current robot state as the initial value and the target pose as the optimization target to construct an objective function; and the open-source software tool CasADi is used to solve the objective function to obtain the joint angle solution that satisfies the constraint condition.

[0189] This method can introduce a weighted strategy among multiple targets, joint restrictions, and a redundant solution determination mechanism to ensure high feasibility and dynamic stability of the solution.

[0190] The open-source library dex_retargeting is used for finger inverse kinematics IK solving, and the calculation process is as follows:

[0191] First, the degree of freedom information of the robot finger is obtained, and in this embodiment, each five-fingered hand contains 20 degrees of freedom; then a scaling factor is set to adapt to the difference in geometric size between the human hand and the robot hand, and the value of the scaling factor is determined according to the ratio of the human hand and the robot hand; then modeling is performed based on joint limiting, inter-finger coordination and other physical constraints, and then optimization is performed according to the target finger end pose and the current pose; the target end pose and the current pose of each finger are taken as input, and the optimizer is called to obtain the complete joint angle output of the five fingers.

[0192] In this embodiment, the control issuing module includes the following contents:

[0193] The control issuing module sends control commands to the robot system at a frequency of 500 Hz through the communication protocol DDS, which not only meets the real-time control requirements, but also maintains the continuity of the action when the communication is interrupted or abnormal, avoiding the phenomenon of robot shaking or instability. In addition, this module supports a fault fallback mechanism that will automatically enter a safe state when no new control instructions are received within a set time, ensuring system stability and personnel safety.

[0194] The application has been verified in multiple practical platforms, including using the motion capture equipment of Magic Leap HZ1, Tiangong F-1, controlling the robots such as Unitree robot Unitree G1 (with three-fingered dexterous hand) and Unitree robot Unitree H1-2 (with five-fingered dexterous hand). Therefore, the application can support multiple brands of motion capture equipment and robot platforms, and has strong universality.

[0195] A specific embodiment of the application is applied:

[0196] In this embodiment, the HZ1 motion capture equipment produced by Magic Leap Company is selected, and the Unitree humanoid robot H1-2 is used as the controlled platform. The robot is equipped with two seven-degree-of-freedom arms and a twenty-degree-of-freedom five-fingered dexterous hand.

[0197] The system communicates with the motion capture equipment through the communication protocol UDP to receive the rotation posture data of the upper limbs and hands of the human body (represented by quaternions) in real time.

[0198] During the operation of the system, the motion capture module receives and analyzes the UDP data packet, and writes the skeletal joint rotation information into the shared memory. The data processing module extracts the corresponding skeletal data from the shared memory, converts it into a unified rotation matrix format, and calculates the global pose of each joint in combination with the human body structure information.

[0199] Through the coordinate system transformation module, the system sequentially converts the above-mentioned pose information from the motion capture equipment Unity coordinate system to the robot operating system ROS coordinate system and the Unitree robot body coordinate system. The solving and optimization module performs inverse kinematics on the dual arms and the five-fingered dexterous hand respectively, and obtains the target joint angle required for the robot to execute.

[0200] Finally, the control issuing module sends the calculation results to the robot bottom control system through the communication protocol DDS at a frequency of 500Hz, so that the robot performs coordinated motion consistent with the human body motion, realizes real-time and natural mapping of human body motion to robot joints, and successfully completes complex operation tasks such as grabbing and gesture demonstration.

[0201] Through this embodiment, it can be verified that the low-latency communication mechanism based on shared memory of the application can realize real-time response; at the same time, combined with forward / inverse kinematics and proportional scaling, the human-machine pose remains consistent, thereby effectively improving the control accuracy; further, it can realize natural human motion reproduction and remote control, so that the robot joint control is intuitive; and it can significantly improve the efficiency of robot data acquisition and actual deployment, and has high engineering value.

[0202] A server embodiment applying the method of the application:

[0203] A server comprises:

[0204] One or more processors;

[0205] a storage device for storing one or more programs;

[0206] The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the motion capture device based robot joint control method described above.

[0207] The module in the present application is an object which constitutes an objective description form structure by means of entity or virtual representation. The object is not equal to the object and is not limited to entity and virtual. It can be a data processing function, a software program, a processing mode, a use method, an operation mode, a work flow, an application process, an electronic hardware, a circuit module, a processing system, a system imitation product or a simulation simulation object.

[0208] Finally, it should be noted that: the above-described embodiments are merely specific embodiments of the present application, used to illustrate the technical solutions of the present application, and are not limited thereto. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications to the technical solutions recorded in the foregoing embodiments or make equivalent replacements to some technical features within the technical range disclosed by the present application. The modifications or replacements do not change the substance of the corresponding technical solutions, and the corresponding technical solutions are still within the spirit and scope of the embodiments of the present application. Any modifications or equivalent replacements without departing from the spirit and scope of the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A robot joint control method based on a motion capture device, characterized by: The following steps are involved: Step 1: Using the motion capture module created in advance, the rotational posture data of the human joints collected by the motion capture device is received; Step 2: Use the previously created data processing module to convert the rotational posture data into a unified rotation matrix, and combine it with the human body structure information to calculate the global posture of each joint to obtain posture information; Step 3: Using the previously created coordinate system transformation module, the pose information is converted from the motion capture device coordinate system to the robot operating system coordinate system and the robot body coordinate system in sequence to obtain the standard pose information of the robot; It includes the following: According to the difference between the coordinate system of the motion capture device and the coordinate system of the robot operating system, a transformation matrix between the coordinate system of the robot operating system and the coordinate system of the motion capture device is constructed; Based on the transformation matrix, the pose information output by the motion capture device is converted into the arm end pose in the robot operating system coordinate system; The robot body coordinate system is a local coordinate system, which includes the left arm coordinate system and the right arm coordinate system; Constructing a first rotation transformation matrix and a second rotation transformation matrix according to the difference between the robot operating system coordinate system and the left arm coordinate system; According to the difference between the robot operating system coordinate system and the right arm coordinate system, the rotation transformation matrix 1 and the rotation transformation matrix 2 are constructed; Based on the first rotation transformation matrix, the second rotation transformation matrix, the rotation transformation matrix 1 and the rotation transformation matrix 2 and the end arm posture in the robot operating system coordinate system, the end arm posture in the robot body coordinate system is calculated. Taking the wrist joint as the reference origin, the finger end pose coordinate system is transformed and appropriately scaled based on the finger length to obtain the finger end pose in the robot body coordinate system. The arm end pose and finger end pose in the robot body coordinates are coupled to obtain the robot's pose standard information; In step 4, the previously created solution optimization module is used to perform posture inverse analysis on the human joints based on the standard posture information, and the target joint control information required for the robot execution is obtained to realize the robot joint control based on the motion capture device.

2. The robot joint control method based on a motion capture device according to claim 1, characterized in that: Step 1: The method for receiving the rotational posture data of the human joints collected by the motion capture device through the motion capture module created in advance is as follows: Build a data transmission channel for connecting the motion capture module and the motion capture device through network or Bluetooth communication; Using motion capture equipment to collect the operator's upper limb and hand posture data to obtain motion capture data; Receiving motion capture data including local rotation information of each joint of the operator through a data transmission channel; Based on the type of human joints, build a mapping table and data sharing array; Based on the mapping table, the local rotation information of each joint is stored in the data sharing array, and the joint index is set according to the storage location of the local rotation information to obtain the rotation posture data.

3. The robot joint control method based on a motion capture device according to claim 1, wherein: In step 2, the previously created data processing module is used to convert the rotational posture data into a unified rotation matrix. Combined with the human body structure information, the global posture of each joint is calculated. The method for obtaining the posture information is as follows: Based on the data sharing array and joint index, the local rotation information of each joint is extracted from the rotation posture data; Convert the local rotation information into quaternion form to obtain the pose quaternion; According to the pose quaternion, the rotation matrix is ​​calculated; According to the topological structure of the human upper limb skeleton, key control joints are selected, including but not limited to the hip, multi-segment spine, shoulder, upper arm, forearm and hand; Based on the rotation matrix and key control joints, forward kinematics is used to calculate the global pose of each joint of the human body, which includes at least the pose of the end of the arm and the pose of the end of the finger; Transform the arm end posture into the local coordinate system with the hip as the origin to obtain the arm posture information; Transform the finger end posture into the local coordinate system with the wrist as the origin to obtain the finger posture information; The arm posture information and the finger posture information are aggregated to obtain the posture information.

4. The robot joint control method based on motion capture equipment according to claim 3, characterized in that: The method of converting local rotation information into quaternion form and obtaining the pose quaternion is as follows: Step 11: Based on the conversion relationship between degrees and radians, the local rotation information is processed to calculate the posture radian data; Step 12: Process the attitude radian data according to the sine and cosine calculation formula to calculate the cosine and sine values ​​of each axis; Step 13: Based on the cosine and sine values ​​of each axis and the rotation order of each axis, calculate the pose quaternion.

5. The robot joint control method based on motion capture equipment according to claim 3, characterized in that: Based on the rotation matrix and key control joints, the method of using forward kinematics to calculate the global pose of each joint of the human body is as follows: Step 21, obtaining a rotation matrix, which at least includes a global rotation matrix of the hip joint and a local rotation matrix of the spinal joint; Step 22, multiplying the global rotation matrix by the local rotation matrix to obtain the global rotation matrix of the spinal joint; Step 23: Based on the global rotation matrix of the spinal joints and combined with the topological relationship of the key control joints, the global rotation matrix of the arm end is calculated to complete the rotation construction of the entire upper limb chain; Step 24: Based on the global rotation matrix, the global pose of each joint is calculated based on forward kinematics and the spatial offset vectors between joints.

6. The robot joint control method based on motion capture equipment according to claim 5, characterized in that: The method to transform the arm end posture into the local coordinate system with the hip as the origin to obtain the arm posture information is as follows: According to the global position difference between the arm and the hip, the offset from the arm joint to the hip joint in the global coordinate system is obtained; Based on the offset and the inverse transformation of the hip rotation matrix, the position information of the arm in the hip coordinate system is obtained; Multiply the arm rotation matrix by the inverse matrix of the hip rotation matrix to obtain the arm pose matrix in the hip coordinate system; The arm position information is combined with the arm posture matrix to obtain the arm posture information and complete the transformation of the arm end posture.

7. The robot joint control method based on motion capture equipment according to claim 1, characterized in that: Step 4: Use the previously created solution optimization module to perform inverse posture analysis on the human joints based on the standard posture information. The method to obtain the target joint control information required for the robot execution is as follows: Obtain standard posture information, which includes the arm end posture and finger end posture in the robot body coordinates; Convert the arm end pose into Euclidean group data; The objective function is constructed with the current robot state as the initial value and the target posture as the optimization target; Based on the Euclidean group data, the objective function is solved to obtain the joint angle solution that meets the constraints; Obtain the degree of freedom information of the robot's fingers and the current posture of the fingers; According to the degree of freedom information, the finger joint limit conditions and inter-finger coordination constraint information are determined; Based on the finger end position, current finger posture, joint limit conditions and inter-finger coordination constraint information, the five-finger angle control information is obtained; The joint angle solution and the five-finger angle control information are coupled to obtain the target joint control information required for robot execution.

8. The robot joint control method based on motion capture equipment according to claim 1, characterized in that: It also includes a control and delivery module; The control issuing module is used to generate control instructions and control the movement of robot joints. The method is as follows: Generate the control instructions required for the robot to execute according to the target joint control information; Send control commands to the robot at a frequency of 500Hz through the communication protocol to control the robot in real time; When the robot communication is interrupted or abnormal, the control instructions of the previous frame are used to maintain the continuity of the robot's movement; When no new control instructions are received within the set time, the robot is controlled to enter a safe state.

9. A robot joint control system based on a motion capture device, characterized by: Including motion capture module, data processing module and solution optimization module; The motion capture module is used to receive the rotation posture data of the human body joints collected by the motion capture device and store the rotation posture data in the shared memory; The data processing module is used to retrieve the rotation posture data from the shared memory, convert the rotation posture data into a unified rotation matrix, and calculate the global posture of each joint in combination with the human body structure information to obtain the posture information; the posture information is then converted from the motion capture device coordinate system to the robot operating system coordinate system and the robot body coordinate system in sequence to obtain the standard posture information of the robot; The method to obtain the standard posture information of the robot is as follows: According to the difference between the coordinate system of the motion capture device and the coordinate system of the robot operating system, a transformation matrix between the coordinate system of the robot operating system and the coordinate system of the motion capture device is constructed; Based on the transformation matrix, the pose information output by the motion capture device is converted into the arm end pose in the robot operating system coordinate system; The robot body coordinate system is a local coordinate system, which includes the left arm coordinate system and the right arm coordinate system; Constructing a first rotation transformation matrix and a second rotation transformation matrix according to the difference between the robot operating system coordinate system and the left arm coordinate system; According to the difference between the robot operating system coordinate system and the right arm coordinate system, the rotation transformation matrix 1 and the rotation transformation matrix 2 are constructed; Based on the first rotation transformation matrix, the second rotation transformation matrix, the rotation transformation matrix 1 and the rotation transformation matrix 2 and the end arm posture in the robot operating system coordinate system, the end arm posture in the robot body coordinate system is calculated. Taking the wrist joint as the reference origin, the finger end pose coordinate system is transformed and appropriately scaled based on the finger length to obtain the finger end pose in the robot body coordinate system. The arm end pose and finger end pose in the robot body coordinates are coupled to obtain the robot's pose standard information; The solution optimization module is used to perform posture inverse analysis on human joints based on standard posture information to obtain the target joint control information required for robot execution.

Images