Real-time mapping method from three-dimensional human body joint point coordinates to biomimetic robot joint angles

Abstract

The application discloses a real-time mapping method from three-dimensional human body joint point coordinates to bionic robot joint angles, and relates to the technical field of data mapping processing. The method comprises the following steps: collecting human motion images in real time, extracting three-dimensional joint point coordinates and preprocessing; calculating a motion aggressiveness index; constructing a nonlinear mapping function containing a position item and a velocity item, dynamically adjusting the weight to obtain a robot joint angle initial value; taking the initial value as a starting point, constructing a target function containing a terminal tracking error item, an initial value traction item and a joint angle limit penalty item, and optimizing and solving an optimal joint angle; performing feedforward compensation based on angular velocity prediction and adaptive compensation gain to generate a final angle instruction; and driving the robot to reproduce human motion. Through dynamic weighted nonlinear mapping, adaptive traction optimization and online learning feedforward compensation, the application realizes high-precision, low-delay and smooth natural motion transfer from human to robot.

Description

Technical Field

[0001] This invention relates to the field of data mapping and processing technology, specifically to a real-time mapping method for three-dimensional human joint coordinates to bionic robot joint angles. Background Technology

[0002] With the gradual application of biomimetic robot technology in indoor environments such as homes and offices, desktop lightweight robots have attracted attention due to their small size, low cost, and user-friendly interaction. These robots typically have movable upper limb joints, enabling them to perform human-like movements such as waving, gesturing, and dancing to enhance the naturalness and fun of human-computer interaction. However, how to accurately map human upper limb movements to robot joints in real time, while ensuring smooth, safe, and low-latency movements, remains a challenge for current technology.

[0003] Existing human motion mapping methods mostly employ inverse kinematics or fixed-ratio linear mapping. Inverse kinematics methods are computationally complex and sensitive to joint noise; linear mapping ignores the nonlinear characteristics of human motion and the coupling relationship between position and velocity, easily leading to mapping distortion or joint angle exceeding limits during rapid movements. Furthermore, system delays exist from visual acquisition to control command transmission, resulting in significant lag in robot movements and disrupting the real-time interactive experience. Most existing methods are designed for industrial or service robots, failing to adequately consider the limited joint range of motion and lack of grasping requirements of desktop lightweight robots, thus lacking targeted adaptive design.

[0004] Therefore, there is an urgent need for a method specifically designed for upper limb motion mapping of small bionic robots, which can adapt to different movement speeds, smoothly handle joint limits, effectively compensate for delays, and achieve natural, real-time human-machine motion synchronization. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a real-time mapping method for three-dimensional human joint coordinates to bionic robot joint angles.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A real-time mapping method from 3D human joint coordinates to biomimetic robot joint angles includes the following steps:

[0008] Step S1: Collect human motion images in real time through a visual sensor, extract the three-dimensional spatial coordinates of each joint of the human body using a pose estimation algorithm, and filter and denoise the three-dimensional spatial coordinates and align the coordinates to obtain the coordinates of the human joints in the world coordinate system.

[0009] Step S2: Based on the coordinates of human joints, calculate the instantaneous velocity vector of each joint and calculate the average value of the acceleration modulus of all joints as an indicator of motion aggression.

[0010] Step S3: Construct a nonlinear mapping function containing position and velocity terms, dynamically adjust the weights of the position and velocity terms according to the motion aggression index, and map the coordinates of human joints to the initial values ​​of the robot's joint angles.

[0011] Step S4: Starting with the initial values ​​of the robot's joint angles, construct an objective function that includes an end-point tracking error term, an initial value traction term, and a joint angle limit penalty term. Obtain the optimal joint angle through optimization. The coefficient of the initial value traction term is adaptively adjusted according to the end-point tracking error of the previous frame.

[0012] Step S5: Calculate the angular velocity based on the optimal joint angle, perform angle prediction compensation by combining system delay time and adaptive compensation gain, and update the adaptive compensation gain online according to the deviation between the optimal joint angle of the previous frame and the actual feedback angle of the robot to generate the final angle command.

[0013] Step S6: Send the final angle command to the robot joint controller to drive the robot to reproduce human movement.

[0014] Furthermore, step S1 specifically includes the following steps:

[0015] Step S1.1: Use median filtering to denoise the original three-dimensional spatial coordinates, and use the median of the coordinates within the sliding window centered on the current frame as the filtered human joint coordinates.

[0016] Step S1.2: Linear interpolation is used to complete the coordinates of the joints that are still missing after filtering;

[0017] Step S1.3: Transform the coordinates in the camera coordinate system to the world coordinate system aligned with the robot's base coordinate system through rigid body transformation to obtain the coordinates of the human joint points in the world coordinate system.

[0018] Further, in step S2, the instantaneous velocity vector is equal to the difference between the position coordinates of the human joint points in the current frame and the previous frame divided by the frame interval time; the motion aggression index is equal to the average value of the difference between the velocity vectors of the current frame and the velocity vectors of the previous frame of all joint points divided by the frame interval time.

[0019] Furthermore, step S3 specifically includes the following steps:

[0020] Step S3.1: For each joint of the robot, a human reference position vector and a reference velocity vector associated with it are pre-set. The human reference position vector is composed of the coordinate difference between two human joints, and the reference velocity vector is composed of the velocity difference between the corresponding human joints.

[0021] Step S3.2: For each robot joint, project the human reference position vector onto the joint motion mapping direction vector, divide it by the length scaling factor, and then perform soft limiting through the hyperbolic tangent function to obtain the position mapping component.

[0022] Step S3.3: For each robot joint, project the human reference velocity vector onto the velocity mapping direction vector, divide by a factor equal to the magnitude of the reference velocity vector, and obtain the velocity mapping component.

[0023] Step S3.4: Calculate the position coefficient and velocity coefficient based on the motion aggression index. The sum of the position coefficient and velocity coefficient is 1. The higher the motion aggression, the larger the velocity coefficient and the smaller the position coefficient.

[0024] Step S3.5: Multiply the position mapping component by the position term coefficient, and add the velocity mapping component multiplied by the velocity term coefficient to obtain the initial values ​​of the joint angles of the robot.

[0025] Furthermore, step S4 specifically includes the following steps:

[0026] Step S4.1: Set several end effectors that the robot needs to track, and calculate the spatial position of each end effector at the current candidate angle using forward kinematics;

[0027] Step S4.2: Calculate the sum of squares of the differences between the positive kinematic position of each end effector and the position of the corresponding human end in the world coordinate system, and use it as the end tracking error term;

[0028] Step S4.3: Calculate the square of the difference between the current candidate angle and the initial value of each joint angle of the robot, multiply it by the adaptive traction coefficient to obtain the initial value traction term, wherein the adaptive traction coefficient is equal to the reference coefficient multiplied by the negative exponential decay function of the end tracking error of the previous frame.

[0029] Step S4.4: For each joint, determine whether the current candidate angle exceeds the physical limit range of the joint. If it does, calculate the square of the ratio of the excess amount to the range width; otherwise, calculate zero. Sum the square values ​​of all joints and multiply by a fixed penalty coefficient to obtain the joint angle limit penalty term.

[0030] Step S4.5: Summate the end tracking error term, the initial traction term, and the joint angle limit penalty term to form the objective function;

[0031] Step S4.6: Using the initial values ​​of the robot's joint angles as the initial solution, optimize the objective function using the quasi-Newton method or gradient descent method to obtain the optimal joint angles.

[0032] Furthermore, in step S4.3, the previous frame end tracking error is the end tracking error value calculated after the previous frame optimization is completed. When the error of the previous frame is larger, the adaptive traction coefficient is smaller.

[0033] Furthermore, step S5 specifically includes the following steps:

[0034] Step S5.1: Calculate the optimized angular velocity of each joint based on the optimal joint angles of the current frame and the previous frame. The optimized angular velocity is equal to the difference between the optimal joint angle of the current frame and the optimal joint angle of the previous frame divided by the frame interval time.

[0035] Step S5.2: Set the system delay time. For each joint, add the optimal joint angle to the adaptive compensation gain multiplied by the optimized angular velocity multiplied by the system delay time to obtain the compensated angle.

[0036] Step S5.3: Update the adaptive compensation gain according to the deviation sign between the optimal joint angle of the previous frame and the actual feedback angle of the robot, as well as the sign of the optimized angular velocity of the previous frame. If the deviation sign is the same as the angular velocity sign, increase the gain by a fixed step. If the signs are opposite, decrease the gain by a fixed step and limit the gain to between 0 and 1.

[0037] Step S5.4: Limit the compensated angle within the joint's physical limits using a truncation function to obtain the final angle command.

[0038] Furthermore, in step S6, the transmission frequency of the final angle command is not less than 30 Hz, and it is transmitted using CANopen, EtherCAT, or a serial bus communication protocol.

[0039] A storage medium storing instructions that, when read by a computer, cause the computer to execute any of the three-dimensional human joint coordinates to bionic robot joint angles real-time mapping methods described in the present invention.

[0040] An electronic device includes a processor and a storage medium, the processor executing instructions in the storage medium.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] 1. This invention uses a hyperbolic tangent function to softly limit the position term and suppress the nonlinear saturation of the velocity term, and dynamically adjusts the position / velocity weights according to the aggressiveness of the motion, so that the mapping result automatically adapts to different actions from slow to fast, and remains within the safe range of robot joint movement, avoiding over-limit and distortion.

[0043] 2. The initial traction coefficient in the optimized objective function constructed in this invention is adaptively decayed according to the tracking error of the previous frame, balancing tracking accuracy and motion smoothness. At the same time, the square-shaped limit penalty term effectively prevents joint angles from exceeding the limit, ensuring the safe operation of the robot.

[0044] 3. This invention is based on online updating of compensation gain according to angular velocity prediction and actual feedback deviation, realizing adaptive correction of system delay, significantly reducing motion lag, improving real-time tracking performance, and enhancing the immersive experience of human-computer interaction. Attached Figure Description

[0045] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0046] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating the nonlinear mapping process according to an embodiment of the present invention.

[0048] Figure 3 This is a flowchart of the feedforward compensation process according to an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] like Figure 1 As shown, the real-time mapping method from 3D human joint coordinates to biomimetic robot joint angles includes the following steps:

[0051] Step S1: Collect human motion images in real time through a visual sensor, extract the three-dimensional spatial coordinates of each joint of the human body using a pose estimation algorithm, and filter and denoise the three-dimensional spatial coordinates and align the coordinates to obtain the coordinates of the human joints in the world coordinate system.

[0052] Step S2: Based on the coordinates of human joints, calculate the instantaneous velocity vector of each joint and calculate the average value of the acceleration modulus of all joints as an indicator of motion aggression.

[0053] Step S3: Construct a nonlinear mapping function containing position and velocity terms, dynamically adjust the weights of the position and velocity terms according to the motion aggression index, and map the coordinates of human joints to the initial values ​​of the robot's joint angles.

[0054] Step S4: Starting with the initial values ​​of the robot's joint angles, construct an objective function that includes an end-point tracking error term, an initial value traction term, and a joint angle limit penalty term. Obtain the optimal joint angle through optimization. The coefficient of the initial value traction term is adaptively adjusted according to the end-point tracking error of the previous frame.

[0055] Step S5: Calculate the angular velocity based on the optimal joint angle, perform angle prediction compensation by combining system delay time and adaptive compensation gain, and update the adaptive compensation gain online according to the deviation between the optimal joint angle of the previous frame and the actual feedback angle of the robot to generate the final angle command.

[0056] Step S6: Send the final angle command to the robot joint controller to drive the robot to reproduce human movement.

[0057] Step S1 specifically includes the following steps:

[0058] Step S1.1: Use median filtering to denoise the original three-dimensional spatial coordinates, and use the median of the coordinates within the sliding window centered on the current frame as the filtered human joint coordinates.

[0059] Step S1.2: Linear interpolation is used to complete the coordinates of the joints that are still missing after filtering;

[0060] Step S1.3: Transform the coordinates in the camera coordinate system to the world coordinate system aligned with the robot's base coordinate system through rigid body transformation to obtain the coordinates of the human joint points in the world coordinate system.

[0061] Human motion images are acquired in real time using visual sensors such as depth cameras or binocular cameras. Deep learning-based pose estimation algorithms, such as OpenPose and MediaPipe, are used to extract the three-dimensional spatial coordinates of the joints of the human skeleton model from the images. The joints include key positions such as head, shoulder, elbow, wrist, hip, knee, and ankle.

[0062] The original 3D coordinates are denoised by median filtering. A sliding window is formed by taking several frames before and after the current frame as the center. The median of the three components of the coordinates of each joint point in each frame within the window is taken as the filtered coordinate value. For joint point coordinates that are still missing after filtering, such as those that fail to be detected due to occlusion, linear interpolation is used to fill in the missing coordinates.

[0063] After filtering and interpolation, the coordinates in the camera coordinate system are transformed to the world coordinate system through rigid body transformation. The rigid body transformation includes a 3×3 rotation matrix and a 3×1 translation vector, and finally the coordinates of the human joints in the world coordinate system of each frame are obtained.

[0064] In step S2, the instantaneous velocity vector is equal to the difference between the position coordinates of the human joint points in the current frame and the previous frame divided by the frame interval time; the motion aggression index is equal to the average value of the difference between the velocity vectors of the current frame and the velocity vectors of the previous frame of all joint points divided by the frame interval time.

[0065] Based on the preprocessed human joint coordinates, the instantaneous velocity vector of each joint is calculated. The velocity vector of the joint in frame t is equal to the difference between the position coordinates of the current frame and the previous frame divided by the frame interval time.

[0066] Simultaneously, the motion aggression index is calculated, which is defined as the average value of the acceleration magnitude of all relevant nodes. The acceleration magnitude is equal to the magnitude of the difference between the velocity vector of the current frame and the velocity vector of the previous frame divided by the frame interval time. The motion aggression index reflects the intensity of the current human movement. The larger the value, the faster the movement or the more rapid the change.

[0067] As auxiliary motion features, cosine similarity of velocity directions of adjacent joints is also extracted for motion smoothness constraints in subsequent steps.

[0068] like Figure 2 As shown, step S3 specifically includes the following steps:

[0069] Step S3.1: For each joint of the robot, a human reference position vector and a reference velocity vector associated with it are pre-set. The human reference position vector is composed of the coordinate difference between two human joints, and the reference velocity vector is composed of the velocity difference between the corresponding human joints.

[0070] Step S3.2: For each robot joint, project the human reference position vector onto the joint motion mapping direction vector, divide it by the length scaling factor, and then perform soft limiting through the hyperbolic tangent function to obtain the position mapping component.

[0071] Step S3.3: For each robot joint, project the human reference velocity vector onto the velocity mapping direction vector, divide by a factor equal to the magnitude of the reference velocity vector, and obtain the velocity mapping component.

[0072] Step S3.4: Calculate the position coefficient and velocity coefficient based on the motion aggression index. The sum of the position coefficient and velocity coefficient is 1. The higher the motion aggression, the larger the velocity coefficient and the smaller the position coefficient.

[0073] Step S3.5: Multiply the position mapping component by the position term coefficient, and add the velocity mapping component multiplied by the velocity term coefficient to obtain the initial values ​​of the joint angles of the robot.

[0074] A nonlinear mapping function containing position and velocity terms is constructed to map the coordinates of human joints to the initial values ​​of the angles of each robot joint. The mapping function is calculated independently for each robot joint, and a motion aggression index is introduced to dynamically adjust the weights of the position and velocity terms.

[0075] Suppose the robot has M active joints. The specific formula for calculating the initial angle estimate of the j-th joint is as follows:

[0076]

[0077] in, Let represent the initial angle estimate of the j-th robot joint. This represents the position coefficient of the j-th joint, with a value range of [0, 1]. Let represent the velocity coefficient of the j-th joint, with a value range of [0, 1], and satisfy . and The sum of is 1. This represents the motion mapping direction vector of the j-th joint. This represents the human reference position vector associated with the j-th joint. This represents the length scaling factor, in meters, and is the expected value of the corresponding skeletal segment length in the robot, used to normalize the human body scale. This represents the unit vector representing the velocity mapping direction of the j-th joint. This represents the human reference velocity vector associated with the j-th joint. The magnitude of the reference velocity vector;

[0078] The position and velocity coefficients are dynamically adjusted based on the calculated motion aggression index, and the specific formula is as follows:

[0079]

[0080] in, The reference value for the position term coefficient is typically taken as 0.7 to 0.9. Indicates the motion aggression of the current frame. This represents the aggression threshold constant; when the aggression of the motion is high, Reduce Increasing the speed information weight makes the mapping more sensitive to rapid motion; when the aggression is low, increasing the position information weight ensures steady-state accuracy.

[0081] Step S4 specifically includes the following steps:

[0082] Step S4.1: Set several end effectors that the robot needs to track, and calculate the spatial position of each end effector at the current candidate angle using forward kinematics;

[0083] Step S4.2: Calculate the sum of squares of the differences between the positive kinematic position of each end effector and the position of the corresponding human end in the world coordinate system, and use it as the end tracking error term;

[0084] Step S4.3: Calculate the square of the difference between the current candidate angle and the initial value of each joint angle of the robot, multiply it by the adaptive traction coefficient to obtain the initial traction term, wherein the adaptive traction coefficient is equal to the reference coefficient multiplied by the negative exponential decay function of the end tracking error of the previous frame.

[0085] Step S4.4: For each joint, determine whether the current candidate angle exceeds the physical limit range of the joint. If it does, calculate the square of the ratio of the excess amount to the width of the range; otherwise, calculate zero. Sum the square values ​​of all joints and multiply by a fixed penalty coefficient to obtain the joint angle limit penalty term.

[0086] Step S4.5: Summate the end tracking error term, the initial traction term, and the joint angle limit penalty term to form the objective function;

[0087] Step S4.6: Using the initial values ​​of the robot's joint angles as the initial solution, optimize the objective function using the quasi-Newton method or gradient descent method to obtain the optimal joint angles.

[0088] In step S4.3, the previous frame end tracking error is the end tracking error value calculated after the previous frame optimization is completed. When the error of the previous frame is larger, the adaptive traction coefficient is smaller.

[0089] The initial angle value obtained in step S3 Starting with the objective function, the optimal joint angle is obtained by solving an optimization problem. The objective function consists of three terms: end-point tracking error, initial traction, and joint angle limit penalty. The specific formula is as follows:

[0090]

[0091] in, Describe the objective function. Represents the optimization variable. , This indicates the number of human extremities that need to be tracked, typically including key points such as the palms of both hands and the ends of the forearms. This represents the three-dimensional spatial position of the e-th end effector calculated by the robot's forward kinematics. This indicates the three-dimensional position of the corresponding human extremity in the world coordinate system. Represents the square of the Euclidean distance. Indicates the adaptive traction coefficient. This represents the j-th variable in the optimization variables. This represents a fixed penalty coefficient. Let represent the minimum and maximum physical limits of the j-th robot joint, respectively. The limit penalty function is represented by the following formula:

[0092]

[0093] That is, only when the angle exceeds The penalty is only applied when the interval is wide, and the penalty amount is normalized relative to the interval width.

[0094] The formula for calculating the adaptive traction coefficient is:

[0095]

[0096] in, This represents the baseline value for the traction coefficient, typically ranging from 0.1 to 0.5. This represents the sum of the end-point tracking errors calculated after the optimization of the previous frame. This represents a scale parameter used to control the decay rate;

[0097] When the tracking error in the previous frame is large The weight of the initial traction term is automatically reduced, and the optimization process prioritizes reducing the end-tracking error; when the error is small, Increasing the initial value traction term makes the angle closer to the initial value, thus improving the smoothness of the motion;

[0098] The optimization solution uses the initial angle value obtained in step S3 as the initial solution and adopts a quasi-Newton method, such as BFGS or gradient descent with boundary constraints, for iterative solution. The number of iterations per frame does not exceed 5.

[0099] like Figure 3 As shown, step S5 specifically includes the following steps:

[0100] Step S5.1: Calculate the optimized angular velocity of each joint based on the optimal joint angles of the current frame and the previous frame. The optimized angular velocity is equal to the difference between the optimal joint angle of the current frame and the optimal joint angle of the previous frame divided by the frame interval time.

[0101] Step S5.2: Set the system delay time. For each joint, add the optimal joint angle to the adaptive compensation gain multiplied by the optimized angular velocity multiplied by the system delay time to obtain the compensated angle.

[0102] Step S5.3: Update the adaptive compensation gain according to the deviation sign between the optimal joint angle of the previous frame and the actual feedback angle of the robot, as well as the sign of the optimized angular velocity of the previous frame. If the deviation sign is the same as the angular velocity sign, increase the gain by a fixed step. If the signs are opposite, decrease the gain by a fixed step and limit the gain to between 0 and 1.

[0103] Step S5.4: Limit the compensated angle within the joint's physical limits using a truncation function to obtain the final angle command.

[0104] Since there is a fixed delay from image acquisition to angle calculation to command transmission, typically 30~80ms, directly sending the optimized angle will cause the robot's movements to lag behind the human body. Therefore, it is necessary to design an adaptive feedforward compensator based on angular velocity prediction to perform phase lead correction on the optimized angle.

[0105] Based on the optimal joint angles of the current frame and the previous frame, calculate the optimized angular velocity of each joint, using the following formula:

[0106]

[0107] in, This represents the optimized angular velocity of the j-th joint in frame t. This represents the optimal joint angle for the current frame. This represents the optimal joint angle in the previous frame. Indicates the time interval between adjacent frames;

[0108] The system delay is compensated using first-order linear prediction to obtain the compensated angle, and the specific formula is as follows:

[0109]

[0110] in, Indicates the angle after compensation. This represents the adaptive compensation gain of the j-th joint. Indicates the total system delay time;

[0111] The adaptive compensation gain is updated online based on the tracking deviation of the previous frame. The specific formula is as follows:

[0112]

[0113] in, This represents the adaptive compensation gain of the previous frame. This represents the gain step size, typically ranging from 0.02 to 0.05. Represents a symbolic function. This represents the tracking deviation in the previous frame, specifically the difference between the optimal joint angle in the previous frame and the angle value actually fed back by the robot. This represents the optimized angular velocity of the previous frame. This represents a cutoff function that restricts values ​​to between 0 and 1;

[0114] When the deviation and angular velocity have the same sign, it indicates that the actual feedback lags behind the expected direction of motion. Increasing the gain can enhance the feedforward compensation and help the robot catch up. When they have opposite signs, it indicates that the robot is ahead or the direction is mismatched. Decreasing the gain can avoid overcompensation.

[0115] The compensated angle is limited to the joint's physical limits using a truncation function. Within, the final angle command is generated.

[0116] In step S6, the transmission frequency of the final angle command is not less than 30 Hz, and it is transmitted using CANopen, EtherCAT or serial bus communication protocols.

[0117] The calculated final angle command is packaged into a data frame according to the robot communication protocol and sent to the robot motion controller through the communication interface at a transmission frequency of not less than 30Hz. The robot motion controller drives the servo motors of each joint to perform angle closed-loop control, so that the robot can reproduce the human body's motion posture in real time. Repeat steps S1 to S6 to realize continuous and real-time mapping and driving of three-dimensional human joint coordinates to bionic robot joint angles.

[0118] A storage medium storing instructions that, when read by a computer, cause the computer to execute any of the three-dimensional human joint coordinates to bionic robot joint angles real-time mapping methods described in the present invention.

[0119] An electronic device includes a processor and a storage medium, the processor executing instructions in the storage medium.

[0120] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0121] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.

Claims

1. A real-time mapping method from three-dimensional human joint coordinates to biomimetic robot joint angles, characterized in that, Includes the following steps: Step S1: Collect human motion images in real time through a visual sensor, extract the three-dimensional spatial coordinates of each joint of the human body using a pose estimation algorithm, and filter and denoise the three-dimensional spatial coordinates and align the coordinates to obtain the coordinates of the human joints in the world coordinate system. Step S2: Based on the coordinates of human joints, calculate the instantaneous velocity vector of each joint and calculate the average value of the acceleration magnitude of all joints as the motion aggression index. The instantaneous velocity vector is equal to the difference between the position coordinates of the human joints in the current frame and the previous frame divided by the frame interval time. The motion aggression index is equal to the average value of the difference between the velocity vectors of the current frame and the velocity vectors of the previous frame of all joints divided by the frame interval time. Step S3: Construct a nonlinear mapping function containing position and velocity terms, dynamically adjust the weights of the position and velocity terms according to the motion aggression index, and map the coordinates of human joints to the initial values ​​of the robot's joint angles. Step S4: Starting with the initial values ​​of the robot's joint angles, construct an objective function that includes an end-point tracking error term, an initial value traction term, and a joint angle limit penalty term. Obtain the optimal joint angle through optimization. The coefficient of the initial value traction term is adaptively adjusted according to the end-point tracking error of the previous frame. Step S5: Calculate the angular velocity based on the optimal joint angle, perform angle prediction compensation by combining system delay time and adaptive compensation gain, and update the adaptive compensation gain online according to the deviation between the optimal joint angle of the previous frame and the actual feedback angle of the robot to generate the final angle command. Step S6: Send the final angle command to the robot joint controller to drive the robot to reproduce human movement.

2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: Step S1.1: Use median filtering to denoise the original three-dimensional spatial coordinates, and use the median of the coordinates within the sliding window centered on the current frame as the filtered human joint coordinates. Step S1.2: Linear interpolation is used to complete the coordinates of the joints that are still missing after filtering; Step S1.3: Transform the coordinates in the camera coordinate system to the world coordinate system aligned with the robot's base coordinate system through rigid body transformation to obtain the coordinates of the human joint points in the world coordinate system.

3. The method according to claim 2, characterized in that, Step S3 specifically includes the following steps: Step S3.1: For each joint of the robot, a human reference position vector and a reference velocity vector associated with it are pre-set. The human reference position vector is composed of the coordinate difference between two human joints, and the reference velocity vector is composed of the velocity difference between the corresponding human joints. Step S3.2: For each robot joint, project the human reference position vector onto the joint motion mapping direction vector, divide it by the length scaling factor, and then perform soft limiting through the hyperbolic tangent function to obtain the position mapping component. Step S3.3: For each robot joint, project the human reference velocity vector onto the velocity mapping direction vector, divide by a factor equal to the magnitude of the reference velocity vector, and obtain the velocity mapping component. Step S3.4: Calculate the position coefficient and velocity coefficient based on the motion aggression index. The sum of the position coefficient and velocity coefficient is 1. The higher the motion aggression, the larger the velocity coefficient and the smaller the position coefficient. Step S3.5: Multiply the position mapping component by the position term coefficient, and add the velocity mapping component multiplied by the velocity term coefficient to obtain the initial values ​​of the joint angles of the robot.

4. The method according to claim 3, characterized in that, Step S4 specifically includes the following steps: Step S4.1: Set several end effectors that the robot needs to track, and calculate the spatial position of each end effector at the current candidate angle using forward kinematics; Step S4.2: Calculate the sum of squares of the differences between the positive kinematic position of each end effector and the position of the corresponding human end in the world coordinate system, and use it as the end tracking error term; Step S4.3: Calculate the square of the difference between the current candidate angle and the initial value of each joint angle of the robot, multiply it by the adaptive traction coefficient to obtain the initial value traction term, wherein the adaptive traction coefficient is equal to the reference coefficient multiplied by the negative exponential decay function of the end tracking error of the previous frame. Step S4.4: For each joint, determine whether the current candidate angle exceeds the physical limit range of the joint. If it does, calculate the square of the ratio of the excess amount to the range width; otherwise, calculate zero. Sum the square values ​​of all joints and multiply by a fixed penalty coefficient to obtain the joint angle limit penalty term. Step S4.5: Summate the end tracking error term, the initial traction term, and the joint angle limit penalty term to form the objective function; Step S4.6: Using the initial values ​​of the robot's joint angles as the initial solution, optimize the objective function using the quasi-Newton method or gradient descent method to obtain the optimal joint angles.

5. The method according to claim 4, characterized in that, In step S4.3, the previous frame end tracking error is the end tracking error value calculated after the previous frame optimization is completed. When the error of the previous frame is larger, the adaptive traction coefficient is smaller.

6. The method according to claim 5, characterized in that, Step S5 specifically includes the following steps: Step S5.1: Calculate the optimized angular velocity of each joint based on the optimal joint angles of the current frame and the previous frame. The optimized angular velocity is equal to the difference between the optimal joint angle of the current frame and the optimal joint angle of the previous frame divided by the frame interval time. Step S5.2: Set the system delay time. For each joint, add the optimal joint angle to the adaptive compensation gain multiplied by the optimized angular velocity multiplied by the system delay time to obtain the compensated angle. Step S5.3: Update the adaptive compensation gain according to the deviation sign between the optimal joint angle of the previous frame and the actual feedback angle of the robot, as well as the sign of the optimized angular velocity of the previous frame. If the deviation sign is the same as the angular velocity sign, increase the gain by a fixed step. If the signs are opposite, decrease the gain by a fixed step and limit the gain to between 0 and 1. Step S5.4: Limit the compensated angle within the joint's physical limits using a truncation function to obtain the final angle command.

7. The method according to claim 6, characterized in that, In step S6, the final angle command is transmitted at a frequency of no less than 30 Hz and is transmitted using CANopen, EtherCAT, or a serial bus communication protocol.

8. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute a real-time mapping method from three-dimensional human joint coordinates to bionic robot joint angles as described in any one of claims 1-7.

9. An electronic device, characterized in that, It includes a processor and the storage medium of claim 8, wherein the processor executes instructions in the storage medium.

Images