Mobile terminal-based humanoid robot real-time action following system and method

Abstract

The present disclosure provides a mobile terminal-based humanoid robot real-time action following system and method, which solves the problems of high cost, poor adaptability, easy jitter in pose calculation, and lack of safety protection of traditional motion capture equipment, and comprises mobile terminal acquisition and transmission, human three-dimensional pose analysis, quaternion SMPLX calculation, joint mapping, and servo issuance units, and additionally sets up a confidence filtering, safety protection, and data sedimentation module. The mobile terminal transmits images through WebSocket and eliminates lag frames; three-dimensional skeletons are solved relying on human priori, and quaternion calculation is used to avoid gimbal lock and is smoothed through adaptive SLERP; multiple robots are adapted through JSON configuration and amplitude limiting formula; joint angles are issued with low latency through UDP, and in abnormal working conditions, the system automatically slows down to a safe posture, and complete motion data sets can also be sedimented for algorithm training. The present disclosure can use an Android phone to collect human actions with light weight, and can also consider real-time following, device safety, and offline data generation, and is suitable for humanoid robot teleoperation and imitation learning.

Description

Technical Field

[0001] This disclosure relates to the field of machine vision, and in particular to a system and control method for realizing real-time motion following and batch motion data generation of a humanoid robot by acquiring human images using an Android mobile device, transmitting them wirelessly via WebSocket, parsing human key points, performing monocular 3D reconstruction, calculating local pose using quaternions, converting SMSLX parameters, and adaptively mapping robot joints. Background Technology

[0002] Currently, humanoid robot motion replication, teleoperation, and motion strategy training mainly rely on inertial motion capture suits, optical motion capture booths, or fixed depth cameras to collect human motion data. These specialized acquisition devices have inherent drawbacks such as high deployment barriers and poor mobility, making it impossible to achieve lightweight wireless data acquisition using ordinary consumer-grade devices. Wearable motion capture requires sensors to be deployed all over the body and repeated zero-point calibration, making the acquisition process cumbersome and prone to sensor drift over long periods of use. Optical motion capture booths are expensive, require fixed locations, and have complex calibration processes, making it difficult to quickly acquire human motion samples in large quantities in ordinary R&D scenarios. Neither of these methods supports mobile acquisition anytime, anywhere, and both suffer from the core pain points of high acquisition equipment cost and limited deployment.

[0003] Existing monocular vision motion capture solutions can only output 2D human keypoints, lacking stable monocular depth calculation logic and standardized skeletal topology transformation mechanisms. Ordinary monocular algorithms lack anthropometry prior support, cannot estimate the depth of keypoints in the head, neck, torso, and limbs layer by layer, and do not convert MediaPipe output keypoints into a unified OpenPose standard skeleton, missing virtual joints for the neck and pelvis. When directly driving the robot, skeletal constraints fail, easily leading to problems such as limb distortion, motion jitter, and joint overshooting, resulting in insufficient stability of 3D pose output.

[0004] Traditional posture transformation commonly uses Euler angles to represent skeletal rotation, which easily leads to problems such as gimbal lock, axial coupling, and discontinuous temporal interpolation when solving multi-degree-of-freedom joints such as the shoulder, spine, and neck. Euler angle rotation representation has singularities, poor smoothing of continuous motion interpolation, and cannot output standardized SMSLX human body axis angle parameters, making it difficult to unify the human posture representation format and providing standardized and continuous posture input for robot motion control.

[0005] Existing human-robot motion mapping algorithms on the market all use hard-coded methods to bind joint parameters of a single robot model, without setting configurable adaptive mapping logic. Different humanoid robots have significant differences in joint axis orientation, positive and negative rotation directions, zero position offset, motion limits, and left and right mirror relationships. Hard-coded solutions require modification of the underlying algorithm code for each robot model, resulting in extremely high porting and debugging costs and a lack of universal adaptability.

[0006] In addition, the existing motion capture control link lacks supporting mechanisms such as asynchronous frame buffering to control latency, attitude safety protection, and standardized dataset accumulation. It has not formed a complete closed-loop control system and cannot simultaneously meet the multiple requirements of real-time low-latency tracking, abnormal attitude safety protection, and offline batch motion data generation. Existing technologies cannot balance real-time performance, operational safety, and data reuse value.

[0007] In summary, existing motion capture solutions suffer from multiple drawbacks, including high acquisition costs, limited deployment, poor monocular 3D posture stability, gimbal lock in Euler angle rotation, difficulty in adapting to multiple robots, and lack of a complete real-time safety closed loop and data accumulation link. They cannot simultaneously meet the comprehensive needs of low-cost mobile acquisition, low-latency real-time tracking, universal adaptation to multiple robot models, safe operation, and batch generation of offline datasets.

[0008] Therefore, a motion-following system that is low-cost, has low calibration burden, can wirelessly move and collect data, can be controlled in real time, has safety protection, can batch accumulate training data, and can be adapted to multiple models of humanoid robots urgently needs to be developed. Summary of the Invention

[0009] The purpose of this disclosure is to provide a real-time motion-following system and method for humanoid robots based on a mobile terminal, which aims to solve at least one technical problem in the prior art.

[0010] The technical solution disclosed herein is:

[0011] A real-time motion following system for humanoid robots based on a mobile terminal includes: a mobile terminal image acquisition and transmission unit, a human three-dimensional posture analysis unit, a quaternion SMPLX posture calculation unit, a robot adaptive joint mapping unit, and a servo sending unit.

[0012] The mobile terminal image acquisition and transmission unit acquires the original image and sends the human body image frame to the server via WebSocket binary stream. It also filters backlogged old frames by using a single frame overlay buffer and outputs the latest valid image frame.

[0013] The human body 3D pose analysis unit receives the latest valid image frame, extracts human body key points, converts standard skeleton topology, generates virtual joints, and then solves the 3D depth of key points based on anthropometry priors, outputting complete human skeleton data with 3D coordinates.

[0014] The quaternion SMPLX attitude calculation unit receives the complete human skeleton data, solves the local rotation quaternion based on the parent and child joint bone vectors and the SMPLX zero-position reference skeleton, and uses orthogonal stabilizing axis compensation when the bone vectors are reversed to obtain the local rotation quaternion.

[0015] The robot adaptive joint mapping unit uses the local rotation quaternion data to complete joint transformation and outputs the robot's executable joint angles.

[0016] The servo sending unit receives the robot's executable joint angles and sends them to the humanoid robot control terminal with low latency, driving the robot to complete the motion following.

[0017] The aforementioned real-time motion tracking system for humanoid robots based on mobile terminals further includes: a confidence-weighted time-series filtering unit;

[0018] The confidence-weighted time-series filtering unit receives the local rotation quaternion data, performs spherical interpolation smoothing, and outputs the smoothed quaternion.

[0019] The robot adaptive joint mapping unit receives the smoothed quaternion and converts it into SMSLX posture parameters to complete the joint transformation based on the JSON configuration file and output the robot's executable joint angles.

[0020] The conversion yields the SMPLX attitude parameters, which are used to complete joint transformation based on a JSON configuration file. The mapping formula used in outputting the robot's executable joint angles is as follows:

[0021] ;

[0022] in, For the target joint angle of the robot; The positive and negative coefficients for the joint direction; This is the motion proportionality coefficient; Output attitude angles for SMPLX; This represents the robot's zero-position offset. These are the minimum and maximum safety limits for the joint, respectively. This is the amplitude limiting function.

[0023] The process of performing spherical interpolation smoothing and outputting the smoothed quaternion includes:

[0024] For consecutive frames, confidence-adaptive spherical interpolation is used for smoothing: ;

[0025] in, for The quaternion after time-smoothing; The interpolation coefficients are determined by the current keypoint confidence, joint angular velocity, network latency, and historical jitter amplitude. for The quaternion after time-smoothing; for The original observation quaternion at each moment; It is a spherical linear interpolation function.

[0026] The process of using the local rotation quaternion data to complete joint transformation and output the robot's executable joint angles includes: using the following formula to smooth the quaternions... Convert to axis-angle vector:

[0027] ;

[0028] ;

[0029] ;

[0030] in, This represents the total rotation angle corresponding to the quaternion; The real part of the quaternion; This represents the imaginary part of the quaternion; The normalized three-dimensional vector of the rotation axis; The required axis angle parameters for SMPLX.

[0031] The quaternion SMPLX attitude calculation unit receives the complete human skeleton data, solves the local rotation quaternion based on the parent-child joint bone vectors and the SMPLX zero-position reference skeleton, and uses orthogonal stabilizing axis compensation when the bone vectors are reversed to obtain the local rotation quaternion, including:

[0032] Let the position of the parent joint be... The sub-joint position is Current frame bone orientation: ;

[0033] Let the reference direction of the corresponding skeleton in the SMPLX zero-position pose be... The method for obtaining the rotation axis and the quaternion local rotation quaternion data is as follows:

[0034] , ; ;

[0035] in, The rotation axis is a three-dimensional vector; Corresponding to the rotation axis The components of the axis; The cross product of vectors is represented by ·; the dot product of vectors is represented by ·. The real part of the quaternion; It is a vector normalization function; This is the normalized local rotation quaternion.

[0036] The JSON configuration file configures one or more of the following: joint sign, scaling factor, zero offset, joint amplitude limit, axis interchange, and left / right mirroring, to adapt to humanoid robots with different servo motor layouts.

[0037] Establish an index correspondence between SMSLX parameters and robot joints for the waist, neck, shoulder, elbow, and wrist, and set a maximum angular velocity limit for the mapped joint angles.

[0038] The aforementioned mobile-based humanoid robot real-time motion tracking system also includes: a safety posture protection unit;

[0039] The safe posture protection unit acquires the mapping data output by the robot adaptive joint mapping unit;

[0040] When human body detection fails, key point confidence is insufficient, network interruption occurs, or posture change exceeds the threshold, the safe posture protection unit outputs preset safe standing posture parameters or slowly returns the joint angle of the previous frame to a safe standing posture before sending it to the servo sending unit.

[0041] The servo sending unit uses the UDP protocol to transmit float32 binary joint parameters, reducing communication overhead and transmission latency.

[0042] The aforementioned mobile-based humanoid robot real-time motion tracking system also includes: a dataset accumulation unit;

[0043] The dataset accumulation unit synchronously collects original images, human key points, SMPLX posture parameters, robot executable joint angles, timestamps, and mapping configuration parameters to generate a training dataset for robot motion control algorithms.

[0044] A mobile-based real-time motion tracking method for humanoid robots, based on the aforementioned system, includes:

[0045] Human image frames are sent to the server via WebSocket binary streams, and backlogged old frames are filtered by single-frame overwrite buffers to output the latest valid image frames;

[0046] Based on the latest valid image frame, extract human key points and convert standard skeleton topology to generate virtual joints. Then, rely on anthropometry priors to solve the three-dimensional depth of key points and output complete human skeleton data with three-dimensional coordinates.

[0047] Based on the complete human skeleton data, the local rotation quaternion is solved by the parent-child joint bone vector and the SMPLX zero-position reference skeleton. When the bone vector is reversed, orthogonal stable axis compensation is used to obtain the local rotation quaternion.

[0048] The joint transformation is completed using the local rotation quaternion data, and the operable joint angles of the robot are output.

[0049] Based on the robot's executable joint angles, the data is sent to the humanoid robot control terminal with low latency to drive the robot to complete the motion following.

[0050] The beneficial effects of this disclosure include at least the following:

[0051] The system described in this disclosure achieves wearable wireless motion acquisition using only an Android mobile application (Android APP) image acquisition unit and a WebSocket wireless transmission unit, eliminating the need for expensive motion capture sheds and inertial sensors. Calibration is simple and deployment is flexible. Simultaneously, this disclosure configures a human keypoint parsing unit, a joint format conversion unit, and a 3D depth solving unit, unifying the OpenPose standard skeletal topology and supplementing virtual joints. Depth is estimated layer by layer based on anthropometry priors to obtain stable and complete 3D skeletal data. Furthermore, this disclosure uses a quaternion-based SMSLX posture calculation unit with confidence-adaptive SLERP spherical interpolation to completely avoid rotational singularities and output continuous and standardized SMSLX 63-dimensional standard posture parameters. A robot joint adaptive mapping unit is also included, flexibly adjusting joint direction, proportion, offset, and limits using a JSON configuration file, adapting to multiple humanoid robots without modifying the underlying algorithm. Additionally, an asynchronous reception and single-frame overlay buffer unit is added to reduce transmission latency. A safe posture protection unit outputs a safe standing posture and limits joint impact in cases of human loss, network interruption, or sudden posture changes, and transmits this information via UDP. The servo-based control unit enables low-latency control, while the dataset accumulation unit provides complete archives of images, key points, poses, and time-series data, enabling simultaneous real-time human-machine tracking and offline algorithm training, significantly improving overall practicality. Attached Figure Description

[0052] Figure 1: Overall architecture diagram of the humanoid robot motion following system based on Android APP-WebSocket and SMPLX;

[0053] Figure 2: Flowchart of quaternion pose transformation and robot joint mapping;

[0054] Figure 3: A screenshot of the actual data collection and visualization of the system's output directory, including FPS, effective joints, valid SMSLX parameters, UDP sending status, and OpenPose human skeleton;

[0055] Figure 4: Schematic diagram of model input and output data link. The left side is the acquisition screenshot, and the right side is the corresponding frame's raw JSON data, including key points, SMPLX parameters, and robot joint angles.

[0056] Figure 5 The flowchart of the real-time motion following method for humanoid robots based on mobile terminals described in this disclosure. Detailed Implementation

[0057] The following describes specific embodiments of the present disclosure to enable those skilled in the art to understand the disclosure. However, it should be understood that the disclosure is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the disclosure as defined and determined by the appended claims. All disclosed creations utilizing the concept of the present disclosure are protected. Specific Implementation Example 1:

[0059] This disclosure provides an embodiment:

[0060] In this embodiment, the mobile device is exemplified by an Android smartphone.

[0061] As shown in Figure 1, a real-time motion-following system for humanoid robots based on a mobile terminal mainly includes the following hardware:

[0062] Android smartphones: Mobile image acquisition and transmission unit, equipped with a matching acquisition APP, which captures human images through the built-in camera and supports timestamp, device metadata appending and WebSocket binary stream transmission;

[0063] Local server computer: Human 3D pose analysis unit, quaternion SMPLX pose calculation unit, robot adaptive joint mapping unit containing joint mapping program, and servo sending unit that enables dual-coroutine asynchronous image reception and processing.

[0064] Humanoid robot controller: Receives float32 binary joint angles via UDP over a local area network to drive the movement of each servo motor;

[0065] Local area network switch: Enables interconnection and interoperability between mobile phones, servers, and robot controllers.

[0066] The specific quaternion attitude transformation process is as follows:

[0067] Let the position of the parent joint be... The sub-joint position is Current frame bone orientation: ;

[0068] Let the reference direction of the corresponding skeleton in the SMPLX zero-position pose be... The method for obtaining the rotation axis and the quaternion local rotation quaternion data is as follows:

[0069] , ; ;

[0070] in, The rotation axis is a three-dimensional vector; Corresponding to the rotation axis The components of the axis; The cross product of vectors is represented by ·; the dot product of vectors is represented by ·. The real part of the quaternion; It is a vector normalization function; This is the normalized local rotation quaternion. When and When approaching the opposite direction, select one that is the same as... An orthogonal stabilizing axis is used as a 180-degree rotation axis for orthogonal stabilizing axis compensation to avoid numerical instability.

[0071] For consecutive frames, confidence-adaptive spherical interpolation is used for smoothing: ;

[0072] in, for The quaternion after time-smoothing; The interpolation coefficients are determined by the current keypoint confidence, joint angular velocity, network latency, and historical jitter amplitude. for The quaternion after time-smoothing; for The original observation quaternion at each moment; It is a spherical linear interpolation function. It increases with higher confidence and faster changes in the action. The confidence level decreases when there is low confidence or when mutations occur. This achieves a dynamic balance between response speed and stability. Ultimately, the smoothed quaternion... Convert to axis-angle vector. Preferably,

[0073] ;

[0074] in, The quaternion angular velocity is observed at time t; These are the upper and lower limits of the interpolation coefficients, respectively. Current joint key point confidence; The current network transmission delay is represented by r; the confidence level adjustment index is represented by r. These are static baseline coefficients; This is the network latency suppression coefficient; This represents the historical jitter suppression coefficient. Let t be the current network transmission delay. This represents the amplitude of historical posture jitter within the sliding time window.

[0075] Preferably, the smoothed quaternion is obtained using the following formula. Convert to axis-angle vector:

[0076] ;

[0077] ;

[0078] ;

[0079] in, This represents the total rotation angle corresponding to the quaternion; The real part of the quaternion; This represents the imaginary part of the quaternion; The normalized three-dimensional vector of the rotation axis; The required axis angle parameters for SMPLX.

[0080] The following is an example of the adaptive mapping process between SMPLX and robot joints:

[0081] The joint drive index mapping relationships are as follows:

[0082] Index 7 corresponds to the waist pitch degree of freedom. Index 8 corresponds to the waist yaw degree of freedom. Index 35 corresponds to the neck yaw degree of freedom. Index 43 corresponds to the neck pitch degree of freedom. Indices 45, 46, and 47 correspond to the left shoulder roll, pitch, and yaw degrees of freedom, respectively. , , Indices 48, 49, and 50 correspond to the right shoulder roll, pitch, and yaw degrees of freedom, respectively. , , Index 52 corresponds to the left elbow pitch degree of freedom. Index 55 corresponds to the right elbow pitch degree of freedom. Index 58 corresponds to the left wrist pitch degree of freedom. Index 61 corresponds to the right wrist pitch degree of freedom. .

[0083] Each joint performs the following constraint mapping, i.e., the maximum angular velocity limit of the joint:

[0084] ;

[0085] in, For the target joint angle of the robot; The positive and negative coefficients for the joint direction; This is the motion proportionality coefficient; Output attitude angles for SMPLX; This represents the robot's zero-position offset. These are the minimum and maximum safety limits for the joint, respectively. This is the limiting function. For robots with camera mirroring, left and right arm swapping, or inconsistent pitch / roll / yaw axis definitions, axis swapping and left / right mirroring strategies can be enabled through the configuration file.

[0086] In specific implementation, such as Figures 1 to 3 The Android app collects human motion image frames with accompanying frame timestamps, resolution, and device orientation metadata, and sends them to the local server computer via WebSocket binary streams. The local server computer adopts an asynchronous separation architecture of receiving and processing coroutines, discarding backlogged old frames using a single-frame overwrite caching mechanism and outputting only the latest valid image frames. MediaPipe detection is performed on the latest image frames to extract 33 human keypoints and their corresponding confidence scores. The MediaPipe 33 points are converted to the OpenPose 25-point standard topology, and virtual joints for the neck and pelvis are generated through interpolation of left and right shoulder and hip keypoints. Based on anthropometry priors for shoulder and hip widths and a pinhole camera model, the 3D depth of keypoints in the head, neck, torso, and upper and lower limbs is solved segmentally, outputting complete 3D skeletal coordinate data. All parent-child joint bone pairs are traversed, and local rotation quaternions are solved using the SMPLX zero-position reference skeleton. Orthogonal stabilizing axis compensation is enabled when the bone vector is opposite to the reference direction to eliminate numerical calculation anomalies. Kalman filtering is used to smooth the keypoint coordinates, and adaptive SLERP is applied. The algorithm performs temporal smoothing on quaternions in consecutive frames, then converts the smoothed quaternions into 63-dimensional axis-angle attitude parameters according to the SMSLX standard. It reads the robot's JSON mapping configuration file, substitutes the mapping formulas to convert the SMSLX attitude parameters into target joint angles of the robot, and performs joint amplitude limiting and maximum joint angular velocity limiting. The safety attitude protection unit verifies the legality of the joint data and switches to a safe standing posture slow-down strategy in abnormal working conditions. After the verification is passed, the data is sent to the humanoid robot controller via UDP in float32 binary format with low latency to drive the robot to replicate human movements. The original images, human key points, SMSLX attitude parameters, robot joint angles, timestamps, and mapping configurations are synchronously stored in the local output directory.

[0087] The following is an excerpt from the robot's JSON mapping configuration file:

[0088] {

[0089] "description": "Arm mapping profile - Add offset support - Add wrist calculation method option",

[0090] "version": "1.14",

[0091] "swap_left_right": false,

[0092] "neck": {

[0093] "neck_yaw_sign": -1,

[0094] "neck_yaw_scale": 6.0,

[0095] "neck_yaw_offset": 0.0,

[0096] "neck_pitch_sign": 1,

[0097] "neck_pitch_scale": 2.0,

[0098] "neck_pitch_offset": -0.68,

[0099] "swap_yaw_pitch": false,

[0100] "description": "Fixed: neck_pitch_sign changed to -1, neck_pitch_scale changed to 1.0. offset is the zero-point offset (in radians)."

[0101] },

[0102] It needs to be clarified that mapping the 33 human body keypoints of MediaPipe to the OpenPose 25-point standard topology using a fixed index, and generating virtual joints of the neck and pelvis through interpolation of the left and right shoulder and left and right hip keypoints, are common and well-known preprocessing methods in the industry.

[0103] Meanwhile, image frames output from Android phones are continuously pushed to the server, with the caching unit retaining only the latest frames to eliminate image lag caused by network stuttering; MediaPipe automatically converts to standard OpenPose topology after outputting 33 points, supplementing the neck and pelvic virtual joints to complete the full-body skeleton; monocular depth is assigned based on prior layering of human shoulder width, allowing z-axis coordinates to be obtained without a depth camera; the quaternion calculation process automatically switches to orthogonal rotation axis when the arm is completely reversed to avoid calculation crashes; SLERP interpolation coefficients are dynamically adjusted in real time according to the confidence of key points, resulting in smooth responses to rapid movements such as raising the arm and turning around, and smooth, non-jumping movements in occluded and low-confidence scenes; the JSON configuration file pre-writes the robot's joint directions, strokes, and zero-position offsets, allowing for direct adaptation without modifying the algorithm code; when a human body is detected leaving the frame, the system automatically outputs a safe standing posture, and the robot slowly returns to its normal position without violent impact; real-time data acquisition is automatically saved, with screenshots corresponding one-to-one with JSON data.

[0104] Actual operating parameters: real-time processing frame rate 15.33 FPS, UDP frame transmission count 255, robot communication address 192.168.110.214:5005; 19 valid OpenPose keypoints detected per frame, 35 non-zero valid parameters in the 63-dimensional pose parameters of SMPLX; Example of output robot joint angles: =-88.9°, =-53.3°, =-0.9°, =38.8° =-36.8°.

[0105] For OpenPose key points, this embodiment provides an example: the information for key points 8 and 9 is as follows:

[0106] "z": -143.18140983581543,

[0107] "confidence": 0.9778139591217041

[0108] },

[0109] "8": {

[0110] "x": 228.55456352233887,

[0111] "y": 350.22497177124023,

[0112] "z": -0.11009573936462402,

[0113] "confidence": 0.95

[0114] },

[0115] "9": {

[0116] "x": 207.23851203918457,

[0117] "y": 351.1664581298828,

[0118] "z": -14.003545939922333,

[0119] "confidence": 0.9999528527259827

[0120] }

[0121] Based on the hardware of Specific Implementation 1, the system function is switched to offline dataset generation mode, as shown in the data link flow in Figure 4. The offline batch motion data generation method includes: importing locally acquired human motion videos or publicly available human motion materials into the system; the system executes a complete pose analysis and processing link frame by frame; for image frames with missing key points or occlusion, confidence interpolation, bone length constraints, and quaternion smoothing compensation for missing pose information are used; three types of standardized data are output in batches: OpenPose 25-point key point JSON file, 63-dimensional SMPLX pose_params temporal array, and mapped robot joint angle sequence; all data is automatically archived to the local output directory, along with corresponding frame screenshots (i.e., attached). Figure 3 The format shown), timestamp, and mapping configuration copy can be directly used for robot motion retrieval, posture clustering, and offline training of imitation learning control algorithms.

[0122] Offline processing eliminates the need for real-time streaming from Android phones; instead, it directly reads local video files and parses them frame by frame. For low-confidence frames of key points caused by screen occlusion or human body out-of-bounds movements, it uses SLERP interpolation with quaternions from preceding and following frames to complete the pose and avoid action sequence breaks. The batch-generated SMPLEX pose parameters have a uniform format and can be directly imported into motion imitation training frameworks. It automatically creates folders to archive images, key points, and robot joint timing data according to video segments, eliminating the need for manual organization and significantly reducing the workload of dataset construction.

[0123] Based on the hardware in Specific Implementation 1, the monocular mobile image acquisition and transmission unit can be replaced with a binocular camera or a depth camera to build a high-precision motion acquisition system. The binocular or depth camera outputs high-precision 3D keypoint coordinates as input to the human body 3D pose analysis unit, achieving high-precision motion acquisition. This implementation is designed for high-precision industrial humanoid robots and scientific motion capture scenarios, requiring only the replacement of the front-end depth acquisition hardware to achieve high-precision motion acquisition without the need for redeveloping algorithms.

[0124] Furthermore, the system described in this embodiment can also support real-time mode and offline mode:

[0125] Real-time mode: The Android app uploads images via WebSocket, and the server parses them in real time and drives the robot via UDP.

[0126] Offline mode: Import images, videos, or publicly available motion data to batch generate OpenPose 25-point and SMSLX 63-dimensional pose parameters and robot joint angle sequences for motion library construction and algorithm training.

[0127] Safety is also a crucial concern when the robot is in operation. In this embodiment, key point confidence thresholds and safe speed thresholds can be set according to actual conditions. When human detection fails, key point confidence falls below the threshold, network is interrupted, or the pose angle change exceeds the safe speed threshold, the system does not send abnormal poses but instead executes the following safety strategy:

[0128] 1. Output preset safe standing posture parameters;

[0129] 2. Gradually return the joint angle from the previous frame to center.

[0130] 3. Limit the maximum angular velocity and maximum angular acceleration in a single cycle;

[0131] 4. Implement independent amplitude limiting for high-risk robot joints such as the shoulder, elbow, wrist, neck, and waist;

[0132] 5. Record abnormal frames for offline analysis, but do not enter the robot servo delivery queue. Specific Implementation Example 2:

[0134] This disclosure provides an embodiment:

[0135] like Figure 5A real-time motion-following method for humanoid robots based on a mobile terminal, based on the system described in Specific Embodiment 1, includes: sending human image frames to the server via WebSocket binary stream, filtering backlogged old frames using a single-frame overlay buffer, and outputting the latest valid image frame; extracting human key points and converting standard skeletal topology based on the latest valid image frame, generating virtual joints, and then solving the three-dimensional depth of the key points based on anthropometry priors to output complete human skeletal data with three-dimensional coordinates; solving local rotation quaternions based on the complete human skeletal data, using parent-child joint bone vectors and the SMPLX zero-position reference skeleton, and using orthogonal stable axis compensation when the bone vectors are reversed to obtain local rotation quaternions; using the local rotation quaternion data to complete joint transformation and output the robot's executable joint angles; and sending the robot's executable joint angles to the humanoid robot control terminal with low latency to drive the robot to complete motion following. Specific Implementation Example 3:

[0137] An electronic device includes: a storage medium and a processing unit; wherein the storage medium is used to store a computer program, and the processing unit exchanges data with the storage medium for executing the computer program during humanoid robot motion tracking detection, performing all the steps of the method as described in Specific Embodiment 2. Specific Implementation Example 4:

[0139] A computer-readable storage medium storing a computer program; when the computer program is run, it performs the steps of the method as described in Specific Embodiment 2.

[0140] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, etc., or any suitable combination thereof.

[0141] The above disclosure only discloses a few specific implementation scenarios. However, this disclosure is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this disclosure.

Claims

1. A real-time motion-following system for humanoid robots based on a mobile terminal, characterized in that, include: The system includes a mobile image acquisition and transmission unit, a human 3D pose analysis unit, a quaternion SMPLX pose calculation unit, a robot adaptive joint mapping unit, and a servo sending unit. The mobile image acquisition and transmission unit acquires the original image and sends the human image frame to the server via WebSocket binary stream. It also filters backlogged old frames by using a single-frame overlay buffer and outputs the latest valid image frame. The human body 3D pose analysis unit receives the latest valid image frame, extracts human body key points, converts standard skeleton topology, generates virtual joints, and then solves the 3D depth of key points based on anthropometry priors, outputting complete human skeleton data with 3D coordinates. The quaternion SMPLX attitude calculation unit receives the complete human skeleton data, solves the local rotation quaternion based on the parent and child joint bone vectors and the SMPLX zero-position reference skeleton, and uses orthogonal stabilizing axis compensation when the bone vectors are reversed to obtain the local rotation quaternion. The robot adaptive joint mapping unit uses the local rotation quaternion data to complete joint transformation and outputs the robot's executable joint angles. The servo sending unit receives the robot's executable joint angles and sends them to the humanoid robot control terminal with low latency, driving the robot to complete the motion following.

2. The real-time motion following system for humanoid robots based on a mobile terminal according to claim 1, characterized in that, It also includes: a confidence-weighted temporal filtering unit, which is disposed between the quaternion SMPLX attitude calculation unit and the robot adaptive joint mapping unit; The confidence-weighted time-series filtering unit receives the local rotation quaternion data, performs spherical interpolation smoothing, and outputs the smoothed quaternion. The robot adaptive joint mapping unit receives the smoothed quaternion and converts it into SMSLX posture parameters to complete the joint transformation based on the JSON configuration file and output the robot's executable joint angles.

3. The real-time motion following system for humanoid robots based on a mobile terminal according to claim 2, characterized in that, The conversion yields the SMPLX attitude parameters, which are used to complete joint transformation based on a JSON configuration file. The mapping formula used in outputting the robot's executable joint angles is as follows: ； in, For the target joint angle of the robot; The positive and negative coefficients for the joint direction; This is the motion proportionality coefficient; Output attitude angles for SMPLX; This is the robot's zero-position offset, used to compensate for the robot's zero position. These are the minimum and maximum safety limits for the joint, respectively. This is the amplitude limiting function.

4. The real-time motion following system for humanoid robots based on a mobile terminal according to claim 2, characterized in that, The process of performing spherical interpolation smoothing and outputting the smoothed quaternion includes: For consecutive frames, confidence-adaptive spherical interpolation is used for smoothing: ; in, for The quaternion after time-smoothing; The interpolation coefficients at time t are determined by the current keypoint confidence, joint angular velocity, network latency, and historical jitter amplitude. for The quaternion after time-smoothing; for The original observation quaternion at each moment; It is a spherical linear interpolation function.

5. The real-time motion following system for humanoid robots based on a mobile terminal according to claim 1, characterized in that, The process of using the local rotation quaternion data to complete joint transformation and output the robot's executable joint angles includes: using the following formula to smooth the quaternions... Convert to axis-angle vector: ； ； ； in, This represents the total rotation angle corresponding to the quaternion; The real part of the quaternion; This represents the imaginary part of the quaternion; The normalized three-dimensional vector of the rotation axis; The required axis angle parameters for SMPLX.

6. The real-time motion following system for humanoid robots based on a mobile terminal according to claim 1, characterized in that, The quaternion SMPLX attitude calculation unit receives the complete human skeleton data, solves the local rotation quaternion based on the parent-child joint bone vectors and the SMPLX zero-position reference skeleton, and uses orthogonal stabilizing axis compensation when the bone vectors are reversed to obtain the local rotation quaternion, including: Let the position of the parent joint be... The sub-joint position is Current frame bone orientation: ; Let the reference direction of the corresponding skeleton in the SMPLX zero-position pose be... The method for obtaining the rotation axis and the quaternion local rotation quaternion data is as follows: , ; ; in, The rotation axis is a three-dimensional vector; Corresponding to the rotation axis The components of the axis; The cross product of vectors is represented by ·; the dot product of vectors is represented by ·. The real part of the quaternion; It is a vector normalization function; The normalized local rotation quaternion; pass and Directional comparison, utilizing and An orthogonal stabilizing axis is used as a 180-degree rotation axis for orthogonal stabilizing axis compensation to avoid numerical instability.

7. The real-time motion tracking system for humanoid robots based on a mobile terminal according to claim 1, characterized in that: The JSON configuration file configures one or more of the following: joint sign, scaling factor, zero offset, joint amplitude limit, axis interchange, and left / right mirroring, to adapt to humanoid robots with different servo motor layouts. Establish an index correspondence between SMSLX parameters and robot joints for the waist, neck, shoulder, elbow, and wrist, and set a maximum angular velocity limit for the mapped joint angles.

8. The real-time motion following system for humanoid robots based on a mobile terminal according to claim 1, characterized in that, Also includes: Safety posture protection unit; The safe posture protection unit acquires the mapping data output by the robot adaptive joint mapping unit; When human body detection fails, key point confidence is insufficient, network is interrupted, or posture change exceeds the threshold, the safe posture protection unit outputs preset safe standing posture parameters or slowly returns the joint angle of the previous frame to a safe standing posture before sending it to the servo sending unit. The servo sending unit uses the UDP protocol to transmit float32 binary joint parameters, reducing communication overhead and transmission latency.

9. The real-time motion following system for humanoid robots based on a mobile terminal according to claim 1, characterized in that, Also includes: Data set accumulation unit; The dataset accumulation unit synchronously collects original images, human key points, SMPLX posture parameters, robot executable joint angles, timestamps, and mapping configuration parameters to generate a training dataset for robot motion control algorithms.

10. A real-time motion tracking method for a humanoid robot based on a mobile terminal, based on the system described in claim 1, characterized in that, include: Human image frames are sent to the server via WebSocket binary streams, and backlogged old frames are filtered by single-frame overwrite buffers to output the latest valid image frames; Based on the latest valid image frame, extract human key points and convert standard skeleton topology to generate virtual joints. Then, rely on anthropometry priors to solve the three-dimensional depth of key points and output complete human skeleton data with three-dimensional coordinates. Based on the complete human skeleton data, the local rotation quaternion is solved by the parent-child joint bone vector and the SMPLX zero-position reference skeleton. When the bone vector is reversed, orthogonal stable axis compensation is used to obtain the local rotation quaternion. The joint transformation is completed using the local rotation quaternion data, and the operable joint angles of the robot are output. Based on the robot's executable joint angles, the data is sent to the humanoid robot control terminal with low latency to drive the robot to complete the motion following.

Images