A method for visual teleoperation and teaching of an anthropomorphic robot and an anthropomorphic robot
By acquiring images of human motion and optimizing the solution of humanoid robot joint movements, the problems of device dependence and motion distortion in teleoperation are solved, achieving efficient and reliable control and cross-environment data alignment for device-free teleoperation, and improving the stability and applicability of humanoid robot manipulation.
Patent Information
- Application Number
- CN202611340077.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies require operators to wear human body tracking devices during remote operation. Optical full-body motion capture relies on fixed locations and is costly. Inertial motion capture suits are cumbersome to wear and have drift errors. Lightweight VR full-body solutions suffer from severe distortion of lower limb movements. There is a lack of balance and limit compensation algorithms adapted to the structure of humanoid robots. Remote operation and full-body teaching functions are disconnected, resulting in insufficient control stability.
By collecting images of human motion of operators, motion observation data is obtained. Combined with the motion planning sequence, the target motion of each joint of the humanoid robot is determined. After safety verification, the humanoid robot is driven to move and the motion screen is displayed for online correction. Remote operation control is performed using a camera or webcam. Bounded nonlinear least squares and damped least squares optimization are used to solve the joint motion.
It achieves efficient and reliable remote operation control without the need for wearing human body tracking devices, reducing the threshold and cost of use, ensuring the rationality and safety of humanoid robot posture, adapting to the conversion of control commands for different robot types, and supporting cross-environmental data alignment and imitation learning in real machine and simulation environments.
Smart Images

Figure CN123103650A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot control, and more specifically, to a humanoid robot visual teleoperation and teaching method and a humanoid robot in the field of robot control. Background Technology
[0002] Teleoperation refers to the real-time control of a robot by an operator at a remote location via communication, enabling the robot to mimic the operator's actions to complete a task. With the development of robotics, virtual reality, and simulation training technologies, robot teleoperation is increasingly being applied in scenarios such as humanoid robot control, robot skill transfer, simulation verification, remote task execution, and embodied intelligence data acquisition. In these scenarios, operators typically wear specialized human tracking equipment (such as motion capture suits, data gloves, and gyroscopes) to demonstrate actions. This equipment collects motion data such as joint postures and movement trajectories, and then generates control commands based on this data. The robot then replicates the operator's actions according to these control commands.
[0003] However, current technologies for full-body teleoperation based on human body tracking devices all have significant shortcomings: optical full-body motion capture relies on a fixed location, is prone to occlusion and frame loss, and is extremely expensive; inertial motion capture suits are cumbersome to wear and have drift errors during long-term tracking; lightweight VR full-body solutions rely on algorithm estimation for lower limb movements, resulting in severe distortion of dynamic full-body movements.
[0004] Therefore, how to achieve efficient and reliable full-body remote operation of humanoid robots without requiring operators to wear human body tracking devices is a technical problem that urgently needs to be solved. Summary of the Invention
[0005] This application provides a visual teleoperation and teaching method for anthropomorphic robots and an anthropomorphic robot. The method can obtain motion observation data by collecting human motion images of operators, determine the target actions of each joint of the anthropomorphic robot by combining the action planning sequence, and then drive the robot according to the target actions. This achieves efficient and reliable teleoperation control of anthropomorphic robots without the need for human tracking devices, relying only on human motion images captured by cameras, webcams and other imaging devices.
[0006] Firstly, a method for visual teleoperation and teaching of an anthropomorphic robot is provided. The method includes: acquiring human motion observation data based on human motion images of the operator; determining the target movements of each joint of the anthropomorphic robot based on the human motion observation data and the movement planning sequence, wherein the movement planning sequence refers to the sequence from the torso to the limbs; driving the anthropomorphic robot to move according to the target movements of each joint; and displaying the motion screen of the anthropomorphic robot so that the operator can correct the movements of the anthropomorphic robot.
[0007] The above technical solution acquires human motion observation data based on human motion images of the operator, and determines the target movements of each joint of the anthropomorphic robot based on this data and the motion planning sequence. Then, the anthropomorphic robot is driven to move based on these target movements. Finally, the motion image of the anthropomorphic robot is displayed, allowing the operator to make online corrections to the robot's movements by observing the motion. This achieves remote control of the anthropomorphic robot solely based on human motion images without the need for human tracking devices, effectively reducing the barrier to entry and cost of using remote robot operation. Furthermore, determining the target movements of each joint according to the motion planning sequence from the torso to the limbs ensures that the anthropomorphic robot's posture after moving based on the target movements conforms to human kinematics, thus guaranteeing the rationality of the robot's posture after moving based on the target movements.
[0008] In some possible implementations, determining the target action of each joint of the anthropomorphic robot based on human motion observation data and action planning sequence includes: determining the initial action of each joint in the next frame based on human motion observation data and action planning sequence; performing a safety check on the initial action of each joint in the next frame, the safety check including at least one of action validity check, joint limit check, inter-frame step size check, and self-collision check; if the initial action check of each joint in the next frame passes, determining the initial action of each joint in the next frame as the target action of each joint in the next frame; if the initial action check of any joint in the next frame fails, determining the target action of each joint in the previous frame as the target action of each joint in the next frame.
[0009] The above technical solution also includes a safety check when determining the target action, thereby ensuring that the humanoid robot can operate safely and smoothly according to the verified target action.
[0010] In some possible implementations, determining the initial movements of each joint in the next frame based on human motion observation data and motion planning sequence includes: determining the target rotation angle of the humanoid robot's torso based on the reference posture, torso orientation data, shoulder line direction data, and hip position data from the human motion observation data; taking the target movement of the humanoid robot's waist joint in the previous frame as the starting point, with the waist joint's limit range and the deviation between the actual rotation angle of the torso and the target rotation angle being less than a preset deviation as constraints, and the target rotation angle as the optimization target, optimizing the waist joint's movement to obtain the initial movement of the waist joint.
[0011] The above technical solution can solve the initial motion of the waist joint by using bounded nonlinear least squares, ensuring that after the anthropomorphic robot moves based on the initial motion of the waist joint, the actual orientation angle of the anthropomorphic robot's torso approaches the orientation angle of the operator's torso in the human motion image.
[0012] In some possible implementations, determining the initial movements of each joint in the next frame based on human motion observation data and action planning sequence includes: Given the initial movement of the humanoid robot's waist joint, determining the target rotation angle of the humanoid robot's head based on the reference posture, the initial movement of the humanoid robot's waist joint, and the head orientation angle in the human motion observation data; taking the target movement of the humanoid robot's neck joint in the previous frame as a starting point, with the neck joint's limit range and the deviation between the actual head rotation angle and the target rotation angle being less than a preset deviation as constraints, and the target rotation angle as the optimization objective, optimizing the neck joint's movement to obtain the initial movement of the neck joint; and determining the initial movement of the waist joint... In the case of motion, the target positions of the humanoid robot's hand and elbow joints are determined based on the initial motion of the waist joint and the hand and elbow joint positions in the human motion observation data. The hand includes the wrist or hand. Taking the target motion of the humanoid robot's arm joint in the previous frame as the starting point, and using the arm joint limit range of the humanoid robot as a constraint, the first deviation between the actual position and the target position of the humanoid robot's hand is less than a preset deviation, and the second deviation between the actual position and the target position of the humanoid robot's elbow is less than a preset deviation. The target position of the hand is the primary optimization objective, and the target position of the elbow is the secondary optimization objective. The motion of the arm joint is optimized and solved to obtain the initial motion of the arm joint.
[0013] The above scheme allows for the initial motion of the neck joints to be solved using bounded nonlinear least squares, ensuring that after the anthropomorphic robot's initial motion based on the neck joints, the actual orientation angle of the robot's head relative to its torso approximates the orientation angle of the operator's head relative to its torso in the human motion image. Simultaneously, the initial motion of the arm joints can be solved using damped least squares, ensuring that after the anthropomorphic robot's initial motion based on the arm joints, the actual position of the anthropomorphic robot's hand tissue approximates the position of the operator's hand tissue in the human motion image. Furthermore, without affecting the positional accuracy of the hand tissue, the actual position of the anthropomorphic robot's elbow joint approximates the position of the operator's elbow joint in the human motion image.
[0014] In some possible implementations, determining the initial movements of each joint in the next frame based on human motion observation data and motion planning sequence includes: acquiring key point information of the hand tissue from the human motion observation data, and determining the target posture of the hand tissue based on the key point information of the hand tissue, where the hand tissue includes the wrist or hand; taking the target movement of the humanoid robot's wrist joint in the previous frame as the starting point, using the humanoid robot's wrist joint limit range, and the deviation between the current posture of the humanoid robot's hand tissue and the target posture of the hand tissue being less than a preset deviation as a constraint, and taking the target posture of the hand tissue as the optimization objective, optimizing the movement of the wrist joint to obtain the initial movement of the wrist joint; or, decomposing the target posture of the hand tissue into a first rotation angle about the forearm axis of the humanoid robot and a second rotation angle perpendicular to the forearm axis. The rotation angle is calculated as follows: Taking the target motion of the wrist joint's roll axis in the previous frame as the starting point, and using the limit range of the humanoid robot's roll axis as a constraint, the deviation between the actual rotation angle of the roll axis and the first rotation angle is less than a preset deviation. The first rotation angle is the optimization target. The motion of the roll axis is optimized to obtain the initial motion of the roll axis. Taking the target motion of the wrist joint's pitch axis and / or yaw axis in the previous frame as the starting point, and using the limit range of the humanoid robot's pitch axis and / or yaw axis as a constraint, the deviation between the actual rotation angle of the pitch axis and / or yaw axis and the second rotation angle is less than a preset deviation. The second rotation angle is the optimization target. The motion of the pitch axis and / or yaw axis is optimized to obtain the initial motion of the pitch axis and / or yaw axis.
[0015] The above approach ensures that after the anthropomorphic robot performs its initial wrist joint movement, the actual posture of the robot's hand tissue closely approximates the posture of the operator's hand tissue in a human motion image.
[0016] In some possible implementations, determining the initial movements of each joint in the next frame based on human motion observation data and motion planning sequence includes: determining the hand coordinate system based on the position data of the wrist key points and the palm and finger root key points of the humanoid robot in the human motion observation data; converting the first position coordinates of each finger key point in the human observation data into the second position coordinates of the hand coordinate system; determining the finger target posture based on the second coordinate positions of each finger key point; taking the target movement of the humanoid robot's hand joint in the previous frame as the starting point, with the hand joint limit range and the deviation between the actual posture of the finger key points and the finger target posture being less than a preset deviation as constraints, and the finger target posture as the optimization target, optimizing the movement of the hand joint to obtain the initial movement of the hand joint.
[0017] The above approach ensures that after the anthropomorphic robot performs its initial hand joint movements, the actual posture of the robot's hand joints closely approximates the posture of the operator's hand joints in a human motion image.
[0018] In some possible implementations, driving the humanoid robot to move based on the target actions of each joint includes: determining the type of the humanoid robot based on the task instructions; determining the target instruction adapter based on the type; using the target instruction adapter to convert the target actions of each joint into instructions to obtain the control instructions for each joint; and driving the humanoid robot to move based on the control instructions for each joint.
[0019] Through the above scheme, the target action is a general action that only expresses the operator's motion intention and is not bound to any specific robot. Subsequently, the target action can be converted into control instructions that can control the specific robot's operation, depending on the robot type. When it is necessary to switch the execution object (such as switching from a real robot to a RoboTwin or SAPIEN simulator), only the corresponding instruction adapter needs to be replaced to convert the same set of target actions into control instructions that the robot can understand, without having to re-perform inverse kinematics solving or modify the upper-level control logic. Furthermore, this application enables the teaching data in the real robot and simulation environments to have a unified semantic format at the action level, facilitating subsequent cross-environment data alignment and imitation learning training.
[0020] In some possible implementations, human motion observation data is obtained from human motion images of operators, including: performing foreground segmentation on the human motion images to obtain target images of operators; inputting the target images into a human posture model to determine the three-dimensional key points corresponding to each joint of the operator; and determining human motion observation data based on the three-dimensional key points of each joint.
[0021] The above approach constructs a target image containing only the human body region using a foreground mask. By inputting the target image into the human pose model, the model's attention can be focused on the area where the operator is located, thereby effectively eliminating background interference, reducing false detections and key point offsets, and improving the accuracy of human motion observation data.
[0022] In some possible implementations, after driving the humanoid robot to move according to the target movements of each joint, the method further includes: generating teaching data based on human motion observation data, the target movements of each joint, the actual motion state of the humanoid robot, and environmental information. The teaching data is used for different types of humanoid robots to imitate and learn.
[0023] Through the above approach, the data structure and field definitions of the teaching data remain consistent in both real robot environments and simulated robot environments, allowing teaching data from different execution environments to be mixed for imitation learning training.
[0024] Secondly, a humanoid robot is provided, including a memory and a processor. The memory is used to store executable program code. The processor is used to call and run the executable program code from the memory, causing the humanoid robot to perform the methods in the first aspect or any possible implementation thereof. Attached Figure Description
[0025] Figure 1 This is a first schematic flowchart of a visual teleoperation and teaching method for an anthropomorphic robot provided in an embodiment of this application; Figure 2 This is a second schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 3 This is a third schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 4 This is a fourth schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 5 This is a fifth schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 6 This is a sixth schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 7 This is the seventh schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 8 This is the eighth schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 9 This is the ninth schematic flowchart of a humanoid robot visual teleoperation and teaching method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of an anthropomorphic robot provided in an embodiment of this application. Detailed Implementation
[0026] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.
[0027] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.
[0028] Teleoperation refers to the real-time control of a robot by an operator at a remote location via communication, enabling the robot to mimic the operator's actions to complete a task. With the development of robotics, virtual reality, and simulation training technologies, robot teleoperation is increasingly being applied in scenarios such as humanoid robot control, robot skill transfer, simulation verification, remote task execution, and embodied intelligence data acquisition. In these scenarios, operators typically wear specialized human tracking equipment (such as motion capture suits, data gloves, and gyroscopes) to demonstrate actions. This equipment collects motion data such as joint postures and movement trajectories, and then generates control commands based on this data. The robot then replicates the operator's actions according to these control commands.
[0029] However, current technologies for full-body teleoperation based on human body tracking devices all have significant shortcomings: optical full-body motion capture relies on fixed locations, is prone to occlusion and frame loss, and is extremely costly; inertial motion capture suits are cumbersome to wear and drift errors occur during long-term tracking; lightweight VR full-body solutions rely on algorithm estimation for lower limb movements, resulting in severe distortion of dynamic full-body movements; at the same time, existing full-body teleoperation lacks balance and limit compensation algorithms adapted to the structure of humanoid robots, has poor synchronization of full-body multimodal data, and disconnects teleoperation from full-body teaching functions, resulting in insufficient overall control stability and practicality.
[0030] Therefore, how to achieve efficient and reliable full-body remote operation of humanoid robots without requiring operators to wear human body tracking devices is a technical problem that urgently needs to be solved.
[0031] To address the above issues, this application provides a method for visual teleoperation and teaching of an anthropomorphic robot.
[0032] The following section will elaborate on the humanoid robot visual teleoperation and teaching method of this application: Figure 1 This application provides a method for visual teleoperation and teaching of a humanoid robot, comprising: Step 01: Obtain human motion observation data based on the human motion images of the operator.
[0033] Human motion images are images containing the operator's limbs, acquired by imaging devices (such as cameras or webcams). Human motion observation data is structured data used to describe the operator's motion state, including not only a three-dimensional human skeleton coordinate sequence arranged according to fixed semantics, but also fine key points of the left and right hands, joint angle information, etc.
[0034] Specifically, the deployment method of the shooting equipment can be flexibly selected according to the actual application scenario, as long as it can stably and clearly capture images of the operator's human movement. For example, the shooting equipment can be the built-in shooting device of a laptop or mobile phone, or it can be externally mounted on a fixed bracket or tripod to ensure consistency in shooting angle and field of view. Alternatively, the shooting equipment can be fixed to a head-mounted device or shoulder support to adapt to scenarios where the operator moves over a wide range.
[0035] Human motion images of operators can be acquired using imaging equipment. The operators can then be identified and their positions determined within these images. Next, the two-dimensional pixel coordinates of each joint of the operator are detected from the motion images and converted into three-dimensional human skeleton coordinates. These three-dimensional human skeleton coordinates are then used to determine the human motion observation data.
[0036] Step 02: Determine the target movements of each joint of the humanoid robot based on human motion observation data and the sequence of motion planning. The sequence of motion planning refers to the order from the torso to the limbs.
[0037] Specifically, based on human motion observation data, the operator's torso orientation angle, upper limb joint positions, and finger joint postures can be determined. Combined with the motion planning sequence, the target movements of each joint of the anthropomorphic robot can be determined in stages. The target movement can be the specific rotation angle value of each joint.
[0038] The sequence of motion planning refers to the order from the trunk to the limbs. For example, when determining the target motion for the upper limbs, the target motion for the lumbar joints can be determined first, followed by the target motion for the cervical joints, then the target motions for both arm joints, then the target motions for both wrist joints, and finally the target motions for both hand joints. When determining the target motion for the lower limbs, the target motion for the hip joints can be determined first, followed by the target motion for the knee joints, then the target motion for the ankle joints, and finally the target motion for the foot joints.
[0039] As the foundation of the limbs, the trunk's posture changes are transmitted through forward kinematics and affect all limb joints. Therefore, determining the target movements of each joint according to the action planning sequence ensures that the anthropomorphic robot's posture after performing the target movements conforms to human kinematics, thus ensuring the rationality of the anthropomorphic robot's posture after performing the target movements. Furthermore, this application decomposes the solution of the target movements of each joint of the anthropomorphic robot into multiple joints that are solved independently layer by layer. This reduces the computational complexity of a single solution, improves the solution speed and numerical stability, and meets the requirements of real-time control for latency and reliability.
[0040] Step 03: Drive the humanoid robot to move according to the target motion of each joint.
[0041] Specifically, corresponding control commands can be generated based on the target actions of each joint, and the humanoid robot can be driven to move according to the control commands, so that each joint of the humanoid robot can perform the corresponding target actions.
[0042] For example, based on the target motion of each joint, the displacement and rotation angle required for each joint to adjust from the current motion to the target motion of the humanoid robot can be determined, and corresponding control commands can be generated based on the displacement and rotation angle to drive the humanoid robot to move.
[0043] Step 04: Display the motion graphics of the humanoid robot so that the operator can correct the robot's movements.
[0044] Specifically, after the humanoid robot moves based on the target action, the display can show the humanoid robot's motion footage. For example, it can acquire a first-person perspective motion footage, a third-person perspective motion footage, or both first-person and third-person perspective motion footage simultaneously. The first-person perspective motion footage refers to the robot's viewpoint captured by the camera equipment mounted on the humanoid robot itself, which is used to show the relative positional relationship between the humanoid robot and the target object. The third-person perspective motion footage refers to the scene viewpoint captured by environmental camera equipment independent of the humanoid robot, which is used to show the humanoid robot's overall posture and spatial position in the working environment.
[0045] In this way, when the operator sees the motion screen, they can adjust their posture according to the motion screen, so that the humanoid robot can correct its movements according to the posture adjusted by the operator.
[0046] The humanoid robot visual teleoperation and teaching method of this application acquires human motion observation data based on human motion images of the operator, determines the target movements of each joint of the humanoid robot based on the human motion observation data and the action planning sequence, then drives the humanoid robot to move based on the target movements of each joint, and finally displays the motion screen of the humanoid robot, allowing the operator to make online corrections to the robot's movements by observing the motion screen. In this way, efficient and reliable teleoperation control of the humanoid robot is achieved without the need for human tracking devices, relying solely on human motion images captured by cameras or other imaging devices. Furthermore, determining the target movements of each joint according to the action planning sequence from the torso to the limbs ensures that the posture of the humanoid robot after moving based on the target movements conforms to human kinematics, thereby ensuring the rationality of the humanoid robot's posture after moving based on the target movements.
[0047] Please see Figure 2 In some implementations, step 02: determining the target movements of each joint of the humanoid robot based on human motion observation data and the sequence of motion planning includes: Step 021: Determine the initial movements of each joint in the next frame based on human motion observation data and motion planning sequence; Step 023: Perform a safety check on the initial motion of each joint in the next frame. The safety check includes at least one of the following: motion validity check, joint limit check, inter-frame step size check, and self-collision check. Step 025: If the initial motion of each joint in the next frame passes the verification, determine the initial motion of each joint in the next frame as the target motion of each joint in the next frame. Step 027: If the initial motion verification of any joint in the next frame fails, determine the target motion of each joint in the previous frame as the target motion of each joint in the next frame.
[0048] Specifically, based on human motion observation data, the operator's torso orientation angle, upper limb joint position, and finger joint posture can be determined. Then, based on the torso orientation angle, upper limb joint position, and finger joint posture, the initial movements of the corresponding joints of the anthropomorphic robot in the next frame can be determined in stages according to the action planning sequence.
[0049] Then, a safety check is performed on the initial movements of each joint in the next frame. This safety check includes at least one of the following: movement validity check, joint limit check, inter-frame step size check, and self-collision check. Movement validity check verifies whether the joint angle values in the initial movement are valid, ensuring that each joint angle value is a normal real number. Joint limit check verifies whether each joint angle is within its mechanical limit range, ensuring that the humanoid robot's joints do not exceed the preset joint movement range and preventing hardware damage. Inter-frame step size check verifies whether the change in angle between the initial movement and the target movement in the previous frame exceeds the maximum allowable step size, preventing abrupt changes in joint angles between adjacent frames and ensuring smooth and continuous movement. Self-collision check verifies whether the initial movement will cause collisions between different parts of the robot, preventing collision damage to the robot itself.
[0050] If the initial movements of all joints in the humanoid robot pass verification in the next frame, then the initial movements of each joint in the next frame are determined as the target movements for that joint in the next frame. If the initial movement verification of any joint in the next frame fails, then the target movements of each joint in the previous frame are determined as the target movements for that joint in the next frame. That is, if even one joint fails the initial movement verification in the next frame, the target movement of the previous frame is directly reused as the target movement for the next frame. Only when all joints pass verification will the target movement for the next frame be determined based on the initial movements of the current frame.
[0051] In this way, a safety check is also performed when determining the target action, thereby ensuring that the humanoid robot can operate safely and smoothly according to the target action that has passed the check.
[0052] Please see Figure 3 In some implementations, step 021: determining the initial movement of each joint in the next frame based on human motion observation data and motion planning sequence, including: Step 0211: Based on the torso orientation data, shoulder line direction data, and hip position data in the human motion observation data, determine the target rotation angle of the humanoid robot's torso; Step 0212: Taking the target action of the humanoid robot's waist joint in the previous frame as the starting point, the waist joint's limit range and the deviation between the actual rotation angle of the torso and the target rotation angle being less than the preset deviation are used as constraints, and the target rotation angle is used as the optimization target. The waist joint's action is optimized to obtain the initial action of the waist joint.
[0053] The initial motion determination process for the humanoid robot's waist joint is as follows: Specifically, based on the torso orientation data, shoulder line direction data, and hip position data from human motion observation data, the target rotation angle of the humanoid robot's torso relative to a reference posture can be determined. The reference posture is pre-set; for example, when the robot starts, the operator maintains a standard standing posture within a reference window, which can serve as the reference posture. Other postures can also be used, selected based on the specific circumstances, and will not be listed here.
[0054] Then, the initial motion of the lumbar joint is solved using bounded nonlinear least squares, with the target rotation angle of the anthropomorphic robot's torso as the optimization objective. Specifically, starting with the target motion of the lumbar joint in the previous frame, a pre-set offset step size is used as the motion adjustment increment, and the offset is iteratively superimposed according to the offset step size to obtain multiple candidate motions for the lumbar joint. For each candidate motion, it is determined whether the lumbar joint is within its limit range after the anthropomorphic robot runs according to the candidate motion, and the deviation between the actual rotation angle of the torso corresponding to the candidate motion and the target rotation angle of the torso is calculated. Finally, the candidate motion of the lumbar joint that is within its limit range and has the smallest deviation is determined as the initial motion of the lumbar joint.
[0055] In this way, the initial motion of the lumbar joint can be solved by bounded nonlinear least squares, ensuring that after the anthropomorphic robot moves based on the initial motion of the lumbar joint, the actual orientation angle of the anthropomorphic robot's torso approaches the orientation angle of the operator's torso in the human motion image.
[0056] Please see Figure 4 In some implementations, step 021: determining the initial movement of each joint in the next frame based on human motion observation data and motion planning sequence, including: Step 0213: Given the initial movement of the humanoid robot's waist joint, determine the target rotation angle of the humanoid robot's head based on the reference posture, the initial movement of the humanoid robot's waist joint, and the head orientation angle in the human motion observation data. Step 0214: Taking the target action of the humanoid robot's neck joint in the previous frame as the starting point, the limit range of the neck joint and the deviation between the actual rotation angle of the head and the target rotation angle are less than the preset deviation as constraints, and the target rotation angle is the optimization target. The action of the neck joint is optimized and solved to obtain the initial action of the neck joint. Step 0215: Given the initial movement of the lumbar joint, determine the target positions of the humanoid robot's hand and elbow joints based on the initial movement of the lumbar joint, the hand tissue position, and the elbow joint position in the human motion observation data. The hand tissue includes the wrist or hand. Step 0216: Taking the target action of the humanoid robot's arm joint in the previous frame as the starting point, and using the arm joint limit range of the humanoid robot, the first deviation between the actual position of the humanoid robot's hand tissue and the target position of the hand tissue being less than the preset deviation, and the second deviation between the actual position of the humanoid robot's elbow and the target position of the elbow joint being less than the preset deviation as constraints, the target position of the hand tissue is the primary optimization objective, and the target position of the elbow joint is the secondary optimization objective, the action of the arm joint is optimized and solved to obtain the initial action of the arm joint.
[0057] The initial motion determination process for the humanoid robot's neck joint is as follows: Specifically, given the initial motion of the lumbar joint, the current orientation angle of the torso is determined based on the initial motion of the lumbar joint. Then, based on the current orientation angle of the torso and the orientation angle of the head in the human motion observation data, the current orientation angle of the head relative to the torso is determined. Finally, based on the relative rotation angle between the current orientation angle of the head relative to the torso and the reference orientation angle of the head relative to the torso in the reference posture, the target rotation angle of the head is determined.
[0058] Next, the initial motion of the cervical joint is solved using bounded nonlinear least squares, with the target rotation angle of the head as the optimization objective. Specifically, starting with the target motion of the cervical joint in the previous frame, a pre-set offset step size is used as the motion adjustment increment, and the offset is iteratively superimposed according to the offset step size to obtain multiple candidate motions for the cervical joint. Then, for each candidate motion, it is determined whether the cervical joint is within its limit range after the anthropomorphic robot runs according to the candidate motion, and the deviation between the actual rotation angle of the head corresponding to the candidate motion and the target rotation angle of the head is calculated. Finally, the candidate motion of the cervical joint that is within its limit range and has the smallest deviation is determined as the initial motion of the cervical joint.
[0059] In this way, the initial motion of the cervical joint can be solved by bounded nonlinear least squares, ensuring that after the anthropomorphic robot moves based on the initial motion of the cervical joint, the actual orientation angle of the anthropomorphic robot's head relative to the torso is close to the orientation angle of the operator's head relative to the torso in the human motion image.
[0060] The initial motion determination process for the humanoid robot's arm joints is as follows: Similarly, based on the determined initial motion of the waist joint, the pose of the humanoid robot's shoulder base is determined. Then, based on the pose of the shoulder base and the hand tissue position in the human motion observation data, the target positions of the left and right hand tissues are determined. Furthermore, based on the pose of the shoulder base and the elbow joint position in the human motion observation data, the target positions of the left and right elbow joints are determined.
[0061] Next, the initial motion of the arm joints is solved using damped least squares. That is, taking the target motion of the arm joint in the previous frame as the starting point, a pre-set offset step size is used as the motion adjustment increment, and the offset is iteratively superimposed according to the offset step size to obtain multiple candidate motions for the arm joints. Then, for each candidate motion of the arm joint, it is determined whether the arm joint is within its limit range after the anthropomorphic robot runs according to the candidate motion, and the first deviation between the actual position of the hand tissue and the target position of the hand tissue corresponding to each candidate motion of the arm joint, and the second deviation between the actual position of the elbow and the target position of the elbow joint corresponding to each candidate motion of the arm joint are calculated. The target position of the hand tissue can be taken as the primary optimization objective, and the target position of the elbow joint as the secondary optimization objective. The primary objective is used as the first priority optimization objective, and the secondary objective is used as the second priority optimization objective during the solution process. Finally, the candidate action of the arm joint that is within the limit range of the arm joint and has the smallest primary task error is determined as the initial action of the arm joint. If there are multiple candidate actions of the arm joint with the smallest primary task error, the candidate action with the smallest secondary task error is selected from these multiple candidate actions and determined as the initial action of the arm joint.
[0062] In this way, the initial movements of the arm joints can be solved by damped least squares, ensuring that after the anthropomorphic robot moves based on the initial movements of the arm joints, the actual position of the anthropomorphic robot's hand tissues is close to the position of the operator's hand tissues in the human motion image, and without affecting the positional accuracy of the hand tissues, the actual position of the anthropomorphic robot's elbow joints is close to the position of the operator's elbow joints in the human motion image.
[0063] Please see Figure 5 In some implementations, step 021: determining the initial movement of each joint in the next frame based on human motion observation data and motion planning sequence, including: Step 0217: Obtain key point information of hand tissue from human motion observation data, and determine the target posture of hand tissue based on the key point information of hand tissue; Step 0218: Taking the target action of the humanoid robot's wrist joint in the previous frame as the starting point, and using the humanoid robot's wrist joint limit range and the deviation between the current pose and the target pose of the hand organization being less than a preset deviation as constraints, and the target pose of the hand organization as the optimization objective, optimize the wrist joint action to obtain the initial wrist joint action; or, Step 0219: Decompose the hand tissue target posture into a first rotation angle about the forearm axis of the humanoid robot and a second rotation angle perpendicular to the forearm axis; Step 0220: Taking the target action of the wrist joint's rotation axis in the previous frame as the starting point, and taking the limit range of the humanoid robot's rotation axis, the deviation between the actual rotation angle of the rotation axis and the first rotation angle being less than the preset deviation as the constraint, and the first rotation angle as the optimization target, optimize the action of the rotation axis to obtain the initial action of the rotation axis. Step 0221: Taking the target motion of the pitch axis and / or yaw axis of the wrist joint in the previous frame as the starting point, and taking the limit range of the pitch axis and / or yaw axis of the humanoid robot, and the deviation between the actual rotation angle of the pitch axis and / or yaw axis and the second rotation angle being less than a preset deviation as constraints, and the second rotation angle as the optimization target, optimize the motion of the pitch axis and / or yaw axis to obtain the initial motion of the pitch axis and / or yaw axis.
[0064] The initial motion determination process for the humanoid robot's wrist joint is as follows: Specifically, key point information of the hand tissue is acquired from human motion observation data, and the target posture of the hand tissue is determined based on this key point information. This key point information may include multiple three-dimensional point positions at the wrist, palm, and base of the fingers.
[0065] In one embodiment, the initial wrist joint movement can be obtained using a bounded optimization approach, with the target pose of the hand tissue as the optimization objective. Starting with the target movement of the hand joint in the previous frame, a pre-set offset step size is used as the movement adjustment increment. The offsets are iteratively stacked according to the offset step size to obtain multiple candidate wrist joint movements. Then, for each candidate wrist joint movement, it is determined whether the wrist joint is within its limit range after the anthropomorphic robot runs according to the candidate movement, and the deviation between the current pose of the hand tissue corresponding to the candidate movement and the target pose of the hand tissue is calculated. Finally, the candidate movement of the wrist joint that is within the limit range and has the smallest deviation is determined as the initial wrist joint movement.
[0066] For example, there can be bounded Levenberg. The Levenberg-Marquardt (LM) method optimizes multiple rotational joints of a humanoid robot's wrist, enabling the robot's palm and pointing postures to simultaneously approximate the palm and pointing postures of a human operator. It's important to note that the optimization process does not truncate each Euler angle individually, but rather jointly fits all directional constraints.
[0067] In another embodiment, the target pose of the hand is decomposed into a first rotation angle about the forearm axis of the anthropomorphic robot and a second rotation angle perpendicular to the forearm axis. The limit ranges of the wrist joint's roll axis and its target motion in the previous frame, the limit ranges of the wrist joint's pitch axis and its target motion in the previous frame, and the limit ranges of the wrist joint's yaw axis and its target motion in the previous frame are obtained. Then, the initial motions of the pitch axis, the initial motion of the yaw axis, and the initial motion of the yaw axis are solved using a bounded optimization method. The first and second rotation angles are used as optimization objectives.
[0068] Starting with the target action of the roll axis in the previous frame, a pre-set offset step size is used as the action adjustment increment. The offset is iteratively superimposed according to the offset step size to obtain multiple candidate actions for the roll axis. Then, for each candidate action, it is determined whether the humanoid robot's wrist joint is within its limit range after running according to the candidate action, and the deviation between the actual rotation angle of the roll axis corresponding to the candidate action and the first rotation angle is calculated. Finally, the candidate action of the roll axis that is within the wrist joint limit range and has the smallest deviation is determined as the initial action of the roll axis.
[0069] Starting with the target motion of the pitch axis in the previous frame, a pre-set offset step size is used as the motion adjustment increment. The offsets are iteratively superimposed according to the offset step size to obtain multiple candidate motions for the pitch axis. Then, for each candidate motion, it is determined whether the humanoid robot's wrist joint is within its limit range after running according to the candidate motion, and the deviation between the actual rotation angle of the pitch axis corresponding to the candidate motion and the second rotation angle is calculated. Finally, the candidate motion of the pitch axis that is within the wrist joint limit range and has the smallest deviation is determined as the initial motion of the pitch axis.
[0070] Starting with the target action of the yaw axis in the previous frame, a pre-set offset step size is used as the action adjustment increment. The offset is iteratively superimposed according to the offset step size to obtain multiple candidate actions for the yaw axis. Then, for each candidate action, it is determined whether the humanoid robot's wrist joint is within its limit range after running according to the candidate action, and the deviation between the actual rotation angle of the yaw axis corresponding to the candidate action and the second rotation angle is calculated. Finally, the candidate action of the yaw axis that is within the wrist joint limit range and has the smallest deviation is determined as the initial action of the yaw axis.
[0071] The initial movements of the yaw axis and pitch axis can be determined individually or simultaneously.
[0072] It should be noted that when the arm extension exceeds the preset threshold and the roll axis has reached its limit, the portion of the first rotation angle that exceeds the limit range of the roll axis is compensated to the shoulder joint according to the shoulder joint state of the previous frame and the preset maximum compensation amount per frame.
[0073] In this way, it can be ensured that after the initial movement of the humanoid robot based on the wrist joint, the actual posture of the humanoid robot's hand tissue is close to the posture of the operator's hand tissue in the human motion image.
[0074] Please see Figure 6 In some implementations, step 021: determining the initial movement of each joint in the next frame based on human motion observation data and motion planning sequence, including: Step 0222: Determine the hand coordinate system based on the location data of key points on the wrist and key points at the base of the palm and fingers in the human motion observation data; Step 0223: Convert the first position coordinates of each finger key point in the human motion observation data into the second position coordinates of the palm coordinate system; Step 0224: Determine the target finger posture based on the second coordinate position of each finger key point; Step 0225: Taking the target action of the humanoid robot's hand joint in the previous frame as the starting point, the hand joint limit range and the deviation between the actual posture of the finger key points and the finger target posture are less than the preset deviation as constraints, and the finger target posture as the optimization target, the action of the hand joint is optimized to obtain the initial action of the hand joint.
[0075] The initial motion determination process for the humanoid robot's hand joints is as follows: Specifically, based on the position data of the wrist key points and the position data of the palm and finger root key points in the human motion observation data, the hand coordinate system is determined. For example, the wrist point, five palm and finger root points, and the midpoint and end point of each finger are determined from each of the 21 hand key points of each hand. The hand coordinate system is constructed based on the wrist point and palm and finger root points.
[0076] Then, the key points of each finger are transformed to the palm coordinate system. The first position coordinates of each key point of the humanoid robot's fingers in the human motion observation data are transformed into the second position coordinates in the palm coordinate system. At this time, scale normalization can also be performed using palm width or selected bone segment length to eliminate the differences in hand size among different operators. Next, the target finger posture is determined based on the second coordinate position of each finger key point. The characteristics of the target finger posture include at least one of the following: the vector from the base of the palm to the fingertip, the direction vector of adjacent phalanges, the position of the fingertip relative to the palm, and the finger bending angle.
[0077] Starting with the target action of the hand joint in the previous frame, a pre-set offset step size is used as the action adjustment increment. The offset is iteratively stacked according to the offset step size to obtain multiple candidate actions for the hand joint. Then, for each candidate action, it is determined whether the humanoid robot operates within the hand joint's limit range after executing the candidate action, and the deviation between the actual finger posture corresponding to the candidate action and the target finger posture is calculated. Finally, using the target finger posture as the optimization target, the candidate action of the hand joint that is within the hand joint's limit range and has the smallest deviation is determined as the initial action of the hand joint.
[0078] In this way, it can be ensured that after the anthropomorphic robot performs its initial hand joint movements, the actual posture of the anthropomorphic robot's hand joints will closely resemble the posture of the operator's hand joints in a human motion image.
[0079] Please see Figure 7 In some implementations, step 03: driving the humanoid robot to move according to the target motion of each joint includes: Step 031: Determine the type of humanoid robot based on the task instructions; Step 032: Determine the target instruction adapter based on the type; Step 033: Use a target instruction adapter to convert the target motion of each joint into control instructions for each joint. Step 034: Drive the humanoid robot to move according to the control commands of each joint.
[0080] Specifically, this application can be applied to different humanoid robots. Humanoid robots include real robots (such as the PrimeU real robot) and simulation robots (such as the RoboTwin simulation robot and the PrimeU SAPIEN simulation robot). Real robots are physical hardware devices, constrained by real physical conditions, and subject to mechanical errors, friction, and control delays. Simulation robots are virtual models built by computers, operating in a software environment, with physical conditions mostly idealized.
[0081] The type of humanoid robot can be determined based on the task instructions. For example, if the task instructions indicate that the technical objective is to verify whether the system can truly control the hardware and complete operations, and the technical conditions require acquiring real sensing, real friction / compliance, hardware latency, final operation verification, or real machine data, then a real robot is needed, and the type of humanoid robot can be determined as a real robot. If the task instructions indicate that the technical objective is to rapidly expand task coverage and human closed-loop trajectory, and the technical conditions require a large number of tasks, objects, scenes, and various off-the-shelf robot configurations, with gripper tasks being prioritized, then a RoboTwin simulation robot is needed, and the type of humanoid robot can be determined as a RoboTwin simulation robot. If the task instructions indicate that the technical objective is digital twin, wrist-hand coordination, contact control, and safety verification, and the technical conditions require PrimeU joint structure, 60-dimensional full-body motion, dexterous hand, and contact dynamics, then a PrimeUSAPIEN simulation robot is needed, and the type of humanoid robot can be determined as a PrimeU SAPIEN simulation robot.
[0082] Different anthropomorphic robots have different structures, which means that different anthropomorphic robots have different instruction adapters.
[0083] When the robot type is a real robot, the selected instruction adapter can output a motion request that can be recognized by the real robot's controller based on the target action, and input the motion request to the real robot's controller so that the controller outputs control instructions, such as the controller generating a movement trajectory, so that the real robot can move according to the movement trajectory to execute the target action.
[0084] In the case of a RoboTwin robot simulator, the selected instruction adapter uses the target motion of the arm joint in the first frame as the operator's reference posture and obtains the increment of the target motion of the arm joint in the current frame relative to the reference posture. For hand movements, the target motion of the hand joint is projected as a normalized gripper opening / closing value through a grasping cooperative function. Then, control commands are determined based on the increment of the target motion of the arm joint in the current frame relative to the reference posture and the gripper opening / closing value.
[0085] When the robot type is PrimeU SAPIEN simulation robot, the selected instruction adapter can load the PrimeU digital twin model and read the list of names of all its active joints, establishing a mapping relationship between the semantic names of each joint in the target action and the active joint indexes. Then, according to the mapping relationship, the angle values of each joint in the target action are assigned one by one to the corresponding active joints in the PrimeU SAPIEN simulation robot. Auxiliary joints not covered in the target action retain their initial values or are passively driven by the physics engine, thereby determining the control commands.
[0086] Therefore, a target instruction adapter can be determined according to the type of humanoid robot. The target instruction adapter is used to convert the target actions of each joint into control instructions for each joint. The humanoid robot is then driven to move according to the control instructions of each joint, ensuring that the humanoid robot can perform the target actions.
[0087] Thus, the target action is a general action that only expresses the operator's movement intention and is not bound to any specific robot. Subsequently, the target action can be converted into control instructions that can control the specific robot's operation, depending on the robot type. When it is necessary to switch the execution object (such as switching from a real robot to a RoboTwin simulation robot or a PrimeU SAPIEN simulation robot), only the corresponding instruction adapter needs to be replaced to convert the same set of target actions into control instructions that the robot can understand, without having to re-perform inverse kinematics solutions or modify the upper-level control logic. Furthermore, this application enables teaching data in real and simulation environments to have a unified semantic format at the action level, facilitating subsequent cross-environment data alignment and imitation learning training.
[0088] Please see Figure 8 In some implementations, step 01: acquiring human motion observation data based on the operator's human motion images, including: Step 011: Perform foreground segmentation on the human motion image to obtain the target image of the operator; Step 012: Input the target image into the human posture model to determine the three-dimensional key points corresponding to each joint of the operator; Step 013: Determine human motion observation data based on the three-dimensional key points of each joint.
[0089] Among them, the human pose model is a computer vision model used to recover the three-dimensional spatial position of the joints of the human body from a two-dimensional image. Its implementation methods include, but are not limited to: an end-to-end model that directly regresses the coordinates of three-dimensional key points from a two-dimensional image, or a combined model composed of a two-dimensional key point detector and a temporal three-dimensional lifting network. This model will obtain two-dimensional pixel features from the image and map the two-dimensional pixel features to three-dimensional spatial coordinates, and output a three-dimensional human skeleton arranged according to a fixed semantic.
[0090] Specifically, the latest human motion image output by the imaging device can be acquired. At the same time, the operator's target bounding box in the human motion image is determined based on the initial center box, target tracking results, or the torso position of the previous frame. Under the constraint of the target bounding box, the human body region is separated from the background through foreground segmentation. A foreground mask is generated based on the separated human body region, thus obtaining the target image.
[0091] Then, the human motion image can be input into the human posture model to determine the three-dimensional key points corresponding to each joint of the operator, such as the main body key points, head and neck, elbow and shoulder auxiliary points, and human body camera translation. The human posture model can include a three-dimensional body model and a hand model. The torso and arms can be output from the three-dimensional body model, and 21 key points for each hand can be output from the hand model.
[0092] Furthermore, in some embodiments, if the operator wears a data glove, the three-dimensional key points of the hand joints can be updated based on the data collected by the data glove. The data output by the data glove typically includes the local rotation or flexion angles of each finger joint, or the spatial pose of each phalanx relative to the wrist or palm coordinate system of the glove. The human pose model is responsible for providing the overall position of the human body, the direction of the forearm, and the global pose of the palm.
[0093] When updating the 3D keypoints of the hand joints, the coordinate transformation relationship from the glove coordinate system to the standard palm coordinate system is first calculated under the reference pose, and the glove zero point or range is recorded. Then, the palm pose is determined based on the output data of the human posture model. If the data glove outputs local point data, it is transformed to the same coordinate system as the body skeleton through coordinate transformation to obtain the updated 3D keypoints of the hand joints. If the glove outputs joint angle data, the standard 21-point hand skeleton is directly generated through the positive motion of the glove's hand model to obtain the 3D keypoints of the hand joints, or the robot's finger joint angles are directly solved through joint mapping and optimization to obtain the 3D keypoints of the hand joints.
[0094] Finally, the human motion observation data is determined based on the coordinate information of the three-dimensional key points of each joint.
[0095] In this way, by constructing a target image containing only the human body region through a foreground mask and inputting the target image into the human pose model, the human pose model can focus its attention on the area where the operator is located, thereby effectively eliminating background interference, reducing false detections and key point offsets, and improving the accuracy of human motion observation data.
[0096] In some implementations, the initial movements of the joints in the lower limbs can be determined by referring to the method used to determine the initial movements of the joints in the upper limbs. For example, the joints of the lower limbs include the hip joint, knee joint, ankle joint, and foot joint. The solution is performed layer by layer in the order of "hip joint → knee joint → ankle joint → foot joint". Each layer uses the target position or orientation of the corresponding joint in the human motion observation data as the optimization objective, uses the results determined in the previous joint group as the kinematic reference, and uses the limit range of the joint as a hard constraint. The optimal joint angle is solved through bounded optimization, so that the actual state of the joint of the anthropomorphic robot approaches the target state of the operator in the human motion image. The determined joint angles are not changed due to the solution of subsequent joints.
[0097] Please see Figure 9 In some implementations, after step 03: driving the humanoid robot to move according to the target movements of each joint, the humanoid robot visual teleoperation and teaching method further includes: Step 05: Generate teaching data based on human motion observation data, target movements of each joint, actual motion state of the humanoid robot, and environmental information. The teaching data is used for different types of humanoid robots to imitate and learn.
[0098] Among them, environmental information refers to data about the surrounding scene and its own state collected by the robot through its own or external sensors during operation, such as first-person perspective motion images and third-person perspective motion images.
[0099] Specifically, after driving the humanoid robot to move, teaching data can be generated based on human motion observation data, target actions of each joint, the actual motion state of the humanoid robot, and environmental information. The teaching data is used for different types of humanoid robots to imitate and learn.
[0100] Specifically, each frame of human motion image can be assigned an incremental source frame identifier, a unique session identifier can be assigned to the entire task, and timestamps from the task timeline can be appended to the data generated in each step. The source frame identifier is an incremental number assigned to each frame of human motion image, used to mark the image time sequence and distinguish image data from different times. The session identifier is a unique number assigned to a single complete teleoperation task, used to isolate different tasks and avoid data confusion between multiple tasks.
[0101] Human motion observation data, target movements of each joint, actual motion state of the humanoid robot, and environmental information are grouped at the task level using session identifiers, frame-level association keys using source frame identifiers, and time-series alignment using timestamps. This ensures that the input skeleton, motion intent, execution commands, actual state, and environmental observations under the same source frame identifier can accurately correspond. After the above alignment, teaching data containing a complete input-processing-execution-feedback chain is formed.
[0102] In this way, the data structure and field definitions of the teaching data remain consistent in both real robot environments and simulated robot environments, allowing teaching data from different execution environments to be mixed for imitation learning training.
[0103] Existing whole-body teleoperation technologies lack balance and limit compensation algorithms adapted to the structure of humanoid robots, suffer from poor synchronization of multimodal data, and exhibit a disconnect between teleoperation and whole-body teaching functions, resulting in insufficient overall control stability and practicality. This application, however, integrates teleoperation and teaching data generation into a single process. Each teleoperation operation is automatically recorded as teaching data in a unified format, eliminating the need for additional data annotation or format conversion. The data simultaneously includes human intent, environmental commands, execution feedback, and task results, which can be directly used for subsequent imitation learning and strategy training, avoiding the functional separation of "teleoperation only controls, teaching is collected separately" in existing systems.
[0104] Please see Figure 10 The anthropomorphic robot 100 of this application includes a memory 10 and a processor 20. The memory 10 is used to store executable program code; the processor 20 is used to call and run the executable program code from the memory, so that the anthropomorphic robot executes the instructions of the anthropomorphic robot visual teleoperation and teaching method as described in any of the above embodiments.
[0105] In the description of this specification, the references to terms such as "some embodiments," "in one example," "exemplarily," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0106] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.
[0107] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A method for visual teleoperation and teaching of an anthropomorphic robot, characterized in that, The method includes: Human motion observation data is obtained based on images of the operator's human motion. The target movements of each joint of the humanoid robot are determined based on the human motion observation data and the action planning sequence, wherein the action planning sequence refers to the sequence from the torso to the limbs; The humanoid robot is driven to move according to the target movements of each joint; Displaying motion images of the anthropomorphic robot allows the operator to correct the robot's movements.
2. The method according to claim 1, characterized in that, The step of determining the target movements of each joint of the humanoid robot based on the human motion observation data and the action planning sequence includes: The initial movements of each joint in the next frame are determined based on the human motion observation data and the motion planning sequence. The initial motion of each joint in the next frame is subjected to a safety check, which includes at least one of motion validity check, joint limit check, inter-frame step size check and self-collision check. If the initial action of each joint in the next frame passes the verification, the initial action of each joint in the next frame is determined as the target action of each joint in the next frame. If the initial motion verification of any of the joints fails in the next frame, the target motion of each joint in the previous frame is determined as the target motion of each joint in the next frame.
3. The method according to claim 2, characterized in that, The determination of the initial movement of each joint in the next frame based on the human motion observation data and the motion planning sequence includes: Based on the reference posture, the torso orientation data, shoulder line direction data, and hip position data in the human motion observation data, the target rotation angle of the humanoid robot's torso is determined. Taking the target action of the humanoid robot's waist joint in the previous frame as the starting point, the constraint is that the limit range of the waist joint and the deviation between the actual rotation angle of the torso and the target rotation angle are less than a preset deviation. The target rotation angle is the optimization target. The action of the waist joint is optimized and solved to obtain the initial action of the waist joint.
4. The method according to claim 2, characterized in that, The determination of the initial movement of each joint in the next frame based on the human motion observation data and the motion planning sequence includes: Given the initial movement of the humanoid robot's waist joint, the target rotation angle of the humanoid robot's head is determined based on the reference posture, the initial movement of the humanoid robot's waist joint, and the head orientation angle in the human motion observation data. Taking the target action of the humanoid robot's neck joint in the previous frame as the starting point, the constraint is that the limit range of the neck joint and the deviation between the actual rotation angle of the head and the target rotation angle are less than a preset deviation, and the target rotation angle is the optimization target. The action of the neck joint is optimized and solved to obtain the initial action of the neck joint. Given the initial movement of the lumbar joint, the target positions of the humanoid robot's hand and elbow joints are determined based on the initial movement of the lumbar joint, the hand tissue position, and the elbow joint position in the human motion observation data. The hand tissue includes the wrist or hand. Taking the target action of the humanoid robot's arm joint in the previous frame as the starting point, and using the arm joint limit range of the humanoid robot as a constraint, the first deviation between the actual position of the humanoid robot's hand tissue and the target position of the hand tissue being less than a preset deviation, and the second deviation between the actual position of the humanoid robot's elbow and the target position of the elbow joint being less than a preset deviation, the target position of the hand tissue being the primary optimization objective and the target position of the elbow joint being the secondary optimization objective, the action of the arm joint is optimized and solved to obtain the initial action of the arm joint.
5. The method according to claim 2, characterized in that, The determination of the initial movement of each joint in the next frame based on the human motion observation data and the motion planning sequence includes: Acquire key point information of hand tissue from human motion observation data, and determine the target posture of hand tissue based on the key point information of hand tissue, wherein the hand tissue includes the wrist or hand; Taking the target action of the wrist joint of the humanoid robot in the previous frame as the starting point, and taking the wrist joint limit range of the humanoid robot and the deviation between the current posture of the humanoid robot's hand tissue and the target posture of the hand tissue being less than a preset deviation as constraints, and taking the target posture of the hand tissue as the optimization target, the action of the wrist joint is optimized and solved to obtain the initial action of the wrist joint. Alternatively, the hand tissue target posture can be decomposed into a first rotation angle about the forearm axis of the anthropomorphic robot and a second rotation angle perpendicular to the forearm axis; Starting from the target action of the wrist joint's rotation axis in the previous frame, and taking the limit range of the humanoid robot's rotation axis and the deviation between the actual rotation angle of the rotation axis and the first rotation angle being less than a preset deviation as constraints, with the first rotation angle as the optimization target, the action of the rotation axis is optimized and solved to obtain the initial action of the rotation axis. Starting from the target motion of the pitch axis and / or yaw axis of the wrist joint in the previous frame, and taking the limit range of the pitch axis and / or yaw axis of the humanoid robot, the deviation between the actual rotation angle of the pitch axis and / or the yaw axis and the second rotation angle as less than a preset deviation as a constraint, and the second rotation angle as the optimization target, the motion of the pitch axis and / or the yaw axis is optimized to obtain the initial motion of the pitch axis and / or the yaw axis.
6. The method according to claim 2, characterized in that, The determination of the initial movement of each joint in the next frame based on the human motion observation data and the motion planning sequence includes: Based on the position data of the wrist key points and the position data of the palm and finger root key points of the humanoid robot in the human motion observation data, the palm coordinate system is determined. The first position coordinates of each of the finger key points in the human body observation data are transformed into the second position coordinates of the palm coordinate system; The target finger posture is determined based on the second coordinate position of each of the aforementioned key finger points; Taking the target action of the humanoid robot's hand joint in the previous frame as the starting point, the deviation between the hand joint's limiting range and the actual posture of the finger key points and the target posture of the finger is less than a preset deviation as a constraint, and the target posture of the finger as the optimization target, the action of the hand joint is optimized and solved to obtain the initial action of the hand joint.
7. The method according to claim 1, characterized in that, The step of driving the humanoid robot to move according to the target movements of each joint includes: The type of the humanoid robot is determined according to the task instructions; The target instruction adapter is determined based on the type; The target instruction adapter is used to convert the target actions of each joint into instructions to obtain the control instructions for each joint. The humanoid robot is driven to move according to the control commands of each joint.
8. The method according to claim 1, characterized in that, The process of acquiring human motion observation data based on the operator's human motion images includes: Foreground segmentation is performed on the human motion image to obtain the target image of the operator; The target image is input into the human posture model to determine the three-dimensional key points corresponding to each joint of the operator; The human motion observation data is determined based on the three-dimensional key points of each joint.
9. The method according to claim 1, characterized in that, After driving the humanoid robot to move according to the target movements of each joint, the method further includes: Teaching data is generated based on the human motion observation data, the target movements of each joint, the actual motion state of the humanoid robot, and environmental information. This teaching data is used for different types of humanoid robots to imitate and learn.
10. A humanoid robot, characterized in that, The anthropomorphic robot includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the humanoid robot to perform the method as described in any one of claims 1 to 9.
