Teleoperation method and robot

Abstract

The invention provides a teleoperation method and a robot. The method comprises the following steps: receiving operator attitude data sent by an inertial measurement unit IMU system; according to the operator attitude data, target attitude data of the humanoid robot are determined, and the target attitude data comprise the tail end target pose and the target arm angle of the seven-degree-of-freedom mechanical arm; the arm angle is a dihedral angle between a plane where the shoulder part, the elbow part and the wrist part of the seven-degree-of-freedom mechanical arm are located and a target pose and a target arm angle surface of a reference flat tail end; according to the target posture data, motion parameters of all joints of the humanoid robot are determined; and generating a control instruction according to the motion parameters. By means of the method, the uncertainty of null-space motion can be effectively eliminated, and the robot motion is more personified.

Description

Technical Field

[0001] This application belongs to the field of robotics technology, and in particular relates to a teleoperation method and a robot. Background Technology

[0002] Teleoperation is a technology that allows human operators to remotely control robots in real time. Teleoperation is responsible for transmitting human actions or intentions in real time, while inverse kinematics is responsible for translating these intentions into angles or velocities that the robot's joints can execute. A 7-DOF robotic arm has motion redundancy compared to a 6-DOF arm, meaning that theoretically, there are infinitely many combinations of joint angles for the same end-effector pose. In traditional inverse kinematics (IK) methods, only the end-effector pose is constrained; the robot's elbow can move freely without changing the end-effector pose (zero-space motion). This leads to non-unique IK solutions and may produce unnatural or joint-constraint configurations. Summary of the Invention

[0003] In view of this, this application provides a teleoperation method and a robot that can effectively eliminate the uncertainty of zero-space motion and make the robot's motion more human-like.

[0004] In a first aspect, this application provides a teleoperation method for controlling a humanoid robot with a 7-DOF robotic arm, the teleoperation method comprising: The system receives operator posture data sent by an inertial measurement unit (IMU) system, wherein the IMU system includes multiple IMUs installed at different parts of the operator's body, and the operator posture data is obtained based on raw data collected by the multiple IMUs. Based on the operator's posture data, the target posture data of the humanoid robot is determined, wherein the target posture data includes the end target pose and target arm angle of the 7-DOF robotic arm, and the arm angle is the dihedral angle between the plane containing the shoulder, elbow and wrist of the 7-DOF robotic arm and the reference plane. Based on the target posture data, determine the motion parameters of each joint of the humanoid robot; Based on the motion parameters, control commands are generated, wherein the control commands are used to drive the various joints of the humanoid robot to perform actions so that the humanoid robot adopts a posture consistent with that of the operator.

[0005] In this application, by simultaneously calculating the joint motion parameters of the humanoid robot based on the end-effector pose and target arm angle of the 7-DOF robotic arm, the solution of the joint motion parameters of the 7-DOF robotic arm can be made unique, thereby eliminating the uncertainty of zero-space motion and making the humanoid robot's motion more human-like.

[0006] Secondly, this application provides a robot, comprising: trunk; A 7-DOF robotic arm, wherein the 7-DOF robotic arm is connected to the torso; Memory and processor, with memory coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the robot to perform the teleoperation method described in the first aspect above.

[0007] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed on a processor, causes a robot to perform the teleoperation method provided in the first aspect above.

[0008] Fourthly, this application provides a computer program product that, when run on a robot, causes the robot to perform the teleoperation method provided in the first aspect above.

[0009] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the teleoperation method provided in the embodiments of this application.

[0012] Figure 2 This is a schematic diagram of the robot provided in the embodiments of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] The relevant technologies include two approaches: 1. See-through VR Teleoperation Technology: This system consists of a VR headset, VR controllers, and a robot control unit. The VR headset has multiple built-in cameras with see-through capabilities, allowing it to capture real-time images of the operating environment and present them to the operator for observation of the work scene. The VR controllers, as the core interaction device, have built-in motion sensors to capture wrist translation and rotation movements, and control hand opening and closing via trigger buttons. The motion data collected by the controllers and the image data from the headset are processed synchronously and transmitted to the robot control unit to generate corresponding joint drive commands, achieving teleoperation control.

[0020] 2. Optical Motion Capture Teleoperation Technology: This system consists of multiple high-definition optical cameras (typically 6–24), motion capture suits / gloves with optical markers, a data processing server, and a robot control unit. The operator wears a full-black motion capture suit and specialized gloves, with more than 50 gray-white reflective markers affixed to the clothing and gloves. Optical cameras (such as the OptiTrack system) distributed around the field emit blue light with sub-millimeter precision and capture the reflected signals from the markers, calculating the three-dimensional spatial coordinates and motion trajectory of each marker based on triangulation principles. The data processing server fuses and calibrates the data from multiple cameras, constructs a full-body human motion model, and converts it into robot joint parameters using a motion mapping algorithm, ultimately transmitting this data to the control unit to drive the robot to perform actions. In humanoid robot teleoperation applications, some solutions require additional specialized finger motion capture attachments to achieve precise hand motion capture.

[0021] However, VR controllers or optical systems can only capture partial limb information and cannot simultaneously acquire multi-degree-of-freedom poses such as chest, head, and upper limbs. Optical solutions are susceptible to occlusion or sudden changes in lighting, resulting in jump points that cause robot movements to jitter and stutter, making it difficult to maintain continuous and natural movements in dynamic environments.

[0022] First, the technical terms used in this application will be explained.

[0023] An Inertial Measurement Unit (IMU) consists of a three-axis accelerometer, a three-axis gyroscope, and an optional three-axis magnetometer. It can measure an object's linear acceleration, angular velocity, and attitude angle information in real time. IMUs have advantages such as small size, low power consumption, and strong resistance to occlusion, and are one of the key sensing technologies in motion capture and attitude estimation systems.

[0024] 7-DOF robotic arm: A 7-DOF robotic arm is a robotic arm designed to mimic the human arm (3 degrees of freedom for the shoulder joint, 1 degree of freedom for the elbow joint, and 3 degrees of freedom for the wrist joint). Because it has 7 independent motion joints, it has 1 more degree of freedom than the 6 degrees of freedom required for precise positioning in three-dimensional space.

[0025] Link: In robotics, a link is a rigid component that connects two adjacent joints on a robotic arm.

[0026] Kinematic Model of a Robotic Arm: A kinematic model is a mathematical model that describes the geometric relationships of a robotic arm's motion. Using modeling methods such as the DH parameter method, a set of parameters (including link parameters and joint parameters) accurately describes the entire geometric structure of the robotic arm; this description is the kinematic model. This model simultaneously includes the ability to solve both forward and inverse kinematics.

[0027] Forward kinematics: Knowing the rotation angle of each joint, the final position and orientation of the end effector are calculated using a geometric model.

[0028] Inverse kinematics: Given the target position and orientation that the end effector is expected to achieve, it is the inverse kinematics that solves for the joint angles required to achieve that orientation.

[0029] Figure 1 The flowchart shown is a teleoperation method provided in an embodiment of this application. This teleoperation method is applied to the control unit of a humanoid robot for controlling a humanoid robot with a 7-DOF robotic arm. The teleoperation method includes: Step 101: Receive operator posture data sent by the inertial measurement unit (IMU) system, wherein the IMU system includes multiple IMUs installed at different parts of the operator's body, and the operator posture data is obtained based on the raw data collected by the multiple IMUs.

[0030] Step 102: Determine the target posture data of the humanoid robot based on the operator posture data. The target posture data includes the end-effector target pose and target arm angle of the 7-DOF robotic arm. The arm angle is the dihedral angle between the plane containing the shoulder, elbow and wrist of the 7-DOF robotic arm and the reference plane.

[0031] Step 103: Determine the motion parameters of each joint of the humanoid robot based on the target posture data.

[0032] Step 104: Generate control commands based on the motion parameters, wherein the control commands are used to drive the joints of the humanoid robot to perform actions so that the humanoid robot makes a posture consistent with the operator.

[0033] In this embodiment, the operator first wears multiple IMU sensors (distributed in the chest, head, upper arm, forearm, etc.). The IMU system includes these IMU sensors worn by the operator and a main processor. Each IMU outputs raw data in real time, including three-axis acceleration, three-axis angular velocity, and three-dimensional attitude quaternions. The main processor synchronously samples all raw data through a unified clock trigger and soft timestamp alignment, ensuring the temporal consistency of the joint poses at the same time.

[0034] Considering the differences in robot configurations (number of joints, kinematic chain length, joint limit range) and operator body size (such as height, arm length, shoulder width, etc.), a human coordinate system (Base Frame) is defined based on the operator's chest IMU pose. Similarly, in the robot's kinematic model, a robot coordinate system is defined based on the chest links. The operator assumes a standard pose (such as a T-shape). The relative positions and poses of each IMU (shoulder IMU, elbow IMU, wrist IMU) relative to the chest IMU are recorded. Again, the robot's link lengths and joint limits are known model parameters. Based on the recorded relative positions and poses of each IMU relative to the chest IMU, the operator's arm length ratio is calculated. For example, operator forearm length / upper arm length = proportionality coefficient P. This proportionality coefficient P is applied to the humanoid robot's kinematic model. If the humanoid robot's upper arm length is L1, then the mapped target forearm length is L1*P. In this way, regardless of the operator's height, weight, or build, the proportions of their movements will be maintained and mapped onto the robot, ensuring the naturalness of the movements.

[0035] Each IMU sensor acquires and outputs raw data in real time, including three-axis acceleration, three-axis angular velocity, and preliminarily calculated three-dimensional attitude quaternions. The humanoid robot receives operator attitude data sent by the Inertial Measurement Unit (IMU) system. To address issues such as high-frequency noise, transient jumps, and insufficient sampling rate in the raw IMU data, an Extended Kalman Filter (EKF) is employed for real-time smoothing. First, a state-space model containing position, velocity, attitude quaternions, and angular velocity is constructed, and a "prediction-update" mechanism is used to filter the IMU data in real time. In the case of missing velocity or angular velocity observation signals, estimation can be performed using the difference between adjacent measurements, and a residual-based compensation mechanism is introduced. Combined with dead-zone thresholding and adaptive gain adjustment, measurement noise is effectively suppressed, and trajectory jitter is reduced. To address the limited frequency of the raw IMU output, a high-frequency prediction mechanism is further introduced into the filtering framework. This mechanism can generate interpolated data within the IMU sampling interval, achieving continuous prediction of attitude and position, thereby ensuring the smoothness of the trajectory in the time dimension. Furthermore, the attitude estimation process employs a quaternion-based update method combined with rotation vectors, avoiding singularities and jumps that may occur in Euler angle representation. Finally, the filtered and interpolated raw data is sent to the humanoid robot as the operator's attitude data. This process ensures the naturalness and continuity of the robot's movements in space from the data source, effectively overcoming the stuttering caused by IMU noise and data abrupt changes. Overall, it significantly improves the stability and robustness of trajectory generation, as well as the system's real-time response capability in dynamic environments.

[0036] The target pose data includes the end-effector target pose and target arm angle of the 7-DOF robotic arm, and may also include the chest pose of the humanoid robot. The reference plane can be formed by the first joint rotation axis and the shoulder-wrist line. The arm angle is a dihedral angle between the shoulder-elbow-wrist plane and the reference plane, describing the rotational position of the elbow about the shoulder-wrist line. Motion parameters include target joint angles; control commands are used to drive the various joints of the humanoid robot to adjust to the target joint angles, so that the humanoid robot makes a pose consistent with the operator's. The end-effector target pose is the desired pose reached by the end of the robotic arm, and the target arm angle is the desired arm angle reached by the robotic arm.

[0037] By having the operator assume a standard pose (such as a T-shape), the human coordinate system and the robot coordinate system are aligned. Subsequently, the pose changes of the operator's chest IMU will be directly mapped to the pose changes of the robot's chest base. The end effector target pose includes the target position (p_ee) and the target posture (R_ee). Using the pose data of the operator's upper arm IMU and forearm IMU (operator pose data), combined with the relative positions and postures of each IMU (shoulder IMU, elbow IMU, wrist IMU) relative to the chest IMU, the three-dimensional position of the operator's wrist in the human coordinate system is calculated. Through scaling by a factor P, the three-dimensional position of the operator's wrist in the human coordinate system is mapped to the robot's motion space, obtaining the target position (p_ee) of the robot's end effector in the robot coordinate system. The pose rotation matrix of the wrist is directly measured by the operator's wrist IMU (or data glove). Considering the initial alignment between the operator and the robot's base coordinate systems, the pose rotation matrix, either directly or after fine-tuning, is used as the target posture (R_ee) of the robot's end effector.

[0038] For the operator, a unique plane is determined based on the actual positions of the shoulder, elbow, and wrist in space (calculated from the operator's posture data). The operator's arm angle is calculated based on this plane, and the calculated operator's arm angle is scaled by a scaling factor P and directly used as the robot's target arm angle.

[0039] In some embodiments, determining the motion parameters of each joint of the humanoid robot based on the target posture data includes: constructing a positive kinematics function, wherein the positive kinematics function includes a function of the end-effector pose of the 7-DOF robotic arm with respect to the joint angles and a function of the arm angle with respect to the joint angles; Calculate the full-dimensional Jacobian matrix of the 7-DOF manipulator. The full-dimensional Jacobian matrix is ​​used to describe the linear relationship between the end-effector pose change and arm angle change of the 7-DOF manipulator and the joint angle change of the 7-DOF manipulator. Singular value decomposition is performed on the full-dimensional Jacobian matrix to obtain multiple singular values; Minimum constraints are applied to each of the singular values ​​to obtain the damped full-dimensional Jacobian matrix. Obtain the initial values ​​for the joint angle iteration; Based on the initial value of the joint angle iteration, the positive kinematic function, the damped full-dimensional Jacobian matrix, the end-effector pose, and the target arm angle, the target joint angle of the 7-DOF robotic arm is calculated using a numerical iteration method.

[0040] In this matrix, the end-effector pose is a function of the joint angles, and the arm angle is also a function of the joint angles. The full-dimensional Jacobian matrix J describes the linear relationship between the changes in the end-effector pose and arm angle of a 7-DOF manipulator and the changes in the joint angles of the 7-DOF manipulator. The full-dimensional Jacobian matrix J of a 7-DOF manipulator is a 7x7 square matrix. The first six rows are the standard Jacobian matrix for a manipulator. The last row consists of the partial derivatives of the arm angle with respect to the joint angles, calculated through numerical difference, for example, by slightly perturbing each joint q_i and observing the change in arm angle. For ease of calculation, these partial derivatives are calculated using small-step numerical difference: ;in It represents the differential step size (the change in joint angle). This represents a 7-dimensional unit vector where only the i-th element is 1. q0 represents the current joint angle vector. φ(q) is the arm angle function, which takes the joint angle vector q as input and outputs the arm angle value.

[0041] When the robotic arm extends, the Jacobian matrix J decreases in rank, and inversion fails, causing the calculated change in joint angle to approach infinity. Therefore, singular value decomposition can be performed on this full-dimensional Jacobian matrix. Minimum constraint is applied to each singular value. Obtain the damped full-dimensional Ajaxometric matrix Σ is a diagonal matrix whose diagonal elements σ i These are called singular values, and their magnitudes represent the ability to move in that direction. A very small σ i This means approaching a singular state. For each singular value, calculate: σ i = max(σ) i , σ min ), where σ min It is a manually set minimum threshold. Damping is applied only in directions with very weak motion (small singular values) to limit its velocity, while maintaining precise control in normal directions. This avoids the drawback of traditional methods that lose accuracy in all directions. The forward kinematic function can be constructed using the DH parameter method.

[0042] Numerical iterative methods require an initial value for joint angle iteration. A poor initial value for joint angle iteration can lead to slow convergence or non-convergence. For example, obtaining the initial value for joint angle iteration includes: The kinematic model of the 7-DOF robotic arm is simplified to obtain a simplified model, wherein the degrees of freedom of the simplified model are lower than those of the kinematic model. Based on the simplified model, for the end-effector pose and the target arm angle, an approximate solution for the joint angle is calculated using a geometric method; Based on the mapping relationship between the simplified model and the kinematic model, the approximate solution of the joint angle is converted into the candidate solution of the joint angle of the kinematic model; When a candidate solution for the joint angle exists, the candidate solution for the joint angle is used as the initial value for the joint angle iteration; When there are multiple candidate solutions for the joint angle, one of them is selected as the initial value for the joint angle iteration according to a preset strategy.

[0043] Specifically, the simplified model can be constructed using a planar projection method: the complex spatial relationship between the three key points—shoulder, elbow, and wrist—is projected onto a best-fit plane defined by the line connecting the shoulder and wrist points and the direction of gravity, thereby constructing a 2-DOF equivalent link model (simplified model) within this plane. Alternatively, a joint locking method can be used: based on the robot configuration and target posture, one or more joints (such as some rotational degrees of freedom of the shoulder) are pre-specified as fixed values ​​or placed in neutral positions within their range of motion, and only the remaining key degrees of freedom are analytically solved.

[0044] It is important to emphasize that the reduction of degrees of freedom in the simplified model is a phased process. The low-dimensional approximate solution obtained through the above method must be transformed and filled into the full 7-DOF space through a mapping relationship between the simplified model and the kinematic model, thereby obtaining candidate solutions for the joint angles of the kinematic model. This mapping process usually assigns a reasonable default value (such as zero or the current angle value) to the locked or ignored degrees of freedom through projection, thus completing the reconstruction from the low-dimensional solution space to the high-dimensional solution space. This strategy of first reducing dimensionality and then increasing it, at the cost of acceptable accuracy loss, achieves a significant increase in computational speed. The geometric method can be calculated using inverse trigonometric functions.

[0045] Optionally, the preset strategy includes any one of the following: Minimum motion strategy: Calculate the Euclidean norm of each candidate joint angle solution and the current joint state of the 7-DOF robot, and select the candidate joint angle solution with the smallest Euclidean norm. Joint limiting strategy: Select candidate solutions for joint angles where all joint angles are far from the physical limits of the joint; Self-collision avoidance strategy: Select candidate joint angles whose positive kinematic results do not lead to the risk of self-collision between links.

[0046] In some embodiments, the target joint angles of the 7-DOF robotic arm are calculated using a numerical iterative method based on the initial values ​​of the joint angle iterations, the positive kinematic function, the damped full-dimensional Jacobian matrix, the end-effector pose, and the target arm angles. This includes: The initial value of the joint angle iteration is used as the current joint angle of the 7-DOF robotic arm; Based on the current joint angles of the 7-DOF robotic arm and the positive kinematics function, the current end-effector pose and current arm angle of the 7-DOF robotic arm are calculated. Calculate the first error between the current end-effector pose and the end-effector target pose; Calculate the second error between the current arm angle and the target arm angle; If the first error and the second error satisfy the preset iteration termination condition, then the current joint angle of the 7-DOF robotic arm is determined as the target joint angle. If the first error and the second error do not meet the iteration termination condition, then the joint angle change that can reduce the first error and the second error is calculated using the damped full-dimensional Ajaxometry matrix. The current joint angles of the 7-DOF manipulator are updated based on the change in joint angles, and the process returns to the step of calculating the current end-effector pose and current arm angle of the 7-DOF manipulator based on the current joint angles of the 7-DOF manipulator and the positive kinematic function.

[0047] Specifically, each iteration of the numerical iterative method updates the current joint angles of the 7-DOF manipulator. In the first iteration, the initial joint angle values ​​are used as the current joint angles of the 7-DOF manipulator. The forward kinematics function is a function of the joint angles. In each iteration, the current joint angles of the 7-DOF manipulator are input into the forward kinematics function, and the function outputs the current end effector pose and current arm angles. The first and second errors can be represented using a quaternion e. e is a 7-dimensional vector containing the first and second errors. In each iteration, the convergence condition is checked: Condition 1 (Success): If the error ||e|| is less than the set threshold, the iteration is successful, and the current joint angle of the 7-DOF robotic arm is determined as the target joint angle.

[0048] Condition 2 (Failure): If the number of iterations exceeds the maximum limit, the iteration fails, the best approximate solution is adopted, and an alarm is triggered.

[0049] If the convergence condition is not met, the joint angle change that reduces the error ||e|| is calculated using the damped full-dimensional Ajaximity matrix. Based on this joint angle change, the current joint angles of the 7-DOF manipulator are updated. Then, using the updated current joint angles, the process returns to the step of calculating the current end-effector pose and current arm angle of the 7-DOF manipulator based on its current joint angles and the positive kinematics function, and continues iterating after that step. For example, the joint angle change is added to the current joint angle q. Get the updated current joint angle Based on the updated current joint angle Using kinematic functions, the new current end-effector pose and new current arm angle of the 7-DOF robotic arm are calculated.

[0050] In some embodiments, the teleoperation method further includes: To detect whether there is a risk of collision between the links of the kinematic model of the 7-DOF robotic arm; Generate constraints for links that pose a collision risk; The calculation of the joint angle change that reduces the first error and the second error using the damped full-dimensional comparability matrix includes: The joint angle change that reduces the first error and the second error is calculated using the constraints and the damped full-dimensional Jacobian matrix.

[0051] Specifically, the system checks for collision risk between any two links in the kinematic model to identify link pairs with collision risk. Constraints are then generated for these links. These constraints can be added as a penalty term to the positive kinematic function. When a collision risk occurs, a slight sacrifice in end-effector positioning accuracy is made to ensure overall safety. Using the constraints and the damped full-dimensional Ajaxamethylene matrix, the joint angle change that reduces the first and second errors is calculated. After the robotic arm adjusts its joints according to this joint angle change, no collision risk occurs.

[0052] Optionally, the constraint conditions for generating links with collision risk include: Create a geometry bounding box for each of the links; Determine whether the bounding boxes of the geometric bodies intersect. If two bounding boxes of geometric bodies do not intersect, then the links corresponding to the two bounding boxes of geometric bodies have no risk of collision. If two bounding boxes of geometry intersect, the distance d between the nearest points of the target links is calculated using the 3D mesh models of the target links corresponding to the intersecting bounding boxes of geometry. If the distance d is less than the safety threshold ε, then the target link is determined to be at risk of collision. Constraints are generated for the target links, wherein the distance d between the nearest points of the target links is greater than a safety threshold ε.

[0053] This involves creating a simple geometric bounding box (AABB) for each link of the robot (e.g., upper arm, forearm). Collision detection is simplified to determining whether two geometric shapes intersect. This is a very fast computation. If two bounding boxes do not intersect, the links they represent are absolutely unlikely to collide, and the detection ends. More refined detection is performed only on link pairs whose bounding boxes intersect: using a 3D mesh model of the links, the distance *d* between their nearest points is calculated. A safety threshold *ε* is set. If *d* < *ε*, a collision risk is considered. The standard IK problem aims to minimize the error *e*. When a collision risk is detected, the IK problem transforms into a constrained optimization problem.

[0054] In some embodiments, detecting whether there is a risk of collision between the links of the kinematic model of the 7-DOF manipulator includes: Excluding parent-child links and adjacent links in the kinematic model, detect whether there is a collision risk between the remaining links in the kinematic model.

[0055] In this embodiment, the self-collision detection calculates the interference of the geometric model (such as bounding box, mesh) of the link in three-dimensional space. During the collision risk detection process, inherent connection pairs such as parent-child links and adjacent links are automatically ignored, and only non-connected links are retained for accurate interference determination, which can avoid unnecessary computation.

[0056] Operator posture data is transmitted to the robot control unit in real time via a high-speed wireless link (such as Wi-Fi 6 or BLE 5.0). To ensure the integrity and stability of data transmission under complex electromagnetic environments and high real-time requirements, this system is designed with a multi-layered communication fault-tolerance mechanism, specifically including: CRC32 Cyclic Redundancy Check: When the sending end packages each data frame, it attaches a checksum generated by the CRC32 algorithm; the receiving end recalculates the checksum and compares it upon unpacking. If the check fails, the data frame is immediately discarded to prevent erroneous data from entering the inverse kinematics solution and control loop. This mechanism requires only 4 bytes of overhead, can efficiently detect common transmission errors, and is suitable for low-latency real-time communication.

[0057] Timeout retransmission mechanism: If the receiving end does not receive a complete data frame within a set time window, a timeout detection is triggered, and a retransmission request is sent to the sending end via the control protocol. This mechanism can effectively cope with transient interference and data packet loss, improving the reliability of the system in weak signal environments.

[0058] Watchdog monitoring module: A communication watchdog is set up in the robot control unit to continuously monitor the data reception and processing status. Once a prolonged period without valid data or a blocked processing thread is detected, the system automatically triggers safety protection logic, causing the robot to enter a safe posture or emergency stop state to prevent the risk of loss of control due to communication link abnormalities.

[0059] Redundant buffering and data smoothing: The receiver is designed with a circular buffer and interpolation algorithm to smooth continuous data, so as to alleviate the sudden changes in data delay caused by network jitter and ensure the continuity and naturalness of the trajectory in inverse kinematics solution and motion control.

[0060] Multi-threaded control mechanism: The system adopts a multi-threaded architecture, with independent threads responsible for key tasks such as end-effector pose publishing, watchdog monitoring, timer triggering, data preprocessing, inverse kinematics solving, and post-processing. Thread locks and other mechanisms ensure thread synchronization and data consistency, meeting high real-time control requirements.

[0061] Low-latency video streaming solution: For remote IMU teleoperation scenarios, the system proposes an online, low-latency, bandwidth-resistant video streaming solution. It employs an SRS-based streaming architecture to achieve high real-time performance and transmission stability. This solution is used to transmit multiple video streams (including end-effector cameras, third-view cameras, and environmental monitoring cameras) between the operator and robot ends, ensuring visual feedback remains available even under low bandwidth or network fluctuation conditions. The system architecture is as follows: Video acquisition end: The robot's vision module acquires RGB or RGB-D images in real time and uses hardware acceleration for H.264 / H.265 encoding, keeping the encoding latency within 10–30 ms.

[0062] Push Streaming Module: The encoded video stream is pushed to the SRS server via RTMP or WebRTC protocol, and SRS performs low-latency forwarding and adaptive bitrate control.

[0063] Receiver: The operator pulls the stream in real time through the WebRTC channel, with the end-to-end latency stably controlled within 150ms, and can adaptively adjust the video bitrate and GOP length according to the current network bandwidth.

[0064] Scalability: Supports parallel transmission of multiple video streams. The SRS server achieves high concurrency processing and low resource consumption through a coroutine architecture.

[0065] As can be seen from the above, in the present application, by simultaneously calculating the joint motion parameters of the humanoid robot based on the end target pose and target arm angle of the 7-DOF robotic arm, the solution of the joint motion parameters of the 7-DOF robotic arm can be made unique, thereby eliminating the uncertainty of zero-space motion and making the humanoid robot's motion more human-like.

[0066] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0067] Figure 2 This is a schematic diagram of the structure of a robot provided in one embodiment of this application. Figure 2 As shown, the robot 2 in this embodiment includes: a torso, a 7-DOF robotic arm connected to the torso, and at least one processor 20. Figure 2 Only one is shown in the diagram), memory 21, and computer program 22 stored in the memory 21 and executable on at least one processor 20. When the processor 20 executes the computer program 22, it causes the robot to perform the steps of the teleoperation method.

[0068] The robot 2 described above may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will understand that... Figure 2 This is merely an example of robot 2 and does not constitute a limitation on robot 2. It may include more or fewer parts than shown in the figure, or combine certain parts, or different parts, such as input / output devices, network access devices, etc.

[0069] The processor 20 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0070] In some embodiments, the aforementioned memory 21 may be an internal storage unit of the robot 2, such as a hard disk or memory of the robot 2. In other embodiments, the aforementioned memory 21 may be an external storage device of the robot 2, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the robot 2. Furthermore, the aforementioned memory 21 may include both internal storage units and external storage devices of the robot 2. The aforementioned memory 21 is used to store operating systems, applications, bootloaders, data, and other programs, such as the program code of the aforementioned computer programs. The aforementioned memory 21 may also be used to temporarily store data that has been output or will be output.

[0071] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0073] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the various method embodiments described above.

[0074] This application provides a computer program product that, when run on a robot, causes the robot to perform the steps described in the various method embodiments above.

[0075] This application provides a chip system including a processor, which is used to call and run a computer program from a memory, causing a robot equipped with the chip system to perform the steps in the various method embodiments described above.

[0076] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a robot-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The robot-readable medium can include at least: any entity or device capable of carrying computer program code to a robot, a recording medium, a robot memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, robot-readable media cannot be electrical carrier signals or telecommunication signals.

[0077] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0078] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of robot software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0079] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0080] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A teleoperation method for controlling a humanoid robot having a 7-DOF manipulator, characterized by, The teleoperation method comprises: receiving operator posture data transmitted by an inertial measurement unit (IMU) system, wherein the IMU system comprises a plurality of IMUs arranged at different parts of the body of an operator, and the operator posture data is obtained based on raw data collected by the plurality of IMUs; determining target posture data of the humanoid robot according to the operator posture data, wherein the target posture data comprises a target end pose of the 7-DOF arm and a target arm angle, and the arm angle is a dihedral angle between a plane in which a shoulder, an elbow and a wrist of the 7-DOF arm are located and a reference plane; determining motion parameters of each joint of the humanoid robot according to the target posture data; generating control instructions according to the motion parameters, wherein the control instructions are used to drive each joint of the humanoid robot to perform an action, so that the humanoid robot assumes a posture consistent with the operator.

2. The teleoperation method of claim 1, wherein, The determination of the motion parameters of each joint of the humanoid robot according to the target posture data comprises: constructing a forward kinematics function, wherein the forward kinematics function comprises a function of the end pose of the 7-DOF arm with respect to joint angles and a function of the arm angle with respect to joint angles; calculating a full-dimensional Jacobian matrix of the 7-DOF arm, wherein the full-dimensional Jacobian matrix is used to describe a linear relationship between a change in the end pose of the 7-DOF arm and a change in the arm angle and a change in the joint angles of the 7-DOF arm; performing singular value decomposition on the full-dimensional Jacobian matrix to obtain a plurality of singular values; performing minimum value constraint on each singular value to obtain a damped full-dimensional Jacobian matrix; obtaining an initial value of the joint angles; calculating the target joint angles of the 7-DOF arm by using a numerical iteration method based on the initial value of the joint angles, the forward kinematics function, the damped full-dimensional Jacobian matrix, the target end pose and the target arm angle.

3. The teleoperation method of claim 2, wherein, The calculation of the target joint angles of the 7-DOF arm by using a numerical iteration method based on the initial value of the joint angles, the forward kinematics function, the damped full-dimensional Jacobian matrix, the target end pose and the target arm angle comprises: taking the initial value of the joint angles as current joint angles of the 7-DOF arm; calculating a current end pose and a current arm angle of the 7-DOF arm according to the current joint angles of the 7-DOF arm and the forward kinematics function; calculating a first error between the current end pose and the target end pose; calculating a second error between the current arm angle and the target arm angle; if the first error and the second error satisfy a preset iteration end condition, determining the current joint angles of the 7-DOF arm as the target joint angles; if the first error and the second error do not satisfy the iteration end condition, calculating a change in the joint angles that can reduce the first error and the second error by using the damped full-dimensional Jacobian matrix. updating the current joint angles of the 7-DOF robot arm based on the joint angle change amount, and returning to perform the step of calculating the current end pose and the current arm angle of the 7-DOF robot arm according to the current joint angles of the 7-DOF robot arm and the forward kinematics function.

4. The teleoperation method of claim 2, wherein, The obtaining of the joint angle iteration initial value comprises: simplifying the kinematics model of the 7-DOF robot arm to obtain a simplified model, the degree of freedom of the simplified model being lower than that of the kinematics model; based on the simplified model, calculating a joint angle approximate solution for the end target pose and the target arm angle by using a geometric method; according to the mapping relationship between the simplified model and the kinematics model, converting the joint angle approximate solution into a joint angle candidate solution of the kinematics model; when there is one joint angle candidate solution, taking the joint angle candidate solution as the joint angle iteration initial value; when there are multiple joint angle candidate solutions, selecting one from the multiple joint angle candidate solutions as the joint angle iteration initial value according to a preset strategy.

5. The teleoperation method of claim 4, wherein, The preset strategy comprises any one of the following: a minimum motion amount strategy: calculating the Euclidean norm of each joint angle candidate solution and the current joint state of the 7-DOF robot arm, and selecting the joint angle candidate solution with the minimum Euclidean norm; a joint limit strategy: selecting a joint angle candidate solution in which all joint angles are away from the joint physical limit; a self-collision avoidance strategy: selecting a joint angle candidate solution that does not cause a self-collision risk between links in the forward kinematics result.

6. The teleoperation method of claim 3, wherein, The teleoperation method further comprises: detecting whether there is a collision risk between each link of the kinematics model of the 7-DOF robot arm; generating a constraint condition for the link with the collision risk; The calculating of the joint angle change amount that can reduce the first error and the second error by using the damped full-dimensional Jacobian matrix comprises: calculating the joint angle change amount that can reduce the first error and the second error by using the constraint condition and the damped full-dimensional Jacobian matrix.

7. The teleoperation method of claim 6, wherein, The generating of the constraint condition for the link with the collision risk comprises: creating a geometric bounding box for each link; judging whether the geometric bounding boxes intersect, if two geometric bounding boxes do not intersect, the links corresponding to the two geometric bounding boxes have no collision risk; if two geometric bounding boxes intersect, using the 3D mesh model of the target links corresponding to the two intersecting geometric bounding boxes to calculate the distance d between the closest points of the target links; if the distance d is less than a safety threshold ε, it is determined that the target links have a collision risk; generating a constraint condition for the target links, the constraint condition being that the distance d between the closest points of the target links is greater than the safety threshold ε.

8. The teleoperation method of claim 7, wherein, The detecting of whether there is a collision risk between each link of the kinematics model of the 7-DOF robot arm comprises: excluding the parent-child links and adjacent links in the kinematics model, and detecting whether there is a collision risk between each link remaining in the kinematics model.

9. A robot, characterized in that comprises: a torso; 7 degrees of freedom robotic arm connected to the torso; a memory coupled with the processor; the memory stores program instructions that, when executed by the processor, cause the robot to perform the teleoperation method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, the computer readable storage medium stores a computer program that, when executed on a processor, causes a robot to perform the teleoperation method of any one of claims 1 to 8.

Images