Humanoid robot teleoperation control system

Abstract

The invention provides a humanoid robot teleoperation control system, and relates to the technical field of humanoid robots, and the system comprises a tail end teleoperation control module which is used for collecting first information and second information; the first information comprises gesture information and / or finger tail end posture information of a user, and the second information comprises spatial position information and / or posture information of a tail end teleoperation control module; the actuator mapping module is used for generating a first operation instruction of an end actuator corresponding to the humanoid robot according to the first information and sending the first operation instruction to the humanoid robot; and the data processing module is used for generating a second operation instruction of the two arms corresponding to the humanoid robot according to the second information and sending the second operation instruction to the humanoid robot. The control precision and flexibility of the teleoperation of the humanoid robot can be effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of humanoid robots, and in particular to a remote operation control system of a humanoid robot. Background Art

[0002] Teleoperation technology refers to a technology that enables operators to remotely control humanoid robots to perform tasks through a remote control system.

[0003] Traditional teleoperation methods generally convert the operator's actions into control signals for humanoid robots through controllers (such as joysticks, buttons, etc.). This method is usually more intuitive, but has poor control accuracy and flexibility. When faced with more complex or dangerous tasks, the operator's operations may be uncertain and have safety risks.

[0004] In summary, how to improve the control accuracy and flexibility of humanoid robot teleoperation is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiment of the present application provides a humanoid robot teleoperation control system, which can effectively improve the control accuracy and flexibility of the humanoid robot teleoperation.

[0006] In some embodiments, the humanoid robot teleoperation control system comprises an end teleoperation control module, an actuator mapping module, and a data processing module;

[0007] The terminal remote operation control module is used to collect first information and second information; the first information includes the user's gesture information and / or finger terminal posture information, and the second information includes the spatial position information and / or posture information of the terminal remote operation control module;

[0008] The actuator mapping module is used to generate a first operation instruction of the end effector corresponding to the humanoid robot according to the first information, and send the first operation instruction to the humanoid robot;

[0009] The data processing module is used to generate second operation instructions for the two arms corresponding to the humanoid robot according to the second information, and send the second operation instructions to the humanoid robot.

[0010] In some implementations, the actuator mapping module is specifically configured to:

[0011] Converting the first information into standard hand posture information;

[0012] According to the above posture information, the angle data of each joint corresponding to the standard hand is generated;

[0013] Convert the above angle data into the desired joint angle data of the end effector;

[0014] A first operation instruction is generated according to the above-mentioned expected joint angle data.

[0015] In some embodiments, the end effector includes a five-finger dexterous hand assembly; the effector mapping module is specifically used for:

[0016] According to the angle data of each joint corresponding to the standard hand, the expected joint angle data of each joint corresponding to the five-finger dexterous hand assembly is determined.

[0017] In some embodiments, the end effector includes a two-finger gripper assembly; the effector mapping module is specifically configured to:

[0018] According to the angle data of each joint corresponding to the thumb and index finger of a standard hand, the expected joint angle data of each joint corresponding to the two-finger gripper assembly is determined.

[0019] In some embodiments, the data processing module is specifically used to:

[0020] Determining a desired position and / or posture of the end effector based on the second information;

[0021] A second operation instruction is generated according to the desired position and / or posture of the end effector.

[0022] In some embodiments, the data processing module is specifically used to:

[0023] receiving actual position and / or posture data of the end effector fed back by the humanoid robot;

[0024] Determine the desired joint angle data of the two arms according to the actual position and / or posture data of the end effector and the desired position and / or posture of the end effector;

[0025] A second operation instruction is generated according to the desired joint angle data of the two arms.

[0026] In some embodiments, the terminal teleoperation control module includes a brass knuckle virtual reality (VR) device; the brass knuckle VR device includes a left-hand brass knuckle VR device and / or a right-hand brass knuckle VR device.

[0027] In some embodiments, the system further comprises a plurality of VR positioning trackers; the VR positioning trackers can be worn on the waist and / or feet of the user;

[0028] The VR positioning tracker is used to collect third information, and the third information includes spatial position information and / or posture information of the VR positioning tracker.

[0029] In some embodiments, the data processing module is further used to:

[0030] Determine the moving speed and / or angular velocity of the humanoid robot according to the third information collected by each VR positioning tracker;

[0031] A third operation instruction is generated according to the above-mentioned moving speed and / or angular velocity, and sent to the humanoid robot.

[0032] In some embodiments, the system further includes a head mounted display device and a positioning base station;

[0033] The head mounted display device is used to receive and display image data collected by the humanoid robot;

[0034] The above-mentioned positioning base station is used to provide spatial position information and / or posture information to the above-mentioned terminal remote operation control module, head-mounted display device, and VR positioning tracker.

[0035] The humanoid robot remote operation control system provided in the embodiment of the present application can realize fine control of the end effector (including fingers) of the humanoid robot by collecting the user's gesture information and / or finger end posture information through the end remote operation control module. This control method is more intuitive and precise than the traditional joystick, and can more naturally reflect the operating intentions of the user's hands; further, by collecting the spatial position information and / or posture information of the end remote operation control module, the arms of the humanoid robot can be further accurately controlled, so that the humanoid robot can more accurately simulate the user's movements and postures, thereby effectively improving the control accuracy and flexibility of the humanoid robot remote operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] Figure 1 A schematic diagram of the structure of a humanoid robot provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the architecture of a humanoid robot remote control system provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the architecture of another humanoid robot teleoperation control system provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a remote control process of a humanoid robot provided in an embodiment of the present application.

[0041] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] Here, various exemplary embodiments of the present application will be described in detail. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0043] It should be noted that the humanoid robot teleoperation system provided in the embodiments of the present application can be used in the field of humanoid robots, and can also be used in other robot fields except the field of humanoid robots, and the present application does not limit this.

[0044] A humanoid robot is a robot that imitates human appearance and behavior. It usually has dual robotic arms and lower limbs similar to humans, and can perform various human-like movements.

[0045] Optional, key components of a humanoid robot include:

[0046] Dual robotic arms: The robotic arms of humanoid robots usually have multiple degrees of freedom and can simulate various movements of human arms, such as rotation, bending and extension, which enables the robot to perform delicate manipulation tasks such as grasping objects, operating tools, etc.

[0047] Lower limbs: The lower limbs of humanoid robots also have multiple degrees of freedom and can simulate human movements such as walking, running, and jumping.

[0048] Head system: Although the heads of humanoid robots do not always have to imitate the human head structure, they are usually equipped with sensors such as cameras and microphones to obtain environmental information. In addition, some humanoid robots can also be equipped with artificial intelligence technologies such as facial recognition and voice recognition to improve their interactive capabilities and intelligence levels.

[0049] For example, refer to Figure 1 , Figure 1 Schematic diagram of the structure of a humanoid robot provided in an embodiment of the present application. In some embodiments, the humanoid robot includes:

[0050] Head system 1: includes the head structure of the humanoid robot, which can be integrated with sensors, cameras, microphones, etc. for environmental perception and interaction.

[0051] Thoracic system 2: Contains the internal electronic equipment and power source of the humanoid robot, such as motor controller, power distribution unit, etc.

[0052] Robotic Arm System 3: It includes dual robotic arms of a humanoid robot, used to perform various manipulation tasks such as grasping, carrying, etc.

[0053] Dexterous hand system 4: Installed at the end of the robotic arm, it is usually designed with multiple joints and degrees of freedom, allowing it to move the fingers and palm parts in a highly flexible way. This design enables the dexterous hand to simulate various complex movements of the human hand, such as grasping, rotating, pinching, etc.

[0054] Thigh 5: The part connecting the hip joint and the knee joint is the main load-bearing structure of the humanoid robot leg.

[0055] Calf 6: The part that connects the knee and ankle joints and is an important part of leg movement.

[0056] Foot 7: The walking and standing part of a humanoid robot, usually designed with anti-slip and shock-absorbing functions.

[0057] Those skilled in the art will understand that Figure 1 The humanoid robot structure shown in the figure does not constitute a limitation on the humanoid robot. The humanoid robot in this application may include Figure 1 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0058] The application scenarios of the above humanoid robots may include but are not limited to the following aspects:

[0059] Industrial manufacturing: In automated production lines, humanoid robots can replace humans to complete some heavy, dangerous or repetitive tasks, such as carrying heavy objects, assembling parts, etc.

[0060] Service industry: In service industry places such as restaurants and hotels, humanoid robots can act as waiters or receptionists to provide customers with services such as ordering, delivering food, and guiding customers.

[0061] Medical rehabilitation: In the medical field, humanoid robots can be used as auxiliary devices to help patients with rehabilitation training or provide daily care.

[0062] Rescue and exploration: At disaster sites such as earthquakes and fires, humanoid robots can replace humans to enter dangerous areas for search and rescue or exploration.

[0063] Entertainment and Education: In entertainment venues and educational institutions, humanoid robots can serve as performance guests or teaching assistants, providing interesting interactive experiences for audiences and students.

[0064] Teleoperation technology refers to a technology that enables operators to control humanoid robots to perform tasks from a distance through a remote control system. With the continuous expansion of the application fields of humanoid robots, teleoperation technology has been widely used in industry, medical care, exploration and other fields.

[0065] The core goal of teleoperation technology is to improve work efficiency, operation accuracy and safety by reducing human contact with dangerous or complex environments. Existing teleoperation methods can be roughly divided into the following types: traditional teleoperation based on physical controllers, teleoperation based on virtual reality (VR) technology, teleoperation based on brain-computer interface (BCI) technology, etc.

[0066] For example, physical controller-based teleoperation generally converts the operator's actions into humanoid robot control signals through controllers (such as joysticks, buttons, motion capture gloves and other physical devices). This method is usually more intuitive, but the control accuracy and flexibility are poor. When facing more complex or dangerous tasks, the operator's operation may be uncertain and have safety risks.

[0067] The introduction of VR technology has brought a higher sense of immersion and perception to the remote operation of humanoid robots. Through the virtual reality system, operators can not only see the current working environment and status of the humanoid robot, but also get real-time feedback on the movements of the humanoid robot.

[0068] Existing VR teleoperation systems usually provide incremental feedback by feeding back the position of the VR handle in the VR headset, and transmit this position feedback information to the humanoid robot body, so that the end effector of the humanoid robot can track the position of the VR handle in real time. Although this incremental control method can control the operation of the humanoid robot to a certain extent, the following technical problems still exist:

[0069] Human posture matching problem: The incremental control method has high requirements on the operator's body posture, because the end effector of the humanoid robot and the operator's body posture are not always completely consistent, and in some cases, the operator's body posture may cause inaccurate control or misalignment, affecting the accuracy and stability of the operation.

[0070] Stroke control problem: The VR handle used in the current VR teleoperation system mainly controls the opening and closing state of the actuator through buttons, and cannot achieve precise stroke control of the actuator. For example, the current VR handle only supports opening and closing control, lacks control over the precise movement of the robotic arm or gripping device, resulting in limited accuracy of the operation.

[0071] Gripper control problem: For grippers with three or more fingers, the existing VR teleoperation system can only control the opening and closing state of the gripper with a specific trajectory, and cannot achieve independent control of each mechanical finger. This control method cannot cope with complex grasping tasks and reduces the flexibility of operation.

[0072] In view of the above-mentioned technical problems, a humanoid robot teleoperation system is provided in an embodiment of the present application. By collecting the user's gesture information and / or finger end posture information through the terminal teleoperation control module, fine control of the humanoid robot's end effector (including fingers) can be achieved. By collecting the spatial position information and / or posture information of the terminal teleoperation control module, the arms of the humanoid robot can be further controlled, so that the humanoid robot can more accurately simulate the user's movements and postures, thereby effectively improving the control accuracy and flexibility of the humanoid robot teleoperation.

[0073] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0074] Reference Figure 2 , Figure 2 Schematic diagram of the architecture of a humanoid robot teleoperation control system provided in an embodiment of the present application. In some embodiments, the humanoid robot teleoperation control system includes: an end teleoperation control module 201 , an actuator mapping module 202 and a data processing module 203 .

[0075] In some embodiments, the terminal teleoperation control module 201 can be worn on both hands of a user (or referred to as an operator in the following embodiments).

[0076] In some embodiments, the terminal teleoperation control module 201 can be used to collect the first information and the second information.

[0077] Optionally, the first information includes user's gesture information and / or finger end posture information; the second information includes spatial position information and / or posture information of the end remote operation control module 201 .

[0078] The first information may be used to identify the user's finger operation intention, that is, the user's gesture and finger tip posture may reflect the type of operation that the user's finger wants to perform.

[0079] The second information can be used to identify the position and posture of the user's hands, such as identifying the exact position and orientation of the user's hands in three-dimensional space, thereby ensuring the accuracy and safety of the operation of the humanoid robot.

[0080] In some embodiments, the terminal teleoperation control module 201 may include a brass knuckle VR device (also referred to as a brass knuckle VR handle); the brass knuckle VR device may include a left-hand brass knuckle VR device and / or a right-hand brass knuckle VR device. The left-hand brass knuckle VR device is worn on the user's left hand, and the right-hand brass knuckle VR device is worn on the user's right hand.

[0081] Optionally, the brass knuckle VR device may adopt a hand-covering design, which may fit the hand better and provide a more stable and comfortable holding experience compared to a traditional VR handle.

[0082] Optionally, the brass knuckles VR device may be equipped with multiple sensors to track the position and movement of each finger, thereby more accurately identifying various actions performed by the user's fingers, such as grasping, releasing, etc.

[0083] In some embodiments, the brass knuckle VR device can feed back the collected first information to the data processing module 203 in real time, and the data processing module 203 sends the first information to the actuator mapping module 202, or can directly feed back the collected first information to the actuator mapping module 202.

[0084] In some implementations, the brass knuckle VR device may feed back the collected second information to the data processing module 203 in real time.

[0085] Optionally, the above-mentioned humanoid robot teleoperation control system further includes a communication module. The terminal teleoperation control module 201 can use the communication module to feed back the above-mentioned first information and second information to the data processing module 203; or, use the communication module to directly feed back the first information to the actuator mapping module 202.

[0086] Optionally, the above-mentioned communication module can be a wired communication module, or a wireless communication module (such as a Wi-Fi module, a Bluetooth module, etc.), or a dedicated communication protocol module, which is not limited in the embodiments of the present application.

[0087] The actuator mapping module 202 may be used to generate a first operation instruction of the end effector corresponding to the humanoid robot according to the first information, and send the first operation instruction to the humanoid robot.

[0088] In some embodiments, the actuator mapping module 202 can parse the first information to identify the specific operation type that the user wants to perform, such as grabbing, releasing, etc. According to the parsing result, the actuator mapping module 202 generates a first operation instruction corresponding to the end effector of the humanoid robot. Finally, the actuator mapping module 202 sends the generated first operation instruction to the humanoid robot through the communication module, thereby controlling the end effector of the humanoid robot to complete the corresponding action.

[0089] The data processing module 203 is used to generate second operation instructions for the two arms of the humanoid robot according to the second information, and send the second operation instructions to the humanoid robot.

[0090] In some embodiments, the data processing module 203 can parse the second information to identify the desired position and orientation of the humanoid robot's arms in three-dimensional space. Further, based on the parsing results, the data processing module 203 can determine the desired joint angles of the humanoid robot's arms, and generate second operation instructions for the humanoid robot's arms according to the desired joint angles. Finally, the generated second operation instructions are sent to the humanoid robot to control its arms to perform corresponding actions.

[0091] The humanoid robot remote operation control system provided in the embodiment of the present application can realize fine control of the end effector (including fingers) of the humanoid robot by collecting the user's gesture information and / or finger end posture information through the end remote operation control module. This control method is more intuitive and accurate than the traditional joystick, and can more naturally reflect the user's operating intentions of both hands; further, by collecting the spatial position information and / or posture information of the end remote operation control module, the arms of the humanoid robot can be further controlled, so that the humanoid robot can more accurately simulate the user's movements and postures, thereby effectively improving the control accuracy and flexibility of the humanoid robot remote operation.

[0092] Based on the contents described in the above embodiments, in some embodiments, the actuator mapping module 202 is specifically used for:

[0093] The first information is converted into the posture information of a standard hand; based on the posture information, the angle data of each joint corresponding to the standard hand is generated; the angle data is converted into the expected joint angle data of the end effector; based on the expected joint angle data, the first operation instruction is generated.

[0094] In some implementations, the actuator mapping module 202 may receive the first information and convert the first information into a standard hand posture, that is, uniformly map the operator's hand shape to the same standard hand shape.

[0095] In order to uniformly map the hand shapes of different operators to the same standard hand shape, the actuator mapping module 202 can adopt a mapping technology, which can ensure that the hand actuators of the humanoid robot can respond in a consistent and accurate manner regardless of the actual hand shape of the operator.

[0096] The standard hand shape may be predefined and represents the desired posture of the humanoid robot hand actuator. Through mapping, the actuator mapping module 202 can convert the user's gesture and finger tip information into control instructions corresponding to the standard hand shape.

[0097] In some embodiments, to ensure accuracy, the actuator mapping module 202 may use the finger end inverse solution and the weighted average of the gesture data to calculate the joint angle of each finger. The finger end inverse solution refers to inferring the angle of each finger joint from the position of the finger end; and the weighted average is to smooth the gesture data and reduce the influence of noise and error.

[0098] In some embodiments, the end effector may include a five-finger dexterous hand assembly; the actuator mapping module 202 is specifically used for:

[0099] According to the angle data of each joint corresponding to the standard hand, the expected joint angle data of each joint corresponding to the above-mentioned five-finger dexterous hand assembly is determined.

[0100] Among them, each finger of the above-mentioned five-finger dexterous hand can move independently, simulating various movements of human fingers, such as grasping, pinching, rotating, etc.

[0101] In some implementations, the actuator mapping module 202 may map the joint angle data of a standard hand to each joint of the five-finger dexterous hand assembly.

[0102] In some embodiments, the mapping process can be customized according to the specific model of the five-finger dexterous hand to ensure that the joint angles of each finger of the standard hand can be accurately mapped to the corresponding joints of the five-finger dexterous hand. Each degree of freedom of the standard hand (such as the thumb, index finger, middle finger, ring finger, and little finger) will correspond to the joint movements of the five-finger dexterous hand one by one through remapping, thereby achieving fine teleoperation.

[0103] In some embodiments, in the control of the five-finger dexterous hand, each joint angle data of the standard hand can be processed individually and transmitted to the corresponding part of the robotic hand.

[0104] Exemplarily, for the thumb joint, it usually has multiple degrees of freedom, including metacarpal joint flexion and flexion of proximal and distal joints. According to the data of the standard hand, the desired flexion angles of these joints can be determined.

[0105] For other finger joints, the index finger, middle finger, ring finger and little finger generally have degrees of freedom such as metacarpal joint flexion, proximal joint flexion and distal joint flexion. Similarly, the desired flexion angles of these joints can be determined based on the data of the standard hand.

[0106] In some embodiments, the actuator mapping module 202 can also dynamically adjust the mapping parameters according to task requirements and environmental conditions to ensure that the five-fingered dexterous hand can accurately restore the movements of a standard hand when performing tasks, thereby achieving high flexibility and high-precision operations.

[0107] In some embodiments, the end effector may include a two-finger gripper assembly; the effector mapping module 202 is specifically used to:

[0108] According to the angle data of each joint corresponding to the thumb and index finger of a standard hand, the expected joint angle data of each joint corresponding to the two-finger gripper assembly is determined.

[0109] In some embodiments, for the control of the two-finger gripper assembly, the mapping of the standard hand may use only the joint angle data of the thumb and index finger as the expected value source, and the data of these two fingers may be mapped to the opening and closing motions of the two-finger gripper.

[0110] It can be understood that, since the two-finger gripper has only two degrees of freedom, namely, opening and closing, some joint angle data of a standard hand (such as the angles of the thumb and index finger) can be mapped to the desired joint angles of the two-finger gripper.

[0111] In some embodiments, the two-finger gripper can adjust the degree of opening and closing according to the angle data of each joint corresponding to the thumb and index finger of a standard hand, so as to accurately perform tasks such as grasping and placing.

[0112] The above implementation method not only simplifies the complexity of data processing, but also ensures that the two-finger gripper can perform remote operation tasks efficiently and stably.

[0113] In the embodiment of the present application, by collecting the user's gesture information and / or finger end posture information through the end remote operation control module, fine control of the humanoid robot's end effector (including fingers) can be achieved. This control method is more intuitive and precise than the traditional joystick, and can more naturally reflect the user's operating intentions of both hands.

[0114] In some embodiments, the data processing module 203 is specifically used to:

[0115] According to the second information, the desired position and / or posture of the arms of the humanoid robot are determined; and according to the desired position and / or posture of the arms, the second operation instruction is generated.

[0116] In some implementations, the data processing module 203 may be specifically used to:

[0117] Receive the actual position and / or posture data of the end effector fed back by the humanoid robot; determine the expected joint angle data of the two arms based on the actual position and / or posture data of the end effector and the expected position and / or posture of the end effector; generate the above-mentioned second operation instruction based on the expected joint angle data of the two arms.

[0118] In some embodiments, the actual position and / or posture data of the end effector can be compared with the expected position and / or posture to calculate the deviation. Based on the deviation and the mechanical structure of the robot, inverse kinematics calculation is performed to determine the expected joint angle data of the arms. In the humanoid robot neighborhood, inverse kinematics can be used to infer the joint angles in the joint space based on the position and posture of the end effector.

[0119] In an embodiment of the present application, by collecting the spatial position information and / or posture information of the terminal remote control module, the arms of the humanoid robot can be further controlled, so that the humanoid robot can more accurately simulate the user's movements and postures, thereby effectively improving the control accuracy and flexibility of the humanoid robot's remote operation.

[0120] Reference Figure 3 , Figure 3 Schematic diagram of the architecture of another humanoid robot remote control system provided in an embodiment of the present application.

[0121] In some embodiments, the system further includes a plurality of VR positioning trackers 301; the VR positioning trackers 301 can be worn on the user's waist and / or feet.

[0122] In some embodiments, the VR positioning tracker 301 is used to collect third information, where the third information includes spatial position information and / or posture information of the VR positioning tracker.

[0123] In some embodiments, the VR positioning tracker 301 can communicate with the positioning base station through the above-mentioned communication module to obtain the relative position with the positioning base station in real time.

[0124] In some embodiments, the VR positioning tracker 301 can transmit its own spatial position information and / or posture information to the data processing module 203 in real time.

[0125] In some embodiments, the data processing module 203 is further used to:

[0126] According to the third information collected by each VR positioning tracker 301, the moving speed and / or angular velocity of the humanoid robot is determined; according to the moving speed and / or angular velocity, a third operation instruction is generated and sent to the humanoid robot.

[0127] Optionally, the third operation instruction may include movement instructions that the humanoid robot needs to execute, such as forward, backward, left turn, right turn, etc., and corresponding speed and acceleration parameters. By sending the third operation instruction to the humanoid robot, precise control of its movement can be achieved.

[0128] In the embodiment of the present application, by introducing the VR positioning tracker 301, the user's lower limb movements and postures can be tracked in real time, the user's intentions can be understood more accurately, and converted into precise movements of the humanoid robot. This control method enables the humanoid robot to perform flexible operations in complex environments, such as avoiding obstacles, crossing narrow spaces, etc.

[0129] In some embodiments, the above system also includes a positioning head-mounted display device and a base station; the head-mounted display device is used to receive and display image data collected by the humanoid robot; the positioning base station is used to provide the above-mentioned terminal remote operation control module 201 and VR positioning tracker 301 with their respective corresponding spatial position information and / or posture information.

[0130] In some embodiments, the head-mounted display device may be a VR head-mounted display device for receiving image data from a humanoid robot and presenting it in a VR head-mounted display, thereby enabling an operator to see the current environment of the humanoid robot to assist the operator in remote operation.

[0131] In some implementations, the head mounted display device may also feed back its own spatial position information and / or posture information to the data processing module 203 .

[0132] In some embodiments, the data processing module 203 can determine the expected position and orientation of the humanoid robot head by analyzing the spatial position information and / or posture information of the head mounted display device, and generate a fourth operation instruction for the humanoid robot head according to the expected position and orientation. Finally, the generated fourth operation instruction is sent to the humanoid robot to control its head to perform a corresponding action.

[0133] Optionally, the fourth operation instruction may include but is not limited to:

[0134] Steering control instructions: used to control the specific direction or angle of the humanoid robot's head.

[0135] Nodding control command: used to control the humanoid robot's head to nod up and down.

[0136] Head shaking control command: used to control the humanoid robot's head to shake left and right.

[0137] For example, by controlling the rotation of the humanoid robot's head, the humanoid robot's field of vision can be adjusted to help the robot locate and operate the target more accurately; by controlling the humanoid robot to nod or shake its head, the humanoid robot can be helped to better avoid obstacles, etc.

[0138] Optionally, the positioning base station uses any positioning technology (such as optical positioning, electromagnetic positioning, ultrasonic positioning, laser positioning or inertial navigation, etc.) to accurately determine the position of each device (including the terminal remote control module, head-mounted display device and VR positioning tracker) in three-dimensional space. In addition to position information, the positioning base station may also provide attitude information, that is, the orientation and tilt angle of the device in space.

[0139] In some embodiments, the positioning base station and each device perform data transmission via the above-mentioned communication module to ensure the real-time and accuracy of the information.

[0140] In the embodiment of the present application, for the head mounted display device, the accurate location information provided by the positioning base station enables the user to immerse in a more realistic and three-dimensional virtual environment. At the same time, the cooperation between the terminal teleoperation control module 201, the VR positioning tracker 301 and the positioning base station enables the user's movements and postures to be tracked and fed back in real time, thereby improving the overall performance and reliability of the system.

[0141] In some embodiments, the data processing module 203 can execute the corresponding operation mode according to the type and number of VR devices connected to the system. For example, when the connected devices are the terminal teleoperation control module and the head-mounted display device, the system works in the upper limb teleoperation mode; when the connected devices are the terminal teleoperation control module, the head-mounted display device and the VR positioning tracker, the system works in the whole body teleoperation mode.

[0142] For example, refer to Figure 4 , Figure 4 A schematic diagram of a remote control process of a humanoid robot provided in an embodiment of the present application.

[0143] In the upper limb teleoperation mode, the teleoperation control process of the above humanoid robot includes:

[0144] S11. System initialization.

[0145] In some embodiments, system initialization includes:

[0146] Start the positioning base station: Make sure that each VR device is within the coverage of the positioning base station to ensure that subsequent device positioning and tracking can proceed normally.

[0147] Start the VR device: respectively open the left-hand braid VR device, the right-hand braid VR device and the VR head display device. The user observes the relative positions of the left-hand braid VR device and the right-hand braid VR device in the VR head display.

[0148] Device position confirmation: If the positions of the left-hand brass knuckle VR device and the right-hand brass knuckle VR device displayed in the VR headset have overlapped, it means that the relative positions of the above two brass knuckle VR devices are correct.

[0149] If the two brass knuckle VR devices do not overlap in the VR head display, position initialization is performed. Specifically, the two brass knuckle VR devices can be placed at a predetermined position, and the initialization function is started in the data processing module, waiting for the system to complete the initialization.

[0150] After the initialization is completed, the above process is repeated until the positions of the two brass knuckle VR devices displayed in the VR headset overlap.

[0151] S12. Equipment wearing and data transmission.

[0152] In some embodiments, the device wearing and data transmission includes:

[0153] Wearable equipment: Operators wear brass knuckle VR equipment and VR head-mounted display equipment.

[0154] Gesture and position feedback: The knuckle VR device provides real-time feedback of the operator's gestures and / or finger tip postures, as well as its own spatial position and / or posture, and transmits this information to the positioning base station through the communication module. At the same time, the VR head display device can also transmit its own spatial position and / or posture information to the positioning base station through the communication module.

[0155] Humanoid robot camera data transmission: The humanoid robot transmits real-time image data to the data processing module through the camera. The data processing module sends the image data to the VR head display device, so that the operator can obtain the current field of view information of the humanoid robot in real time.

[0156] Receiving joint information of the humanoid robot: The data processing module can also receive the whole body joint information of the humanoid robot and the status data of the end effector.

[0157] S13, data processing and actuator mapping.

[0158] In some embodiments, data processing and actuator mapping includes:

[0159] Processing gestures and / or finger end posture information: The data processing module receives gesture information, finger end posture information and end effector type data from the brass knuckle VR device, and transmits this information to the actuator mapping module.

[0160] Generate standard hand information: The actuator mapping module maps the current operator's hand to the posture information of the standard hand according to the operator's gesture and / or finger end posture, and generates the angle data of each joint of the standard hand.

[0161] Remapping of standard hand and actuator: According to the type of end effector, the actuator mapping module converts the joint angle data of the standard hand into the expected joint angle data of the corresponding end effector. This process ensures the coordination and accuracy between the standard hand and the end effector of the humanoid robot.

[0162] Calculate the joint angles of the humanoid robot: After receiving the actual position and posture of the end effector, the data processing module performs inverse kinematics solution to calculate the expected joint angle data of the humanoid robot's arms.

[0163] S14. Humanoid robot control and remote operation.

[0164] In some embodiments, humanoid robot control and teleoperation includes:

[0165] Transmitting expected joint angle information: The data processing module transmits the expected joint angle information of the end effector and the double arms to the humanoid robot body.

[0166] Tracking control: The humanoid robot body tracks the desired joint angles through a control algorithm (such as PID control algorithm) and adjusts the movements of the humanoid robot to complete the entire teleoperation process.

[0167] Exemplarily, in the whole body teleoperation mode, the above humanoid robot teleoperation control process includes:

[0168] S21. System initialization.

[0169] In some embodiments, system initialization includes:

[0170] Start the positioning base station: Make sure that each VR device is within the coverage of the positioning base station to ensure that subsequent device positioning and tracking can proceed normally.

[0171] Start the VR device: respectively open the left-hand braid VR device, the right-hand braid VR device, the VR head display device and each VR positioning tracker. The user observes the relative positions of the left-hand braid VR device, the right-hand braid VR device and each VR positioning tracker in the VR head display.

[0172] Device position confirmation: Place the two brass knuckle VR devices and each VR positioning tracker in the initialization bracket, and use the VR headset to observe whether the relative positions of the two brass knuckle VR devices and each VR positioning tracker are normal. If normal, skip the next step.

[0173] If it is not normal, the position is initialized. Specifically, the two brass knuckle VR devices and each VR positioning tracker can be placed in the predetermined bracket position, and the initialization function is started in the data processing module, waiting for the system to complete the initialization.

[0174] After initialization is complete, re-observe the relative positions of the two VR knuckle devices and each VR positioning tracker to see if they are normal, and confirm that the moving direction and distance are correct. If they are still abnormal, you need to return and re-initialize.

[0175] S22. Equipment wearing and data transmission.

[0176] In some embodiments, device wearing and data transmission include:

[0177] Wearable equipment: The operator wears a brass knuckle device, a VR head display device and various VR positioning trackers. Optionally, the user can wear a VR positioning tracker on the waist and ankles.

[0178] Gesture and position feedback: The knuckle VR device provides real-time feedback of the operator's gestures and / or finger tip postures, as well as its own spatial position and / or posture, and transmits this information to the positioning base station through the communication module. At the same time, the VR headset and each VR positioning tracker can also transmit their own spatial position and / or posture information to the positioning base station through the communication module.

[0179] Humanoid robot camera data transmission: The humanoid robot transmits real-time image data to the data processing module through the camera. The data processing module sends the image data to the VR head display device, so that the operator can obtain the current field of view information of the humanoid robot in real time.

[0180] Receiving joint information of the humanoid robot: The data processing module can also receive the whole body joint information of the humanoid robot and the status data of the end effector.

[0181] S23, data processing and actuator mapping.

[0182] In some embodiments, data processing and actuator mapping includes:

[0183] Processing gestures and / or finger end posture information: The data processing module receives gesture information, finger end posture information and end effector type data from the brass knuckle VR device, and transmits this information to the actuator mapping module.

[0184] Generate standard hand information: The actuator mapping module maps the current operator's hand to the posture information of the standard hand according to the operator's gesture and / or finger end posture, and generates the angle data of each joint of the standard hand.

[0185] Remapping of standard hand and actuator: According to the type of end effector, the actuator mapping module converts the joint angle data of the standard hand into the expected joint angle data of the corresponding end effector. This process ensures the coordination and accuracy between the standard hand and the end effector of the humanoid robot.

[0186] Calculate the joint angles of the humanoid robot: After receiving the actual position and posture of the end effector, the data processing module performs inverse kinematics solution to calculate the expected joint angle data of the humanoid robot's arms.

[0187] Speed ​​and angular velocity command calculation: The data processing module can analyze the change in the position of each VR positioning tracker and map the change into the motion data of the humanoid robot.

[0188] For example, when the operator walks in a straight line, the above-mentioned change amount is mapped to the moving speed of the humanoid robot; when the operator performs a turning operation, the above-mentioned change amount is mapped to the angular velocity of the humanoid robot.

[0189] S24. Humanoid robot control and remote operation.

[0190] In some embodiments, humanoid robot control and teleoperation includes:

[0191] Transmitting expected joint angle information: The data processing module transmits the expected joint angle information of the end effector and the dual arms to the humanoid robot body, and transmits the determined speed and angular velocity information to the robot body.

[0192] Tracking control: The upper limbs of the humanoid robot body track the desired joint angles through the control algorithm and adjust the movement of the upper limbs of the humanoid robot. The lower limbs complete the walking and turning of the humanoid robot by tracking the speed and angular velocity instructions.

[0193] The humanoid robot teleoperation control system provided in the embodiments of the present application can have the following beneficial effects:

[0194] 1. End effectors that can adapt to a variety of humanoid robots: Compared with ordinary VR handle teleoperation systems, this system uses a brass knuckle VR device that can adapt to a variety of end effectors. For example, in the control of the five-finger dexterous hand, it can accurately adjust the opening and closing movements of each finger, while the traditional VR handle can only control the five-finger dexterous hand to perform a certain degree of opening and closing control. Therefore, this system greatly improves the flexibility and precision of the end effector and enhances the diversity and complexity of humanoid robot operations.

[0195] Second, it can reduce costs: Compared with motion capture gloves, the brass knuckle VR device has a significant price advantage. It effectively reduces the hardware cost of the entire system by providing low-cost, high-precision gesture recognition and capture technology. This cost advantage makes the system more economically competitive, lowers the threshold for using VR devices for remote operation, and helps promote the widespread application of this technology.

[0196] 3. Hand shape optimization and enhanced adaptability: This system supports personalized optimization based on the hand shapes of different operators. By mapping the operator's hand shape to the standard hand shape, and then mapping the standard hand shape to the joint angle of the manipulator, the system's adaptability to various users is enhanced. This optimization process enables operators with different hand shapes to use the system seamlessly, improving the universality and user-friendliness of the teleoperation system.

[0197] 4. Whole-body teleoperation control: This system also expands the control range of teleoperation and combines the tracking of leg movements. By using a VR tracker to capture leg movements, the linear velocity and angular velocity of the legs are converted into control instructions to control the robot to move forward and turn. This integration of leg movements expands the robot's control dimension, realizes teleoperation control of the robot's entire body, and further improves the scalability and flexibility of the system.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the above module division is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0199] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of each solution in this embodiment.

[0200] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0201] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some of the steps provided in various embodiments of the present application.

[0202] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. Some of the solutions disclosed in the application can be directly implemented as hardware processors, or can be implemented by a combination of hardware and software modules in the processor.

[0203] The memory may include high-speed memory, and may also include non-volatile storage, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0204] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0205] The storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The storage medium may be any available medium that can be accessed by a general or special purpose computer.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A humanoid robot teleoperation control system, characterized in that: It includes a terminal teleoperation control module, an actuator mapping module and a data processing module; The terminal remote operation control module is used to collect first information and second information; the first information includes the user's gesture information and / or finger terminal posture information, and the second information includes the spatial position information and / or posture information of the terminal remote operation control module; The actuator mapping module is used to generate a first operation instruction of the end effector corresponding to the humanoid robot according to the first information, and send the first operation instruction to the humanoid robot; The data processing module is used to generate second operation instructions for the two arms corresponding to the humanoid robot according to the second information, and send the second operation instructions to the humanoid robot.

2. The system according to claim 1, characterized in that The actuator mapping module is specifically used for: Converting the first information into standard hand posture information; Generating angle data of each joint corresponding to the standard hand according to the posture information; Converting the angle data into desired joint angle data of the end effector; The first operation instruction is generated according to the expected joint angle data.

3. The system according to claim 2, characterized in that The end effector includes a five-finger dexterous hand assembly; the actuator mapping module is specifically used for: According to the angle data of each joint corresponding to the standard hand, the expected joint angle data of each joint corresponding to the five-finger dexterous hand assembly is determined.

4. The system according to claim 2, characterized in that The end effector comprises a two-finger gripper assembly; the actuator mapping module is specifically used for: According to the angle data of each joint corresponding to the thumb and index finger of the standard hand, the expected joint angle data of each joint corresponding to the two-finger gripper assembly is determined.

5. The system according to claim 1, characterized in that The data processing module is specifically used for: determining a desired position and / or posture of the end effector according to the second information; The second operation instruction is generated according to the expected position and / or posture of the end effector.

6. The system according to claim 5, characterized in that The data processing module is specifically used for: Receiving actual position and / or posture data of the end effector fed back by the humanoid robot; Determining the desired joint angle data of the dual arms according to the actual position and / or posture data of the end effector and the desired position and / or posture of the end effector; The second operation instruction is generated according to the expected joint angle data of the two arms.

7. The system according to any one of claims 1 to 6, characterized in that: The terminal remote operation control module includes a brass knuckle VR device; the brass knuckle VR device includes a left-hand brass knuckle VR device and / or a right-hand brass knuckle VR device.

8. The system according to claim 1, characterized in that The system further comprises a plurality of VR positioning trackers; the VR positioning trackers can be worn on the waist and / or feet of the user; The VR positioning tracker is used to collect third information, and the third information includes spatial position information and / or posture information of the VR positioning tracker.

9. The system according to claim 8, characterized in that The data processing module is also used for: Determine the moving speed and / or angular velocity of the humanoid robot according to the third information collected by each of the VR positioning trackers; A third operation instruction is generated according to the moving speed and / or angular speed, and sent to the humanoid robot.

10. The system according to claim 8 or 9, characterized in that The system also includes a head mounted display device and a positioning base station; The head mounted display device is used to receive and display image data collected by the humanoid robot; The positioning base station is used to provide spatial position information and / or posture information to the terminal remote operation control module, the head-mounted display device, and the VR positioning tracker.

Images