Humanoid robot teleoperation control system

Abstract

The application provides a humanoid robot remote control system, and relates to the technical field of humanoid robots, which comprises: a terminal remote control module, which is used for collecting first information and second information; the first information comprises gesture information and / or finger tip posture information of a user, and the second information comprises spatial position information and / or posture information of the terminal remote control module; an actuator mapping module, which is used for generating first operation instructions of a terminal actuator corresponding to the humanoid robot according to the first information, and sending the first operation instructions to the humanoid robot; and a data processing module, which is used for generating second operation instructions of a double arm corresponding to the humanoid robot according to the second information, and sending the second operation instructions to the humanoid robot. The application can effectively improve the control accuracy and flexibility of the humanoid robot remote control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of humanoid robots, and in particular to a humanoid robot teleoperation control system. BACKGROUND

[0002] Teleoperation technology refers to a technology that enables an operator to remotely control a humanoid robot to perform tasks through a remote control system.

[0003] Traditional teleoperation methods generally convert the operator's actions into control signals for the humanoid robot through controllers (such as joysticks, buttons, etc.). This approach generally has good intuitiveness, but the control precision and flexibility are poor, and when facing complex or dangerous tasks, the operator's operation may have uncertainties and safety hazards.

[0004] In summary, how to improve the control precision and flexibility of humanoid robot teleoperation is a technical problem that needs to be solved at present. SUMMARY

[0005] The embodiments of the present application provide a humanoid robot teleoperation control system, which can effectively improve the control precision and flexibility of humanoid robot teleoperation.

[0006] In some embodiments, the above-mentioned humanoid robot teleoperation control system includes an end teleoperation control module, an actuator mapping module, and a data processing module;

[0007] The end teleoperation control module is configured to collect first information and second information; the first information includes hand gesture information and / or finger tip posture information of a user, and the second information includes spatial position information and / or posture information of the end teleoperation control module;

[0008] The actuator mapping module is configured to generate a first operation instruction of an end effector corresponding to the humanoid robot according to the first information, and send the first operation instruction to the humanoid robot;

[0009] The data processing module is configured to generate a second operation instruction of a double arm corresponding to the humanoid robot according to the second information, and send the second operation instruction to the humanoid robot.

[0010] In some embodiments, the actuator mapping module is specifically configured to:

[0011] convert the first information into standard hand posture information;

[0012] generate angle data of each joint corresponding to the standard hand according to the above-mentioned posture information;

[0013] convert the above-mentioned angle data into expected joint angle data of the end effector;

[0014] A first operation instruction is generated based on the above-mentioned expected joint angle data.

[0015] In some embodiments, the end effector includes a five-finger dexterous hand assembly; and the actuator mapping module is specifically configured to:

[0016] According to the angle data of each joint corresponding to the standard hand, the expected joint angle data of each joint corresponding to the five-finger dexterous hand assembly is determined.

[0017] In some embodiments, the end effector includes a two-finger gripper assembly; and the effector mapping module is specifically configured to:

[0018] According to the angle data of each joint corresponding to the thumb and index finger of a standard hand, the expected joint angle data of each joint corresponding to the two-finger gripper assembly is determined.

[0019] In some embodiments, the data processing module is specifically configured to:

[0020] determining a desired position and / or posture of the end effector based on the second information;

[0021] A second operation instruction is generated according to the desired position and / or posture of the end effector.

[0022] In some embodiments, the data processing module is specifically configured to:

[0023] receiving actual position and / or posture data of the end effector fed back by the humanoid robot;

[0024] Determining desired joint angle data of the two arms based on the actual position and / or posture data of the end effector and the desired position and / or posture of the end effector;

[0025] A second operation instruction is generated based on the desired joint angle data of the two arms.

[0026] In some embodiments, the terminal teleoperation control module includes a brass knuckles virtual reality (VR) device; the brass knuckles VR device includes a left brass knuckles VR device and / or a right brass knuckles VR device.

[0027] In some embodiments, the system further comprises a plurality of VR positioning trackers; the VR positioning trackers can be worn on the user's waist and / or feet;

[0028] The VR positioning tracker is used to collect third information, which includes spatial position information and / or posture information of the VR positioning tracker.

[0029] In some embodiments, the data processing module is further configured to:

[0030] According to the third information collected by the VR positioning tracker, the moving speed and / or angular speed of the humanoid robot are determined;

[0031] According to the moving speed and / or angular speed, the third operation instruction is generated and sent to the humanoid robot.

[0032] In some embodiments, the system further comprises a head-mounted display device and a positioning base station;

[0033] The head-mounted display device is configured to receive and display the image data collected by the humanoid robot;

[0034] The positioning base station is configured to provide spatial position information and / or attitude information to the end teleoperation control module, the head-mounted display device, and the VR positioning tracker.

[0035] The teleoperation control system for the humanoid robot provided by the embodiments of the present application can collect gesture information and / or finger end attitude information of the user through the end teleoperation control module, and can achieve fine control of the end effector (including the fingers) of the humanoid robot. This control method is more intuitive and accurate than the traditional joystick, and can more naturally reflect the operation intention of the user's two hands. Further, by collecting spatial position information and / or attitude information of the end teleoperation control module, the two arms of the humanoid robot can be further accurately controlled, so that the humanoid robot can more accurately simulate the user's actions and attitudes, thereby effectively improving the control accuracy and flexibility of the teleoperation of the humanoid robot. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0037] Figure 1 FIG. 1 is a structural schematic diagram of a humanoid robot provided in an embodiment of the present application;

[0038] Figure 2 FIG. 2 is an architectural schematic diagram of a teleoperation control system for a humanoid robot provided in an embodiment of the present application;

[0039] Figure 3 FIG. 3 is another architectural schematic diagram of a teleoperation control system for a humanoid robot provided in an embodiment of the present application;

[0040] Figure 4 FIG. 4 is a teleoperation control flowchart for a humanoid robot provided in an embodiment of the present application.

[0041] The specific embodiments of the present application have been shown and described in the above drawings, which will be described in more detail below. These drawings and written descriptions are not intended to limit the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] The present application will be described in detail below with reference to the drawings, and the same numbers in different drawings represent the same or similar elements unless otherwise indicated.

[0043] It should be noted that the humanoid robot teleoperation system provided in the embodiments of the present application can be used in the field of humanoid robots, and can also be used in other robot fields other than the field of humanoid robots, which is not limited in the present application.

[0044] A humanoid robot is a robot that imitates the appearance and behavior of a human being. It usually has double mechanical arms, lower limbs, etc. similar to a human being, and can perform various actions similar to a human being.

[0045] Optionally, the key components of the humanoid robot include:

[0046] Double mechanical arms: The mechanical arms of a humanoid robot usually have multiple degrees of freedom, which can simulate various movements of a human arm, such as rotation, bending, and stretching, etc. This enables the robot to perform fine operation tasks, such as grabbing objects, operating tools, etc.

[0047] Lower limbs: The lower limbs of a humanoid robot also have multiple degrees of freedom, which can simulate human actions such as walking, running, jumping, etc.

[0048] Head system: Although the head of a humanoid robot does not always have to imitate the structure of a human head, it is usually equipped with sensors such as cameras and microphones for obtaining environmental information. In addition, some humanoid robots can also be equipped with artificial intelligence technologies such as facial recognition and voice recognition to improve their interaction ability and intelligent level.

[0049] For example, with reference to Figure 1 , Figure 1 is a schematic diagram of the structure of a humanoid robot provided in the embodiments of the present application. In some embodiments, the above-mentioned humanoid robot includes:

[0050] Head system 1: contains the head structure of the humanoid robot, which can be integrated with sensors, cameras, microphones, etc. for environmental perception and interaction.

[0051] Chest system 2: contains the internal electronic devices and power sources of the humanoid robot, such as motor controllers, power distribution units, etc.

[0052] Mechanical arm system 3: Contains a pair of mechanical arms of humanoid robots, used to perform various operational tasks such as grabbing, carrying, etc.

[0053] Dexterous hand system 4: Installed at the end of the mechanical arm, usually designed with multiple joints and degrees of freedom, allowing it to move individual fingers and palm parts in a highly flexible manner. This design enables the dexterous hand to simulate various complex movements of the human hand, such as grabbing, rotating, pinching, etc.

[0054] Thigh 5: The part connecting the hip joint and the knee joint, is the main load-bearing structure of the leg of the humanoid robot.

[0055] Calf 6: The part connecting the knee joint and the ankle joint, is an important part of leg movement.

[0056] Foot 7: The walking and standing component of the humanoid robot, usually designed with anti-skid and shock-absorbing functions.

[0057] Those skilled in the art can understand that, Figure 1 The structure of the humanoid robot shown in the figure does not constitute a limitation on the humanoid robot, and the humanoid robot in this application can include more or fewer components than Figure 1 the figure, or combine certain components, or different component arrangements.

[0058] The application scenarios of the above-mentioned humanoid robot can include but are not limited to the following aspects:

[0059] Industrial manufacturing: In the automated production line, the humanoid robot can replace humans to complete some heavy, dangerous or repetitive work, such as carrying heavy objects, assembling parts, etc.

[0060] Service industry: In service industry places such as restaurants and hotels, the humanoid robot can serve as a waiter or receptionist to provide customers with ordering, delivering, guiding, etc.

[0061] Medical rehabilitation: In the medical field, the humanoid robot can serve as an auxiliary device to help patients with rehabilitation training or provide daily care.

[0062] Rescue and exploration: In disaster sites such as earthquakes and fires, the humanoid robot can replace humans to enter dangerous areas for search and rescue or detection.

[0063] Entertainment and education: In entertainment venues and educational institutions, the humanoid robot can serve as a performance guest or teaching assistant to provide interesting interactive experiences for the audience and students.

[0064] Teleoperation technology refers to a technology that enables an operator to control a humanoid robot to perform tasks remotely through a remote control system. With the continuous expansion of the application field of humanoid robots, teleoperation technology has been widely used in industries, medical treatment, exploration, and other fields.

[0065] The core goal of teleoperation technology is to reduce human contact with dangerous or complex environments, improve work efficiency, operation accuracy, and safety. Existing teleoperation methods can be roughly divided into the following types: traditional teleoperation based on physical controllers, teleoperation based on virtual reality (VR) technology, teleoperation based on brain-computer interface (BCI) technology, etc.

[0066] For example, teleoperation based on physical controllers generally converts the operator's actions into humanoid robot control signals through controllers such as joysticks, buttons, motion capture gloves, and other physical devices. This approach generally has good intuitiveness, but the control accuracy and flexibility are poor, and when facing complex or dangerous tasks, the operator's operation may have uncertainties and safety hazards.

[0067] The introduction of VR technology brings higher immersion and perception capabilities to the teleoperation of humanoid robots. Through a virtual reality system, the operator can not only see the current working environment and state of the humanoid robot, but also can real-time feedback the actions of the humanoid robot.

[0068] The existing VR teleoperation system usually provides incremental feedback by feeding back the position of the VR handle in the VR headset and transmitting this position feedback information to the humanoid robot body, so that the end effector of the humanoid robot can track the position of the VR handle in real time. Although this incremental control method can control the operation of the humanoid robot to some extent, it still has the following technical problems:

[0069] Human posture matching problem: The incremental control method has high requirements for the operator's body posture, as the end effector of the humanoid robot and the operator's body posture are not always completely consistent, and in some cases, the operator's body posture may cause inaccurate control or dislocation phenomenon, affecting the accuracy and stability of the operation.

[0070] Stroke control problem: The VR handle used in the current VR teleoperation system mainly controls the opening and closing state of the effector through buttons, and cannot achieve precise stroke control of the effector. For example, the current VR handle only supports opening and closing control, lacks precise motion control of mechanical arms or gripping devices, resulting in limited operation accuracy.

[0071] Claw control problem: for three or more claws, the existing VR teleoperation system can only control the opening and closing state of the claws in a specific trajectory, and cannot realize independent control of each mechanical finger. This control method cannot cope with complex grasping tasks, reducing the flexibility of operation.

[0072] In the face of the above technical problems, the embodiment of the application provides a humanoid robot teleoperation system. The gesture information and / or finger end posture information of the user can be collected by the end teleoperation control module, which can realize fine control of the end effector (including the fingers) of the humanoid robot. The spatial position information and / or posture information of the end teleoperation control module can be collected to further control the arms of the humanoid robot, so that the humanoid robot can more accurately simulate the actions and postures of the user, thereby effectively improving the control accuracy and flexibility of the humanoid robot teleoperation.

[0073] The technical solutions of the application and how the technical solutions of the application solve the above technical problems will be described in detail below with specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0074] Reference Figure 2 , Figure 2 The figure is a schematic diagram of the architecture of a humanoid robot teleoperation control system provided in the embodiment of the application. In some embodiments, the humanoid robot teleoperation control system includes an end teleoperation control module 201, an effector mapping module 202, and a data processing module 203.

[0075] In some embodiments, the end teleoperation control module 201 can be worn on the user's (hereinafter referred to as the operator) hands.

[0076] In some embodiments, the end teleoperation control module 201 can be used to collect first information and second information.

[0077] Optionally, the first information includes gesture information and / or finger end posture information of the user; and the second information includes spatial position information and / or posture information of the end teleoperation control module 201.

[0078] The first information can be used to identify the finger operation intention of the user, i.e., the gesture and finger end posture of the user can reflect the type of operation that the user's fingers want to perform.

[0079] The second information can be used to identify the position and posture of the user's hands, such as identifying the exact position and orientation of the user's hands in three-dimensional space, thereby ensuring the accuracy and safety of the operation of the humanoid robot.

[0080] In some embodiments, the end teleoperation control module 201 can include a finger VR device (also referred to as a finger VR handle); the finger VR device can include a left-hand finger VR device and / or a right-hand finger VR device. The left-hand finger VR device is worn on the left hand of the user, and the right-hand finger VR device is worn on the right hand of the user.

[0081] Optionally, the finger VR device described above can adopt a sleeve hand design, which can fit the hand more closely compared with a conventional VR handle, and can provide a more stable and comfortable holding experience.

[0082] Optionally, the finger VR device described above can be built-in with multiple sensors for tracking the position and movement of each finger, so as to more accurately identify various movements of the user's fingers, such as grabbing and releasing.

[0083] In some embodiments, the finger VR device described above can feed the collected first information to the data processing module 203 in real time, and the data processing module 203 can send the first information to the actuator mapping module 202, or the finger VR device can directly feed the collected first information to the actuator mapping module 202.

[0084] In some embodiments, the finger VR device described above can feed the collected second information to the data processing module 203 in real time.

[0085] Optionally, the humanoid robot teleoperation control system described above further includes a communication module. The end teleoperation control module 201 can use the communication module to feed the first information and the second information to the data processing module 203; or use the communication module to directly feed the first information to the actuator mapping module 202.

[0086] Optionally, the communication module described above can be a wired communication module, or a wireless communication module (such as a Wi-Fi module, a Bluetooth module, etc.), or a special communication protocol module, which is not limited in the embodiments of the present application.

[0087] The actuator mapping module 202 can be configured to generate a first operation instruction of an end effector of a humanoid robot corresponding to the first information, and send the first operation instruction to the humanoid robot.

[0088] In some embodiments, the actuator mapping module 202 can analyze the first information to identify the specific operation type that the user wants to perform, such as grabbing and releasing. According to the analysis result, the actuator mapping module 202 generates a first operation instruction corresponding to the end effector of the humanoid robot. Finally, the actuator mapping module 202 sends the generated first operation instruction to the humanoid robot through the communication module, so as to control the end effector of the humanoid robot to complete the corresponding action.

[0089] The data processing module 203 is configured to generate the second operation instruction of the double arms of the humanoid robot according to the second information, and send the second operation instruction to the humanoid robot.

[0090] In some embodiments, the data processing module 203 can parse the second information to identify the expected position and orientation of the arms of the humanoid robot in the three-dimensional space. Further, based on the parsing result, the data processing module 203 can determine the expected joint angle of the double arms of the humanoid robot, and generate the second operation instruction of the double arms of the humanoid robot according to the expected joint angle. Finally, the generated second operation instruction is sent to the humanoid robot to control the double arms of the humanoid robot to perform corresponding actions.

[0091] The remote control system for the humanoid robot provided by the embodiments of the present application can collect gesture information and / or finger tip posture information of the user through the end remote control module, and can realize fine control of the end effector (including the fingers) of the humanoid robot. This control method is more intuitive and accurate than the traditional joystick, and can more naturally reflect the operation intention of the user's two hands. Further, by collecting the spatial position information and / or posture information of the end remote control module, the double arms of the humanoid robot can be further controlled, so that the humanoid robot can more accurately simulate the action and posture of the user, thereby effectively improving the control accuracy and flexibility of the remote control of the humanoid robot.

[0092] Based on the content described in the above embodiments, in some embodiments, the effector mapping module 202 is specifically configured to:

[0093] convert the first information into the posture information of the standard hand; generate angle data of each joint of the standard hand according to the posture information; convert the angle data into expected joint angle data of the end effector; and generate the first operation instruction according to the expected joint angle data.

[0094] In some embodiments, the effector mapping module 202 can receive the first information, and convert the first information into the posture of the standard hand, i.e., uniformly map the hand shape of the operator to the same standard hand shape.

[0095] In order to uniformly map the hand shapes of different operators to the same standard hand shape, the effector mapping module 202 can use a mapping technology, which can ensure that the hand effector of the humanoid robot can respond in a consistent and accurate manner regardless of the actual hand shape of the operator.

[0096] The standard hand morphology can be predefined and represents a desired pose of the anthropomorphic robotic hand effector. Through the mapping, the effector mapping module 202 can convert the user's hand gestures and finger tip information into control instructions corresponding to the standard hand morphology.

[0097] In some embodiments, to ensure accuracy, the effector mapping module 202 can employ a weighted average of the finger tip inverse solution and the gesture data to calculate the joint angles of each finger. The finger tip inverse solution refers to the angles of the joints of a finger that are inversely solved from the position of the finger tip; and the weighted average is to smooth the gesture data, reduce the influence of noise and errors.

[0098] In some embodiments, the end effector can include a five-fingered dexterous hand assembly; and the effector mapping module 202 is specifically configured to:

[0099] According to the angle data of each joint corresponding to the standard hand, determine the expected joint angle data of each joint corresponding to the five-fingered dexterous hand assembly.

[0100] Each finger of the five-fingered dexterous hand can move independently to simulate various actions of human fingers, such as grasping, pinching, rotating, etc.

[0101] In some embodiments, the effector mapping module 202 can map the joint angle data of the standard hand to each joint of the five-fingered dexterous hand assembly.

[0102] In some embodiments, the mapping process can be customized according to the specific model of the five-fingered dexterous hand to ensure that the joint angle of each finger of the standard hand can be accurately mapped to the corresponding joint of the five-fingered dexterous hand. Each degree of freedom of the standard hand (such as the thumb, index finger, middle finger, ring finger, and little finger) will be mapped through remapping to correspond to the joint action of the five-fingered dexterous hand, thereby achieving fine teleoperation.

[0103] In some embodiments, in the control of the five-fingered dexterous hand, each joint angle data of the standard hand can be processed individually and transmitted to the corresponding part of the robot hand.

[0104] For example, for the thumb joint, it usually has multiple degrees of freedom, including metacarpal joint flexion and flexion of the proximal and distal joints. According to the data of the standard hand, the expected flexion angles of these joints can be determined.

[0105] For other finger joints, the index finger, middle finger, ring finger, and little finger usually have degrees of freedom such as metacarpal joint flexion, proximal joint flexion, and distal joint flexion. Similarly, the expected flexion angles of these joints can be determined according to the data of the standard hand.

[0106] In some embodiments, the manipulator mapping module 202 can also dynamically adjust the mapping parameters according to the task requirements and environmental conditions, so as to ensure that the dexterous hand can accurately restore the actions of the standard hand when performing the task, and achieve high flexibility and high precision operation.

[0107] In some embodiments, the above end effector can include a two-fingered gripper assembly, and the manipulator mapping module 202 is specifically configured to:

[0108] According to the angle data of the respective joints corresponding to the thumb and index finger of the standard hand, the expected joint angle data of the respective joints corresponding to the two-fingered gripper assembly is determined.

[0109] In some embodiments, for the control of the two-fingered gripper assembly, the mapping of the standard hand can only use the joint angle data of the thumb and index finger as the expected value source, and the data of the two fingers can be mapped into the opening and closing actions of the two-fingered gripper.

[0110] It can be understood that, since the two-fingered gripper has only two degrees of freedom of opening and closing, the joint angle data of part of the standard hand (such as the angles of the thumb and index finger) can be mapped as the expected joint angle of the two-fingered gripper.

[0111] In some embodiments, the two-fingered gripper can adjust the opening and closing degree according to the angle data of the respective joints corresponding to the thumb and index finger of the standard hand, so as to accurately perform tasks such as grabbing and placing.

[0112] The above embodiments not only simplify the complexity of data processing, but also ensure that the two-fingered gripper can efficiently and stably perform remote operation tasks.

[0113] In the embodiments of the present application, the gesture information and / or finger end posture information of the user can be collected by the end remote operation control module, so as to realize fine control of the end effector (including the fingers) of the humanoid robot. This control mode is more intuitive and accurate than the traditional joystick, and can more naturally reflect the operation intention of the user's two hands.

[0114] In some embodiments, the data processing module 203 is specifically configured to:

[0115] According to the above second information, the expected position and / or posture of the two arms of the humanoid robot is determined; and the above second operation instruction is generated according to the expected position and / or posture of the two arms.

[0116] In some embodiments, the data processing module 203 can be specifically configured to:

[0117] receive actual position and / or attitude data of the end effector fed back by the humanoid robot; determine desired joint angle data of the dual arms according to the actual position and / or attitude data of the end effector and a desired position and / or attitude of the end effector; and generate the second operation instruction according to the desired joint angle data of the dual arms.

[0118] In some embodiments, the actual position and / or attitude data of the end effector can be compared with the desired position and / or attitude to calculate a deviation. Based on the deviation and the mechanical structure of the robot, inverse kinematics calculation is performed to determine the desired joint angle data of the dual arms. In the humanoid robot neighborhood, inverse kinematics can be used to back-propagate the joint angle in the joint space according to the position and attitude of the end effector.

[0119] In the embodiments of the present application, by collecting the spatial position information and / or attitude information of the teleoperation control module of the end effector, the dual arms of the humanoid robot can be further controlled, so that the humanoid robot can more accurately simulate the actions and postures of the user, thereby effectively improving the control accuracy and flexibility of the teleoperation of the humanoid robot.

[0120] Reference Figure 3 , Figure 3 FIG. 2 is a schematic diagram of another architecture of a teleoperation control system of a humanoid robot provided in the embodiments of the present application.

[0121] In some embodiments, the system further includes a plurality of VR positioning trackers 301, which can be worn on the waist and / or feet of the user.

[0122] In some embodiments, the VR positioning tracker 301 is configured to collect third information including spatial position information and / or attitude information of the VR positioning tracker.

[0123] In some embodiments, the VR positioning tracker 301 can communicate with the positioning base station through the communication module to obtain the relative position with the positioning base station in real time.

[0124] In some embodiments, the VR positioning tracker 301 can transmit the spatial position information and / or attitude information of itself to the data processing module 203 in real time.

[0125] In some embodiments, the data processing module 203 is further configured to:

[0126] determine a moving speed and / or angular speed of the humanoid robot according to the third information collected by each VR positioning tracker 301; generate a third operation instruction according to the moving speed and / or angular speed and send the third operation instruction to the humanoid robot.

[0127] Optionally, the third operation instruction can include movement instructions required to be executed by the humanoid robot, such as forward movement, backward movement, left turn, right turn, etc., as well as corresponding speed and acceleration parameters. By sending the third operation instruction to the humanoid robot, precise control of its movement can be achieved.

[0128] In the embodiments of the present application, by introducing the VR positioning tracker 301, the user's lower limb actions and postures can be tracked in real time, the user's intentions can be more accurately understood, and the user's intentions can be converted into precise movements of the humanoid robot. This control method enables the humanoid robot to perform flexible operations in complex environments, such as avoiding obstacles and traversing narrow spaces.

[0129] In some embodiments, the system further includes a positioning head-mounted display device and a base station; the head-mounted display device is configured to receive and display image data collected by the humanoid robot; and the positioning base station is configured to provide respective spatial position information and / or posture information to the end teleoperation control module 201 and the VR positioning tracker 301.

[0130] In some embodiments, the head-mounted display device can be a VR head-mounted display device, which is configured to receive image data from the humanoid robot and present the image data in the VR head-mounted display. In this way, the operator can see the environment in which the humanoid robot is currently located, thereby assisting the operator in teleoperation.

[0131] In some embodiments, the head-mounted display device can further feed back its spatial position information and / or posture information to the data processing module 203.

[0132] In some embodiments, the data processing module 203 can determine the desired position and orientation of the head of the humanoid robot by analyzing the spatial position information and / or posture information of the head-mounted display device, and generate a fourth operation instruction for the head of the humanoid robot according to the desired position and orientation. Finally, the generated fourth operation instruction is sent to the humanoid robot to control the head of the humanoid robot to perform corresponding actions.

[0133] Optionally, the fourth operation instruction can include but is not limited to:

[0134] Turning control instruction: used to control the specific direction or angle of turning of the head of the humanoid robot.

[0135] Nodding control instruction: used to control the up-and-down nodding action of the head of the humanoid robot.

[0136] Shaking control instruction: used to control the left-and-right shaking action of the head of the humanoid robot.

[0137] For example, by controlling the turning of the head of the humanoid robot, the field of view of the humanoid robot can be adjusted to help the robot locate and operate the target more accurately; by controlling the nodding or shaking of the head of the humanoid robot, the humanoid robot can be helped to better avoid obstacles, etc.

[0138] Optionally, the positioning base station uses any positioning technology (such as optical positioning, electromagnetic positioning, ultrasonic positioning, laser positioning, or inertial navigation, etc.) to accurately determine the positions of each device (including the end teleoperation control module, the head-mounted display device, and the VR positioning tracker) in three-dimensional space. In addition to position information, the positioning base station can also provide attitude information, i.e., the orientation and tilt angle of the device in space.

[0139] In some embodiments, the positioning base station and each device transmit data through the above-mentioned communication module, ensuring the real-time and accuracy of the information.

[0140] In the embodiments of the present application, for the head-mounted display device, the accurate position information provided by the positioning base station enables the user to immerse in a more realistic and three-dimensional virtual environment. At the same time, the cooperation of the end teleoperation control module 201, the VR positioning tracker 301, and the positioning base station enables the user's actions and postures to be tracked and fed back in real time, thereby improving the overall performance and reliability of the system.

[0141] In some embodiments, the data processing module 203 can execute corresponding operation modes according to the types and quantities of the VR devices accessed by the system. For example, when the accessed devices are the end teleoperation control module and the head-mounted display device, the system works in the upper limb teleoperation mode; when the accessed devices are the end teleoperation control module, the head-mounted display device, and the VR positioning tracker, the system works in the full-body teleoperation mode.

[0142] For example, with reference to Figure 4 , Figure 4 is a schematic diagram of a humanoid robot teleoperation control process provided in the embodiments of the present application.

[0143] In the upper limb teleoperation mode, the above-mentioned humanoid robot teleoperation control process includes:

[0144] S11, system initialization.

[0145] In some embodiments, the system initialization includes:

[0146] Starting the positioning base station: ensuring that each VR device is within the coverage range of the positioning base station, so as to ensure that the subsequent device positioning and tracking can be normally performed.

[0147] Start VR devices: Turn on the left finger tiger VR device, the right finger tiger VR device, and the VR headset device respectively. The user observes the relative positions of the left finger tiger VR device and the right finger tiger VR device in the VR headset.

[0148] Device position confirmation: If the positions of the left finger tiger VR device and the right finger tiger VR device displayed in the VR headset have coincided, it indicates that the relative positions of the two finger tiger VR devices are correct.

[0149] If the two finger tiger VR devices do not coincide in the VR headset, position initialization is performed. Specifically, the two finger tiger VR devices can be placed at a predetermined position, and the initialization function in the data processing module is started, and the system is waited to complete the initialization.

[0150] After the initialization is completed, the above process is repeated until the positions of the two finger tiger VR devices displayed in the VR headset coincide.

[0151] S12, device wearing and data transmission.

[0152] In some embodiments, device wearing and data transmission includes:

[0153] Wear devices: The operator wears the finger tiger VR device and the VR headset device.

[0154] Gesture and position feedback: The finger tiger VR device feeds back the operator's gestures and / or finger tip postures, as well as its own spatial position and / or posture in real time, and transmits these information to the positioning base station through the communication module. At the same time, the VR headset device can also transmit its own spatial position and / or posture information to the positioning base station through the communication module.

[0155] Humanoid robot camera data transmission: The humanoid robot transmits real-time image data to the data processing module through the camera. The data processing module sends the image data to the VR headset device, so that the operator can obtain the current visual information of the humanoid robot in real time.

[0156] Receive joint information of humanoid robot: The data processing module can also receive the whole body joint information of the humanoid robot and the state data of the end effector.

[0157] S13, data processing and effector mapping.

[0158] In some embodiments, data processing and effector mapping includes:

[0159] Process gesture and / or finger tip posture information: The data processing module receives gesture information, finger tip posture information, and type data of the end effector from the finger tiger VR device, and transmits these information to the effector mapping module.

[0160] Standard hand information generation: The operator mapping module maps the current operator's hand to the pose information of a standard hand according to the operator's gestures and / or finger tip poses, and generates angle data of each joint of the standard hand.

[0161] Standard hand and actuator remapping: According to the type of the end effector, the operator mapping module converts the joint angle data of the standard hand into expected joint angle data of the corresponding end effector. This process ensures the coordination and accuracy between the standard hand and the end effector of the humanoid robot.

[0162] Calculation of joint angles of the humanoid robot: After receiving the actual position and pose of the end effector, the data processing module performs inverse kinematics calculation to calculate the expected joint angle data of the humanoid robot's arms.

[0163] S14, humanoid robot control and teleoperation.

[0164] In some embodiments, the humanoid robot control and teleoperation includes:

[0165] Transmission of expected joint angle information: The data processing module transmits the expected joint angle information of the end effector and the arms to the humanoid robot body.

[0166] Tracking control: The humanoid robot body tracks the expected joint angles through a control algorithm (such as a PID control algorithm) and adjusts the motion of the humanoid robot, thereby completing the entire teleoperation process.

[0167] Exemplarily, in the full-body teleoperation mode, the above-mentioned humanoid robot teleoperation control process includes:

[0168] S21, system initialization.

[0169] In some embodiments, the system initialization includes:

[0170] Starting the positioning base station: Ensure that each VR device is within the coverage range of the positioning base station to ensure that subsequent device positioning and tracking can be performed normally.

[0171] Starting the VR device: Turn on the left finger tiger VR device, the right finger tiger VR device, the VR head-mounted device, and each VR positioning tracker. The user observes the relative positions of the left finger tiger VR device, the right finger tiger VR device, and each VR positioning tracker in the VR head-mounted device.

[0172] Device position confirmation: Place the two finger tiger VR devices and each VR positioning tracker in the initialization support, and observe whether the relative positions of the two finger tiger VR devices and each VR positioning tracker are normal through the VR head-mounted device. If normal, skip the next step.

[0173] If not, position initialization is performed. Specifically, the two finger VR devices and the VR positioning trackers can be placed in predetermined support positions, and an initialization function is started in the data processing module, waiting for the system to complete initialization.

[0174] After initialization is completed, the relative positions of the two finger VR devices and the VR positioning trackers are re-observed to determine whether the movement direction and distance are correct. If not, re-initialization is required.

[0175] S22, device wearing and data transmission.

[0176] In some embodiments, device wearing and data transmission includes:

[0177] Wearing devices: the operator wears the finger device, the VR head-mounted device, and the VR positioning tracker. Optionally, the user can wear a VR positioning tracker on the waist and ankles of both feet.

[0178] Gesture and position feedback: the finger VR device feeds back the operator's gestures and / or finger tip posture, as well as its own spatial position and / or posture in real time, and transmits this information to the positioning base station through the communication module. At the same time, the VR head-mounted device and the VR positioning tracker can also transmit their own spatial position and / or posture information to the positioning base station through the communication module.

[0179] Humanoid robot camera data transmission: the humanoid robot transmits real-time image data to the data processing module through the camera. The data processing module sends the image data to the VR head-mounted device, so that the operator can obtain the current field of view information of the humanoid robot in real time.

[0180] Receiving joint information of the humanoid robot: the data processing module can also receive the whole body joint information of the humanoid robot and the state data of the end effector.

[0181] S23, data processing and effector mapping.

[0182] In some embodiments, data processing and effector mapping includes:

[0183] Processing gesture and / or finger tip posture information: the data processing module receives gesture information, finger tip posture information, and type data of the end effector from the finger VR device, and transmits this information to the effector mapping module.

[0184] Generating standard hand information: the effector mapping module maps the current operator's hand to the posture information of the standard hand according to the operator's gestures and / or finger tip posture, and generates angle data of each joint of the standard hand.

[0185] Standard hand and executor remapping: According to the type of the end executor, the executor mapping module converts the joint angle data of the standard hand into the expected joint angle data of the corresponding end executor. This process ensures the coordination and accuracy between the standard hand and the end executor of the humanoid robot.

[0186] Calculate the joint angle of the humanoid robot: After receiving the actual position and posture of the end executor, the data processing module performs inverse kinematics calculation to calculate the expected joint angle data of the humanoid robot's arms.

[0187] Velocity and angular velocity instruction calculation: The data processing module can analyze the change in the position of each VR positioning tracker and map this change to the motion data of the humanoid robot.

[0188] For example, when the operator walks straight, the above change is mapped to the moving speed of the humanoid robot; when the operator performs a steering operation, the above change is mapped to the angular velocity of the humanoid robot.

[0189] S24, humanoid robot control and remote operation.

[0190] In some embodiments, the humanoid robot control and remote operation includes:

[0191] Transmit the expected joint angle information: The data processing module transmits the expected joint angle information of the end executor and the arms to the humanoid robot body. And the determined speed and angular velocity information is transmitted to the robot body.

[0192] Tracking control: The upper limbs of the humanoid robot body track the expected joint angle through the control algorithm, and adjust the motion of the upper limbs of the humanoid robot. The lower limbs track the speed and angular velocity instructions to complete the walking and steering of the humanoid robot.

[0193] The humanoid robot remote operation control system provided in the embodiments of the present application can have the following beneficial effects:

[0194] I. It can adapt to various end executors of humanoid robots: Compared with the ordinary VR handle remote operation system, the system uses a tiger VR device, which can adapt to various types of end executors. For example, in the control of a five-fingered dexterous hand, it can accurately adjust the opening and closing action of each finger, while the traditional VR handle can only control the five-fingered dexterous hand to perform a specific degree of opening and closing control. Therefore, the system greatly improves the flexibility and precision of the end executor, and enhances the diversity and complexity of the humanoid robot operation.

[0195] II. Cost Reduction: Compared to motion capture gloves, the VR device has a significant price advantage. It effectively reduces the hardware cost of the entire system by providing low-cost, high-precision gesture recognition and capture technology. This cost advantage makes the system more competitive in terms of economy, lowers the threshold for using VR devices for teleoperation, and helps to promote the widespread application of this technology.

[0196] III. Hand Shape Optimization and Adaptability Enhancement: The system supports personalized optimization based on the hand shape of different operators. By mapping the operator's hand shape to a standard hand shape, and then mapping the standard hand shape to the joint angles of the robot hand, the system's adaptability to various users is enhanced. This optimization process enables operators with different hand shapes to seamlessly use the system, improving the universality and user-friendliness of the teleoperation system.

[0197] IV. Full-Body Teleoperation Control: The system also expands the control range of teleoperation by incorporating leg movement tracking. By using VR trackers to capture leg movements, the linear and angular velocities of the legs are converted into control commands to control the robot's forward and turning movements. This integration of leg movements expands the dimensions of robot control, enabling full-body teleoperation control of the robot, further enhancing the system's scalability and flexibility.

[0198] In the several embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the division of the above-mentioned modules is only a logical functional division, and actual implementation can have another division method, such as combining or integrating multiple modules into another system, or ignoring or not executing some features.

[0199] The modules described above as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical units, i.e., they can be located in one place or distributed to multiple network units. Depending on actual needs, some or all of the modules can be selected to achieve the purpose of each scheme in the embodiments.

[0200] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each module can exist physically alone, or two or more modules can be integrated in one unit. The above-mentioned modules can be realized in the form of hardware or in the form of hardware plus software functional units.

[0201] The integrated modules realized in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium and include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps provided in the embodiments of the present application.

[0202] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The disclosed part of the application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution.

[0203] The memory can include a high-speed memory, and can also include a non-volatile storage, for example, at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0204] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0205] The storage medium described above can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0206] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A humanoid robot teleoperation control system, characterized in that: It includes terminal teleoperation control module, actuator mapping module and data processing module; The terminal remote control module is used to collect first information and second information; the first information includes user gesture information and / or finger end posture information, and the second information includes spatial position information and / or posture information of the terminal remote control module; The actuator mapping module is configured to convert the user's hand posture information into standard hand posture information based on the first information; perform finger end inverse solution and weighted average of gesture data on the standard hand posture information to generate angle data of each joint corresponding to the standard hand; and convert the angle data into expected joint angle data of the corresponding end effector based on the type of the end effector; generating a first operation instruction for an end effector corresponding to the humanoid robot according to the expected joint angle data, and sending the first operation instruction to the humanoid robot; The data processing module is used to determine the expected position and / or posture of the end effector based on the second information; receive the actual position and / or posture data of the end effector fed back by the humanoid robot; determine the deviation between the actual position and / or posture data and the expected position and / or posture based on the actual position and / or posture data of the end effector and the expected position and / or posture of the end effector; perform inverse kinematics solution based on the deviation and the mechanical structure of the humanoid robot to determine the expected joint angle data of the two arms corresponding to the humanoid robot; generate second operation instructions for the two arms based on the expected joint angle data of the two arms, and send them to the humanoid robot.

2. The system according to claim 1, wherein: The end effector includes a five-finger dexterous hand assembly; the actuator mapping module is specifically used to: According to the angle data of each joint corresponding to the standard hand, the expected joint angle data of each joint corresponding to the five-finger dexterous hand assembly is determined.

3. The system according to claim 1, wherein: The end effector includes a two-finger gripper assembly; the actuator mapping module is specifically used to: According to the angle data of each joint corresponding to the thumb and index finger of the standard hand, the expected joint angle data of each joint corresponding to the two-finger gripper assembly is determined.

4. The system according to any one of claims 1 to 3, characterized in that The terminal teleoperation control module includes a brass knuckles VR device; the brass knuckles VR device includes a left-hand brass knuckles VR device and / or a right-hand brass knuckles VR device.

5. The system according to claim 1, wherein: The system further includes a plurality of VR positioning trackers; the VR positioning trackers can be worn on the waist and / or feet of the user; The VR positioning tracker is used to collect third information, and the third information includes spatial position information and / or posture information of the VR positioning tracker.

6. The system according to claim 5, characterized in that The data processing module is further configured to: determining a movement speed and / or angular velocity of the humanoid robot based on the third information collected by each of the VR positioning trackers; A third operation instruction is generated according to the movement speed and / or angular speed, and is sent to the humanoid robot.

7. The system according to claim 5 or 6, characterized in that The system also includes a head-mounted display device and a positioning base station; The head mounted display device is used to receive and display image data collected by the humanoid robot; The positioning base station is used to provide spatial position information and / or posture information to the terminal remote operation control module, the head-mounted display device, and the VR positioning tracker.

Images