VR device-based semi-human robot teleoperation control method

Abstract

The application discloses a semi-human robot remote control method based on a VR device, and relates to the field of VR devices, and comprises the following steps: step one, data flow tracking and connection: a data flow tracking APP is developed and compiled, the APP is installed on the used VR device, and the VR device collects relative pose data of gestures and heads of operators in real time; step two, coordinate conversion of VR data flow: the coordinate systems of wrists and heads in the VR data flow are one-to-one corresponding to VR base coordinates; and step three, an inverse kinematics solving program is written based on C++, and joint angles and gestures are solved and designed.The application breaks through the perspective limitation of the existing system, combines the head control function of the VR device, so that the movement of the robot head can be adjusted in real time along with the rotation of the head of the operator, thereby realizing dynamic first perspective switching.This not only enhances the immersion of the operator, but also improves the operation accuracy, so that the operator can better operate in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of VR equipment, and in particular to a remote operation control method of a semi-humanoid robot based on VR equipment. Background Art

[0002] In recent years, with the advancement of robotics, the application of semi-humanoid robots has become increasingly widespread in industrial manufacturing, service industries, and specialized environments. To enhance robots' ability to perform complex tasks, teleoperation technology has become a research hotspot, with virtual reality (VR)-based teleoperation methods attracting particular attention. VR technology significantly enhances the operator's perception and control precision through immersive user interfaces and intuitive interaction. Currently, many teleoperation systems focus on utilizing VR devices to precisely manipulate the robot's arms and grippers, achieving some success. However, to adapt to the diverse demands of complex scenarios, relying solely on the control of the arms and grippers is insufficient to fully meet the flexibility and versatility required in practical applications.

[0003] The existing remote operation systems based on VR devices have the following main shortcomings:

[0004] 1. Limited viewing angle: Existing systems usually adopt a third-person perspective or a fixed first-person perspective. This approach cannot flexibly adjust the movement direction of the robot head, resulting in the operator being unable to obtain dynamic perspective switching, reducing immersion and operational accuracy.

[0005] 2. Insufficient freedom of movement: Most systems only support control of the arms and gripper, lacking the ability to control the robot's waist and chassis movements. Waist movement can significantly increase the operating space, while chassis control can expand the range of motion and flexibility, which are particularly important in complex manipulation tasks.

[0006] 3. Delay and jamming issues: During remote operation, existing systems often experience delays and jamming due to complex control algorithms or slow solution speeds, affecting the real-time and smoothness of task execution.

[0007] Therefore, it is necessary to propose a semi-human robot teleoperation control method based on VR equipment to solve the above problems. Summary of the Invention

[0008] The purpose of the present invention is to provide a remote control method for a semi-human robot based on VR equipment to solve the problems raised in the above background technology.

[0009] To achieve the above-mentioned object, the present invention provides the following technical solution: a method for remote control of a semi-humanoid robot based on a VR device, comprising the following steps: Step 1, data stream tracking and connection: developing and compiling a data stream tracking APP, installing the APP on the VR device in use, and the VR device collecting the operator's gestures and relative head posture data in real time;

[0010] Step 2: Coordinate conversion of VR data stream: The coordinate system of the human wrist and head in the VR data stream corresponds one-to-one with the VR base coordinate system;

[0011] Step 3: Write an inverse kinematics solver based on C++ to solve the joint angles and gestures. The specific steps include:

[0012] A1. Design of arm and waist: Perform inverse kinematics on the arm and waist as a whole, using the first joint at the bottom of the waist as the base coordinate system.

[0013] A2. Finger solution design: A control system based on finger bending angle and gestures to achieve intelligent control of the robot chassis and arms;

[0014] A3. Head solution design: The head has two degrees of freedom;

[0015] A4. Mode switching: A "snapping finger" gesture is designed based on chassis control and arm control to switch control modes;

[0016] Step 4: Send control instructions: Based on the joint angles, data is transmitted through ZeroMQ (ZMQ); the server transmits the data to the motors of each joint, and the motors move to complete the predetermined action.

[0017] Preferably, the VR device and the robot host are connected to the same local area network.

[0018] Preferably, the robot host system is installed with an avp_stream software package, and the robot host accesses the gesture and head posture data in the VR device through code.

[0019] Preferably, the position of the robot wrist relative to the robot base coordinates is recorded as ,

[0020] The pose calculation based on the robot wrist relative to the robot base coordinates can be obtained:

[0021] ;

[0022] The transformation matrix of the VR base coordinate system relative to the robot base coordinate system is recorded as The position of the human wrist in the VR base coordinate system is recorded as The transformation matrix of the robot wrist coordinate system relative to the human wrist coordinate system is recorded as .

[0023] Preferably, the length ratio of the human arm and the robot arm is set as a scale factor, denoted as S, and scaled by the posture to simulate the actual proportional relationship:

[0024] .

[0025] Preferably, the inverse kinematics solution calculates the motion angle of each joint based on the desired position and posture of the end effector.

[0026] Preferably, the finger solution specifically includes:

[0027] Open your hands and face forward: This gesture indicates that the chassis is moving forward;

[0028] Open your hands, facing backwards: This gesture means the chassis is moving backwards;

[0029] Make a fist with the thumb pointing to the left: This gesture indicates that the chassis turns left;

[0030] Make a fist with your thumb pointing to the right: This gesture indicates that the chassis is turning right.

[0031] Preferably, the "snapping fingers" gesture is achieved by identifying the bending angle of the fingers and a specific action.

[0032] Preferably, the head degrees of freedom do not need to be solved.

[0033] The technical effects and advantages of the present invention are as follows:

[0034] 1. This invention overcomes the viewing angle limitations of existing systems. By integrating the head control function of VR devices, the robot's head movement can be adjusted in real time as the operator's head rotates, thus achieving dynamic first-person perspective switching. This not only enhances the operator's sense of immersion but also improves operational precision, enabling better performance in complex environments.

[0035] 2. Unlike traditional systems that only support dual-arm and gripper control, this invention expands the control range, adding multi-degree-of-freedom control of the robot's head, waist, and chassis. Waist control significantly increases the operator's operating space, while chassis movement increases the robot's range of motion and flexibility. This design enables the robot to handle more complex and diverse tasks, significantly improving its adaptability and flexibility.

[0036] 3. Optimized real-time performance and stability: This invention optimizes the kinematic solution algorithm, increasing solution speed and reducing latency and lag. Through an optimized inverse kinematics algorithm and efficient computational process, the system can respond to operator commands in real time, ensuring smooth and accurate task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the flow chart of the method for remote control of a semi-humanoid robot of a VR device of the present invention.

[0038] Figure 2 This is a schematic diagram of the startup interface of the data stream tracking APP for remote operation of a semi-human robot that controls VR equipment in the present invention.

[0039] Figure 3 This is a schematic diagram of the connection of the hardware equipment for the remote control of the half-human robot of the VR device of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0041] The present invention provides Figure 1-Figure 3 A method for remote control of a semi-humanoid robot based on a VR device is shown, comprising the following steps:

[0042] Step 1: Data stream tracking and connection: Develop and compile a data stream tracking app, and install it on the VR device being used. The VR device will collect the operator's gestures and head relative position data in real time. During solution implementation, the compiled data stream tracking app will be installed on the Apple Vision Pro. The VR remote control system based on the Apple Vision Pro device will be used to control the movement of the humanoid robot. Combined with multi-degree-of-freedom control of the head, waist, and chassis, it provides an efficient and real-time remote control experience. The VR device will be able to collect the operator's gestures and head relative position data in real time.

[0043] It should be noted that VR devices can use other devices, and the specific devices are not limited. The present invention takes the Apple Vision Pro device as an example.

[0044] Step 2: Coordinate conversion of VR data stream: The coordinate systems of the human wrist and head in the VR data stream correspond one-to-one with the VR base coordinates; the coordinate systems of the wrist and head in the VR data stream are all relative to the VR base coordinates.

[0045] Step 3: Write an inverse kinematics solver based on C++ to solve the joint angles and gestures. The specific steps include:

[0046] A1. Arm and waist solution design: The inverse kinematics solution for the arm and waist is performed as a whole, with the first joint at the bottom of the waist as the base coordinate system. This invention uses C++ to write the inverse kinematics solution program. Compared with other high-level programming languages, C++ has higher execution efficiency and can therefore provide a faster solution process in real-time control systems.

[0047] It should be noted that in traditional inverse kinematics solutions, the choice of initial solution has a significant impact on solution speed. To further accelerate the solution, the present invention uses the current joint angle as a reference solution, based on which the optimization calculation is performed. By using the current joint angle as the starting point, the optimization algorithm can perform calculations closer to the target, greatly reducing the search space and the number of iterations required, thereby significantly improving the solution speed.

[0048] A2. Finger solution design: A control system based on finger bending angle and gestures realizes intelligent control of the robot chassis and arms. Through different finger postures, the operator can flexibly control the movement direction of the robot chassis and the movement of the arms.

[0049] A3. Head solution design: The head has two degrees of freedom; after coordinate transformation, it can be directly mapped to the robot joints.

[0050] A4. Mode switching: A "snapping fingers" gesture is designed based on chassis control and arm control to switch control modes. When the operator wants to switch the control mode, he can trigger the switching signal by "snapping his fingers".

[0051] Step 4: Send control instructions: Based on the joint angles, data is transmitted through ZeroMQ (ZMQ); the server transmits the data to the motors of each joint, and the motors move to complete the predetermined action.

[0052] It should be noted that the VR device and the robot host are connected to the same local area network; ensure that the VR device and the robot host are connected to the same local area network and start the data flow tracking app.

[0053] In the present invention, the avp_stream software package is installed in the robot host system, and the robot host accesses the gesture and head posture data in the VR device through code; in the robot host system, the avp_stream software package is installed, and when in use, the gesture and head posture data in the VR device can be accessed through code.

[0054] The pose of the robot wrist relative to the robot base coordinates is recorded as .

[0055] The transformation matrix of the VR base coordinate system relative to the robot base coordinate system is recorded as ; The pose of the human wrist in the VR base coordinate system is recorded as ; The transformation matrix of the robot wrist coordinate system relative to the human wrist coordinate system is recorded as .

[0056] The length ratio of the human arm and the robot arm is set as the scale factor, denoted as S, and scaled by the posture to simulate the actual proportional relationship:

[0057] .

[0058] It should be noted that the inverse kinematics solution calculates the motion angle of each joint based on the expected position and posture of the end effector; the goal of the inverse kinematics solution is to calculate the motion angle of each joint based on the expected position and posture of the end effector to meet the precise positioning requirements of the robot end. In traditional inverse kinematics solutions, the choice of the initial solution has a great influence on the solution speed.

[0059] In the present invention, the finger solution specifically includes:

[0060] Hands open, palms facing forward: This gesture indicates that the chassis is moving forward. The operator opens his hands, palms facing forward, simulating the action of walking, and the chassis moves forward.

[0061] Open hands, palms facing backward: This gesture indicates that the chassis moves backward. The operator opens his hands, palms facing backward, simulating the movement of moving backward, and the chassis moves backward.

[0062] Make a fist with the thumb pointing to the left: This gesture indicates that the chassis turns left. The operator makes a fist and points the thumb to the left, and the chassis turns left.

[0063] Make a fist with the thumb pointing to the right: This gesture indicates that the chassis turns right. The operator makes a fist and points the thumb to the right, and the chassis performs a right turn.

[0064] Furthermore, the "snapping fingers" gesture is realized by recognizing the finger bending angle and specific movements; if the current control mode is chassis control, the snapping finger gesture will switch to arm control mode, and vice versa, ensuring that the operator can flexibly switch between chassis control and arm control.

[0065] It should be noted that the head degrees of freedom do not need to be solved; the head has only two degrees of freedom and does not need to be solved.

[0066] In the present invention, the workflow is as follows:

[0067] 1. Using Apple Vision Pro as the VR device, a dedicated app was developed for this device to collect real-time pose data of the operator's fingers, wrists, and head. Using the Vuer framework and WebXR technology, images captured by the robot's Realsense camera were transmitted to the VR device in real time, allowing the operator to intuitively see the robot's surroundings in real time within the VR environment, enhancing their immersion and control precision.

[0068] 2. After receiving the pose data from the VR device, the robot control system host first processes and verifies the data. Since the transmitted data is relative to the VR device's base coordinate system, a coordinate system conversion is required. The specific process involves converting this data from the VR device's base coordinate system to the robot's base coordinate system, ensuring that the operator's movements are correctly mapped to the robot's motion space.

[0069] 3. After obtaining the converted pose data, the control system uses an inverse kinematics algorithm to calculate the target angles for each robot joint. The input data is the target position and pose of the robot's end effector. The inverse kinematics algorithm calculates the desired angle for each joint based on the robot's kinematic model.

[0070] 4. The joint angles obtained by inverse kinematics will be converted into control instructions and sent to the servo motors of each joint through the controller to control the robot to complete the predetermined action.

Claims

1. A method for remote control of a semi-humanoid robot based on VR equipment, characterized by: The following steps are involved: Step 1: Data flow tracking and connection: Develop and compile a data flow tracking app, install it on the VR device being used, and the VR device collects the operator's gestures and head position data in real time. Step 2: Coordinate conversion of VR data stream: The coordinate system of the human wrist and head in the VR data stream corresponds one-to-one with the VR base coordinate system; Step 3: Write an inverse kinematics solver based on C++ to solve the joint angles and gestures. The specific steps include: A1. Design of arm and waist: Perform inverse kinematics on the arm and waist as a whole, using the first joint at the bottom of the waist as the base coordinate system. A2. Finger solution design: A control system based on finger bending angle and gestures to achieve intelligent control of the robot chassis and arms; A3. Head solution design: The head has two degrees of freedom; A4. Mode switching: A "snapping finger" gesture is designed based on chassis control and arm control to switch control modes; Step 4: Send control instructions: Based on the joint angles, data is transmitted through ZeroMQ (ZMQ); the server transmits the data to the motors of each joint, and the motors move to complete the predetermined action.

2. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The VR device and the robot host are connected to the same local area network.

3. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The robot host system is installed with the avp_stream software package, and the robot host accesses the gesture and head posture data in the VR device through code.

4. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The position of the robot wrist relative to the robot base coordinates is recorded as , The pose calculation based on the robot wrist relative to the robot base coordinates can be obtained: ； The transformation matrix of the VR base coordinate system relative to the robot base coordinate system is recorded as The position of the human wrist in the VR base coordinate system is recorded as The transformation matrix of the robot wrist coordinate system relative to the human wrist coordinate system is recorded as .

5. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 4, characterized in that: The length ratio of the human arm and the robot arm is set as the scale factor, denoted as S, and scaled by the posture to simulate the actual proportional relationship: 。 6. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The inverse kinematics solution calculates the motion angle of each joint based on the desired position and posture of the end effector.

7. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The finger solution specifically includes: Open your hands and face forward: This gesture indicates that the chassis is moving forward; Open your hands, facing backwards: This gesture means the chassis is moving backwards; Make a fist with the thumb pointing to the left: This gesture indicates that the chassis turns left; Make a fist with your thumb pointing to the right: This gesture indicates that the chassis is turning right.

8. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The "snapping fingers" gesture is achieved by identifying the bending angle of the fingers and specific movements.

9. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The head degrees of freedom do not need to be solved.

Images