Half-humanoid robot teleoperation control method based on VR equipment

Abstract

The invention discloses a semi-humanoid robot teleoperation control method based on VR equipment, and relates to the field of VR equipment, and the method comprises the following steps: 1, data stream tracking and connection: developing and compiling a data stream tracking APP, installing the APP on the used VR equipment, and carrying out the real-time collection of the gesture of an operator and the relative pose data of the head through the VR equipment; and step 2, coordinate conversion of the VR data stream: the coordinate systems of the wrist and the head in the VR data stream are in one-to-one correspondence with VR base coordinates. And 3, writing an inverse kinematics solution program based on C + +, and carrying out solution design on the joint angle and the gesture. The visual angle limitation of an existing system is broken through, the head control function of VR equipment is combined, the movement of the head of the robot can be adjusted in real time along with the rotation of the head of an operator, and therefore dynamic first visual angle switching is achieved. The immersion feeling of an operator is enhanced, the operation accuracy is improved, and the operator can better perform task operation in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of VR equipment, and in particular to a remote operation control method of a semi-human robot based on VR equipment. Background Art

[0002] In recent years, with the development of robotics technology, semi-human robots have been increasingly used in industrial manufacturing, service industries, and special environment operations. In order to improve the ability of robots to perform complex tasks, teleoperation technology has become a research hotspot, among which teleoperation methods based on virtual reality (VR) devices are particularly eye-catching. VR technology significantly enhances the operator's perception ability and control accuracy of the robot through an immersive operating interface and intuitive interaction. At present, many teleoperation systems focus on using VR devices to perform fine operations on the robot's arms and grippers, and have achieved certain results. However, in order to adapt to the diverse needs of complex scenes, relying solely on the control of arms and grippers can no longer fully meet the flexibility and extensiveness requirements in practical applications.

[0003] The existing remote operation systems based on VR devices have the following main shortcomings:

[0004] 1. Limited viewing angle: Existing systems usually adopt the third perspective or a fixed first perspective. This method cannot flexibly adjust the movement direction of the robot head, resulting in the operator being unable to obtain dynamic viewing angle switching, reducing the immersion and operation accuracy.

[0005] 2. Insufficient freedom of movement: Most systems only support the control of the arms and grippers, and lack the ability to control the waist and chassis movement of the robot. Waist movement can greatly increase the operating space, and chassis control can expand the range of movement and flexibility, which are particularly important in complex operating tasks.

[0006] 3. Delay and jamming problems: During the remote operation process, existing systems often experience delays and jamming due to complex control algorithms or slow solution speeds, which affects the real-time and smoothness of task execution.

[0007] Therefore, it is necessary to propose a semi-human robot teleoperation control method based on VR equipment to solve the above problems. Summary of the invention

[0008] The purpose of the present invention is to provide a remote operation control method of a semi-human robot based on a VR device to solve the problems raised in the above background technology.

[0009] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: A remote operation control method of a semi-humanoid robot based on a VR device, comprising the following steps: Step 1, data stream tracking and connection: Develop and compile a data stream tracking APP, install the APP on the VR device in use, and the VR device collects the operator's gestures and head relative posture data in real time;

[0010] Step 2: Coordinate transformation of VR data stream: The coordinate systems of the wrist and head in the VR data stream correspond one-to-one with the VR base coordinates.

[0011] Step 3: Write an inverse kinematics solver based on C++ to solve the joint angles and gestures. The specific steps include:

[0012] A1. Solution design for arms and waist: Take the arms and waist as a whole for inverse kinematics solution, and take the first joint at the bottom of the waist as the base coordinate system.

[0013] A2. Finger solution design: A control system based on finger bending angle and gestures to achieve intelligent control of the robot chassis and arms.

[0014] A3. Head solution design: The head has two degrees of freedom.

[0015] A4. Mode switching: A "snapping fingers" gesture is designed based on chassis control and arm control to switch control modes.

[0016] Step 4: Send control instructions: Based on the joint angle, data is transmitted through ZeroMQ (ZMQ); the server transmits it to the motors of each joint, and the motors move to complete the predetermined action.

[0017] Preferably, the VR device and the robot host are connected to the same local area network.

[0018] Preferably, the robot host system is installed with an avp_stream software package, and the robot host accesses the gesture and head posture data in the VR device through code.

[0019] Preferably, the position of the robot wrist relative to the robot base coordinates is recorded as

[0020] The position calculation based on the robot wrist relative to the robot base coordinates can be obtained:

[0021]

[0022] Preferably, the transformation matrix of the VR base coordinate system relative to the robot base coordinate system is expressed as The position of the human wrist in the VR base coordinate system is recorded as The transformation matrix of the robot wrist coordinate system relative to the human wrist coordinate system is recorded as

[0023] Preferably, the length ratio of the human arm and the robot arm is set as a scale factor, denoted as S, and scaled by posture to simulate the actual proportional relationship:

[0024]

[0025] Preferably, the inverse kinematics solution calculates the motion angle of each joint based on the desired position and posture of the end effector.

[0026] Preferably, the finger solution specifically includes:

[0027] Open your hands and face forward: This gesture indicates that the chassis is moving forward;

[0028] Open your hands, facing backwards: This gesture means the chassis is moving backwards;

[0029] Make a fist with the thumb pointing to the left: This gesture indicates that the chassis is turning left;

[0030] Make a fist with the thumb pointing to the right: This gesture indicates that the chassis is turning right.

[0031] Preferably, the "snapping fingers" gesture is achieved by identifying the bending angle of the fingers and a specific action.

[0032] Preferably, the head degrees of freedom do not need to be solved.

[0033] Technical effects and advantages of the present invention:

[0034] 1. This invention breaks through the viewing angle limitation of the existing system. Combined with the head control function of the VR device, the movement of the robot head can be adjusted in real time as the operator's head turns, thereby realizing dynamic first-person perspective switching. This not only enhances the operator's sense of immersion, but also improves the accuracy of operation, enabling it to better perform task operations in complex environments.

[0035] 2. Unlike traditional systems that only support dual-arm and gripping control, this invention expands the control range and adds multi-degree-of-freedom control of the robot's head, waist and chassis. Through waist control, the operator can greatly increase the operating space, while chassis movement can increase the robot's range of motion and flexibility. This design enables the robot to cope with more complex and changing task requirements, significantly improving the robot's adaptability and flexibility.

[0036] 3. Optimized real-time performance and stability: The present invention optimizes the kinematic solution algorithm, improves the solution speed, and reduces delays and freezes. Through the optimized inverse kinematics algorithm and efficient calculation process, the system can respond to the operator's instructions in real time to ensure the smoothness and accuracy of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic flow chart of the semi-human robot teleoperation control method of the VR device of the present invention.

[0038] Figure 2 This is a schematic diagram of the startup interface of the semi-human robot teleoperation control data stream tracking APP for controlling VR equipment in the present invention.

[0039] Figure 3 This is a schematic diagram of the connection of the semi-human robot teleoperation control hardware device of the VR device of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0041] The present invention provides Figure 1-Figure 3 A method for remote control of a semi-humanoid robot based on a VR device is shown, comprising the following steps:

[0042] Step 1: Data stream tracking and connection: Develop and compile a data stream tracking APP, install the APP on the VR device in use, and the VR device will collect the operator's gestures and relative head posture data in real time; when implementing the solution, install the developed compiled data stream tracking APP on Apple Vision Pro, and control the movement of the semi-human robot based on the VR remote control system of the Apple Vision Pro device. Combined with the multi-degree-of-freedom control of the head, waist and chassis, it provides an efficient and real-time remote control experience. The VR device can collect the operator's gestures and relative head posture data in real time.

[0043] It should be noted that VR devices can use other devices, and the specific devices are not limited. The Apple Vision Pro device is taken as an example in the present invention.

[0044] Step 2: Coordinate conversion of VR data stream: The coordinate systems of the wrist and head in the VR data stream correspond one-to-one to the VR base coordinates; the coordinate systems of the wrist and head in the VR data stream are all relative to the VR base coordinates.

[0045] Step 3: Write an inverse kinematics solver based on C++ to solve the joint angles and gestures. The specific steps include:

[0046] A1. Design of arm and waist solution: The arm and waist are taken as a whole for inverse kinematics solution, and the first joint at the bottom of the waist is taken as the base coordinate system; the present invention adopts C++ to write the inverse kinematics solution program. Compared with other high-level programming languages, C++ has higher execution efficiency, so it can provide a faster solution process in the real-time control system.

[0047] It should be noted that in traditional inverse kinematics solutions, the choice of the initial solution has a great influence on the solution speed. In order to further speed up the solution speed, the present invention uses the current joint angle as the reference solution and performs optimization calculations on this basis. By using the current joint angle as the starting point, the optimization algorithm can perform calculations closer to the target, greatly reducing the search space and number of iterations of the calculation, thereby significantly improving the solution speed.

[0048] A2. Finger solution design: A control system based on finger bending angle and gestures realizes intelligent control of the robot chassis and arms; through different finger postures, the operator can flexibly control the movement direction of the robot chassis and the movement of the arms.

[0049] A3. Head solution design: The head has two degrees of freedom; after coordinate transformation, it can directly correspond to the robot joints.

[0050] A4. Mode switching: A "snapping fingers" gesture is designed based on chassis control and arm control to switch control modes; when the operator wants to switch the control mode, the switching signal can be triggered by "snapping fingers".

[0051] Step 4: Send control instructions: Based on the joint angle, data is transmitted through ZeroMQ (ZMQ); the server transmits it to the motors of each joint, and the motors move to complete the predetermined action.

[0052] It should be noted that the VR device and the robot host are connected to the same local area network; make sure that the VR device and the robot host are connected to the same local area network and start the data flow tracking APP.

[0053] In the present invention, the avp_stream software package is installed in the robot host system, and the robot host accesses the gesture and head posture data in the VR device through code; in the robot host system, the avp_stream software package is installed, and when in use, the gesture and head posture data in the VR device can be accessed through code.

[0054] The position of the robot wrist relative to the robot base coordinates is recorded as

[0055] The transformation matrix of the VR base coordinate system relative to the robot base coordinate system is recorded as The pose of the human wrist in the VR base coordinate system is recorded as The transformation matrix of the robot wrist coordinate system relative to the human wrist coordinate system is recorded as

[0056] The length ratio of the human arm and the robot arm is set as the scale factor, denoted as S, and scaled by the posture to simulate the actual proportional relationship:

[0057]

[0058] It should be noted that the inverse kinematics solution calculates the motion angle of each joint based on the expected position and posture of the end effector; the goal of the inverse kinematics solution is to calculate the motion angle of each joint based on the expected position and posture of the end effector to meet the precise positioning requirements of the robot end. In the traditional inverse kinematics solution, the choice of the initial solution has a great influence on the solution speed.

[0059] In the present invention, the finger solution specifically includes:

[0060] Hands open, palms facing forward: This gesture indicates that the chassis is moving forward; This gesture indicates that the chassis is moving forward. The operator opens his hands, palms facing forward, simulating the action of walking, and the chassis moves forward.

[0061] Hands open, palms facing backward: This gesture indicates that the chassis moves backward; This gesture indicates that the chassis moves backward. The operator opens his hands, palms facing backward, simulating the movement of moving backward, and the chassis moves backward.

[0062] Fist, thumb pointing to the left: This gesture indicates that the chassis turns left; This gesture indicates that the chassis turns left. The operator clenched his fist and pointed his thumb to the left, and the chassis executed a left turn.

[0063] Fist, thumb pointing to the right: This gesture indicates that the chassis turns right; This gesture indicates that the chassis turns right. The operator makes a fist and points the thumb to the right, and the chassis performs a right turn.

[0064] Furthermore, the "snapping fingers" gesture is realized through the recognition of finger bending angle and specific movements; if the current control mode is chassis control, the snapping fingers gesture will switch to arm control mode, and vice versa, ensuring that the operator can flexibly switch between chassis control and arm control.

[0065] It should be noted that the head degrees of freedom do not need to be solved; the head has only two degrees of freedom and does not need to be solved.

[0066] In the present invention, the workflow is as follows:

[0067] 1. Use Apple Vision Pro as VR device, and develop a dedicated application (App) on the device to collect the operator's finger, wrist and head posture data in real time. Through the Vuer framework and WebXR technology, the images collected by the robot's Realsense camera are transmitted to the VR device in real time, allowing the operator to intuitively see the real-time view around the robot in the VR environment, improving the operator's immersion and control accuracy.

[0068] 2. After the robot control system host receives the posture data from the VR device, it first processes and verifies the data. Since the transmitted data is relative to the base coordinate system of the VR device, a coordinate system conversion is required. The specific process is to convert this data from the base coordinate system of the VR device to the base coordinate system of the robot to ensure that the operator's actions can be correctly mapped to the robot's motion space.

[0069] 3. After obtaining the converted posture data, the control system uses the inverse kinematics algorithm to calculate the target angle of each joint of the robot. The input data is the target position and posture of the robot's end effector. The inverse kinematics algorithm calculates according to the robot's kinematic model to obtain the angle that each joint needs to achieve.

[0070] 4. The joint angles obtained by inverse kinematics will be converted into control instructions and sent to the servo motors of each joint through the controller to control the robot to complete the predetermined action.

Claims

1. A remote control method for a semi-humanoid robot based on VR equipment, characterized in that: The following steps are involved: Step 1: Data flow tracking and connection: Develop and compile a data flow tracking APP, install the APP on the VR device used, and the VR device collects the operator's gestures and head relative posture data in real time; Step 2: Coordinate transformation of VR data stream: The coordinate systems of the wrist and head in the VR data stream correspond one-to-one with the VR base coordinates. Step 3: Write an inverse kinematics solver based on C++ to solve the joint angles and gestures. The specific steps include: A1. Solution design for arms and waist: Take the arms and waist as a whole for inverse kinematics solution, and take the first joint at the bottom of the waist as the base coordinate system. A2. Finger solution design: A control system based on finger bending angle and gestures to achieve intelligent control of the robot chassis and arms. A3. Head solution design: The head has two degrees of freedom. A4. Mode switching: A "snapping fingers" gesture is designed based on chassis control and arm control to switch control modes. Step 4: Send control instructions: Based on the joint angle, data is transmitted through ZeroMQ (ZMQ); the server transmits it to the motors of each joint, and the motors move to complete the predetermined action.

2. A method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The VR device and the robot host are connected to the same local area network.

3. A method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The robot host system is installed with the avp_stream software package, and the robot host accesses the gesture and head posture data in the VR device through code.

4. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The position of the robot wrist relative to the robot base coordinates is recorded as The position calculation based on the robot wrist relative to the robot base coordinates can be obtained:

5. A method for remote control of a semi-humanoid robot based on VR equipment according to claim 4, characterized in that: The transformation matrix of the VR base coordinate system relative to the robot base coordinate system is recorded as The position of the human wrist in the VR base coordinate system is recorded as The transformation matrix of the robot wrist coordinate system relative to the human wrist coordinate system is recorded as 6. A method for remote control of a semi-humanoid robot based on VR equipment according to claim 5, characterized in that: The length ratio of the human arm and the robot arm is set as a scale factor, denoted as S, and scaled by posture to simulate the actual proportional relationship:

7. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The inverse kinematics solution calculates the motion angle of each joint according to the desired position and posture of the end effector.

8. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The finger solution specifically includes: Open your hands and face forward: This gesture indicates that the chassis is moving forward; Open your hands, facing backwards: This gesture means the chassis is moving backwards; Make a fist with the thumb pointing to the left: This gesture indicates that the chassis is turning left; Make a fist with the thumb pointing to the right: This gesture indicates that the chassis is turning right.

9. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The "snapping fingers" gesture is achieved by identifying the bending angle of the fingers and specific movements.

10. The method for remote control of a semi-humanoid robot based on VR equipment according to claim 1, characterized in that: The head degrees of freedom do not need to be solved.

Images