Robot teleoperation system, method and equipment based on VR head-mounted display

Through the robot remote operating system based on VR headset, the operator's body posture data is collected using motion capture equipment and VR headset, and combined with the robot control algorithm to generate control signals, the problem of mismatch in the redundancy degree of freedom of the robot arm is solved, and the problem of robotic arm is realized More precise control and wider viewing.

CN118927245APending Publication Date: 2024-11-12GUANGZHOU PENGXING INTELLIGENT TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411034463.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

During the remote operation of the robot, the operation handle is used to collect the 6-degree of freedom position information at the end of the operator's arm and the redundant degree of freedom between the 7-degree of freedom robot arm that controls the humanoid, causing the robot arm to be out of control in a strange position. Or a specific movement cannot be achieved.

Method used

Using a robot remote operating system based on VR headset, the position data of the operator's body parts is collected through the motion capture device, and the filtering and algorithm conversion are combined with the VR headset and the upper computer to generate control signals to control the robot's waist and arms. , hands and neck movements, and use the excess data collected by the motion capture equipment and VR headset to control the redundant degree of freedom of the robotic arm.

Benefits of technology

It effectively solves the control problem of the robotic arm in a singular position, improves control accuracy, overcomes the limitations of fixed camera viewing angle, and allows operators to observe the surrounding environment of the robot through the VR headset.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118927245A_ABST
    Figure CN118927245A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of robot teleoperation, and discloses a robot teleoperation system, method and device based on a VR head-mounted display, the system comprises a motion capture device, the VR head-mounted display, an upper computer and a robot, the motion capture device is used for collecting poses of one or more body parts of a worn person through at least one IMU and sending the poses to the VR head-mounted display; the VR head-mounted display is used for filtering poses in the motion capture signals and sending the filtered poses of one or more body parts and the poses of the VR head-mounted display to the upper computer; the upper computer is used for processing the pose and the VR head display pose according to a robot control algorithm to generate at least one control signal and sending the control signal to the robot, and the robot is used for receiving the at least one control signal, controlling a specified part of the robot to execute corresponding motion according to the at least one control signal and increasing the redundant degree of freedom for controlling the mechanical arm. Therefore, the problem that the mechanical arm is out of control at the singular position is effectively solved, and the control precision of the robot is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of robot teleoperation, and in particular to a robot teleoperation system, method and device based on a VR head display. Background Art

[0002] With the development of robot intelligent control technology, data-driven robot technology is gradually being recognized. Therefore, it is particularly important to efficiently and accurately collect robot motion data through teleoperation to train its own intelligent model. In the process of teleoperation, the combination of VR (Virtual Reality) headsets enables the operator to obtain images from the robot's perspective and avoid the robot body or other environmental obstacles blocking the operator's line of sight, which helps to improve the accuracy of teleoperation and improve data quality.

[0003] At present, in the process of remote operation of robots and their control, an operating handle is mostly used to collect the operator's motion posture. If the operating handle is used to collect the 6-DOF posture information of the operator's arm end, the degrees of freedom are different from those of the 7-DOF humanoid robotic arm controlled by it. For example, if one or more redundant degrees of freedom are not controlled, the robotic arm will be out of control in a singular position or unable to achieve specific movements. Summary of the invention

[0004] In view of this, the present invention provides a robot teleoperation system, method and device based on a VR head display to solve the problem of inaccurate robot control.

[0005] In a first aspect, the present invention provides a robot teleoperation system based on a VR headset, the system comprising: a motion capture device, a VR headset, a host computer and a robot, wherein the motion capture device is a wearable device and is equipped with at least one inertial measurement unit IMU;

[0006] The motion capture device is used to collect the posture of one or more body parts of the wearer through the at least one IMU, and send a motion capture signal carrying the posture to the VR head display;

[0007] The VR head display is used to filter the posture in the motion capture signal, and send the filtered posture of one or more body parts and the VR head display posture to the host computer;

[0008] The host computer is used to receive the filtered posture of one or more body parts and the posture of the VR head display sent by the VR head display, process the posture and the posture of the VR head display according to the robot control algorithm, generate at least one control signal, and send the at least one control signal to the robot, wherein the at least one control signal is used to control the movement of at least one part of the waist, arms, hands and neck of the robot;

[0009] The robot is used to receive at least one control signal sent by the host computer, and control a designated part of the robot to perform corresponding movement according to the at least one control signal.

[0010] In combination with the first aspect, in a possible implementation manner, the postures of one or more body parts collected by the motion capture device include: the relative postures of the waist, left / right shoulders, left / right upper arms, left / right elbows, and left / right wrists;

[0011] The host computer processes the posture and the VR head display posture according to the robot control algorithm, generates at least one control signal, and sends the at least one control signal to the robot, including:

[0012] The host computer performs matrix transformation, coordinate system alignment, Euler angle conversion and extraction processing on the filtered waist relative posture according to the first control algorithm to generate a first control signal including the robot waist control angle;

[0013] The host computer optimizes the relative positions of the robot's waist control angle, left / right shoulder, left / right upper arm, left / right elbow and left / right wrist according to the second control algorithm, and calculates a second control signal for the robot's mechanical arm to move in at least one degree of freedom;

[0014] The host computer sends the first control signal and the second control signal to the robot.

[0015] In combination with the first aspect, in another possible implementation manner, the host computer performs matrix transformation, coordinate system alignment, Euler angle conversion and extraction processing on the filtered waist relative posture according to the first control algorithm to generate a first control signal including the robot waist control angle, including:

[0016] Converting the filtered waist relative posture into a first homogeneous coordinate transformation matrix;

[0017] Multiplying the first homogeneous coordinate transformation matrix by a homogeneous coordinate transformation matrix including rotation and translation so as to align the world coordinate system of the waist relative position with the first origin and orientation parameters of the robot world coordinate system;

[0018] Converting the aligned first origin and the orientation parameter into a first Euler angle, wherein the first Euler angle includes a plurality of angles with a plurality of degrees of freedom;

[0019] A target angle corresponding to a target degree of freedom is extracted from a plurality of angles of a plurality of degrees of freedom of the first Euler angle, and a first control signal including the target angle is generated.

[0020] In combination with the first aspect, in another possible implementation, the host computer optimizes the relative posture of the robot's waist control angle, left / right shoulder, left / right upper arm, left / right elbow and left / right wrist according to the second control algorithm, and calculates a second control signal for the robot's mechanical arm to move in at least one degree of freedom, including:

[0021] According to the relative postures of the robot waist control angle, the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist, based on the robot kinematics algorithm and coordinate transformation calculation, the posture of the left / right wrist relative to the robot coordinate system is obtained;

[0022] Performing kinematic processing on the relative posture of the left / right upper arm and the left / right elbow to obtain the rotation angle of the redundant joint of the manipulator with at least one degree of freedom;

[0023] The posture of the left / right wrist relative to the robot coordinate system and the rotation angle of the redundant joint of the at least one degree of freedom robotic arm are taken as optimization targets, the torque of each joint of the robotic arm is taken as optimization object, and the physical performance of the motor of each joint of the robotic arm is taken as constraint conditions, a multi-objective optimization function is constructed, and the multi-objective optimization function is solved to obtain the second control signal for the robotic arm to move in at least one degree of freedom.

[0024] In combination with the first aspect, in another possible implementation, the relative postures of the left / right wrist relative to the robot coordinate system are obtained based on the robot kinematics algorithm and coordinate transformation calculation according to the robot waist control angle, the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist, including:

[0025] The relative postures of the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist form two kinematic chains from the waist to the left / right wrist, and the postures of the left / right wrist on the two kinematic chains relative to the waist coordinate system are calculated based on the robot forward kinematics algorithm, and the postures are converted into a second homogeneous coordinate transformation matrix;

[0026] Multiplying the second homogeneous coordinate transformation matrix by a homogeneous coordinate transformation matrix including rotation and translation to align the world coordinate system of the motion capture signal with the second origin and orientation data of the robot waist coordinate system;

[0027] Perform an inverse operation on the robot waist control angle to obtain a corresponding third homogeneous coordinate transformation matrix;

[0028] The aligned second origin and the orientation data are multiplied by the third homogeneous coordinate transformation matrix to obtain the position and posture of the left / right wrist relative to the robot coordinate system.

[0029] In combination with the first aspect, in another possible implementation manner, after the host computer generates the first control signal including the waist control angle of the robot, the method further includes:

[0030] Filtering the VR head display posture to obtain a filtered VR head display posture;

[0031] Convert the filtered VR head display posture into a fourth homogeneous coordinate transformation matrix, and multiply it by the inverse of the homogeneous coordinate transformation matrix converted by the first control signal to obtain the posture of the neck relative to the waist;

[0032] Converting the posture of the neck relative to the waist into a corresponding second Euler angle, wherein the second Euler angle includes at least one angle of at least one degree of freedom;

[0033] extracting a target angle corresponding to a target degree of freedom from at least one angle of at least one degree of freedom of the second Euler angle, and generating a third control signal including the target angle;

[0034] The third control signal is sent to the robot, and the third control signal is used to control the movement of the neck part of the robot.

[0035] In combination with the first aspect, in another possible implementation, the postures of one or more body parts collected by the motion capture device include: the relative postures of the left / right wrist and each finger of the hand;

[0036] The host computer processes the posture and the VR head display posture according to the robot control algorithm, generates at least one control signal, and sends the at least one control signal to the robot, including:

[0037] The host computer performs kinematics algorithm, Euler angle conversion and extraction processing on the relative postures of the left / right wrist and each finger of the hand after filtering, and generates a fourth control signal containing each finger of the robot hand in at least one degree of freedom;

[0038] The method further comprises:

[0039] The host computer sends the fourth control signal to the robot, and the fourth control signal is used to control the movement of the hand part of the robot.

[0040] In combination with the first aspect, in another possible implementation manner, a binocular camera is installed on the head of the robot;

[0041] The robot is also used to collect images of the corresponding movements and status information of the designated parts of the robot in real time through the binocular camera, and feed the images and status information back to the VR head display;

[0042] The VR head display is also used to receive the image and the status information fed back from the robot, and update the Unity scene screen in real time according to the image and the status information.

[0043] In combination with the first aspect, in another possible implementation, the VR headset is also used to perform kinematic transformation on the posture of one or more filtered body parts, align the motion capture signal with the coordinate system origin and orientation of the dummy model in the Unity scene, and update the dummy model movement in the Unity scene.

[0044] In a second aspect, the present invention provides a robot teleoperation method based on a VR head display, which is applied to a host computer, and the method comprises:

[0045] Receive a posture sent by a VR head display and a VR head display posture, wherein the posture sent by the VR head display is output after the VR head display performs filtering processing on the posture of one or more body parts of the wearer collected by a motion capture device, and the VR head display posture is the posture of the VR head display device itself.

[0046] Processing the posture sent by the VR head display and the posture of the VR head display according to a robot control algorithm to generate at least one control signal;

[0047] The at least one control signal is sent to the robot, and the at least one control signal is used to control the movement of at least one part of the waist, arms, hands and neck of the robot.

[0048] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the robot remote operation method based on the VR head display shown in the second aspect above by executing the computer instructions.

[0049] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the robot remote operation method based on a VR head display as shown in the second aspect above.

[0050] In addition, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the robot remote operation method based on VR head display shown in the second aspect above.

[0051] The robot teleoperation system, method and device provided by the present invention collects real-time posture data of designated parts of the wearer's body through a motion capture device equipped with an IMU and a VR head display. After receiving these posture data, the host computer converts the motion capture signal into at least one control signal of the robot's waist, arms, hands and neck through the robot control algorithm, and sends it to the robot to control the relevant parts. The beneficial effects of this solution include the following:

[0052] 1. Compared with the operating handle, the use of motion capture equipment to collect the operator's arm movements can not only collect the wrist posture, but also the shoulder, upper arm, and elbow posture. The extra data can be used to control the redundant degrees of freedom of the robotic arm after kinematic conversion, effectively solving control problems such as the robotic arm "flying" in strange positions and abnormal configurations.

[0053] Second, compared with the operating arm, the operator's arm movement collected by the motion capture device is consistent with the actual movement of the robotic arm after conversion by the control algorithm, which effectively solves the problem of insufficient control accuracy when the input signal size is smaller than the output signal size.

[0054] 3. Use a VR headset to collect the operator's neck movements, and then convert them into neck motor control signals through a control algorithm, so that the operator can observe the robot's surroundings through the VR headset, overcoming the limitations of a fixed camera perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 is an architecture diagram of a robot teleoperation system provided by an embodiment of the present invention;

[0057] Figure 2 is a schematic structural diagram of a robot provided by an embodiment of the present invention;

[0058] Figure 3 This is a signaling flow chart of a robot teleoperation based on a VR head display provided by an embodiment of the present invention;

[0059] Figure 4is a schematic diagram of a process flow of robot teleoperation provided by an embodiment of the present invention;

[0060] Figure 5 is a schematic diagram of another process of robot teleoperation provided by an embodiment of the present invention;

[0061] Figure 6 is a schematic diagram of another process of robot teleoperation provided by an embodiment of the present invention;

[0062] Figure 7 is a structural block diagram of another robot teleoperation device provided by an embodiment of the present invention;

[0063] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0065] In the description of the present invention, the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and do not require that the present invention must be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on the present invention. The terms "connected" and "connected" used in the present invention should be understood in a broad sense. For example, it can be a fixed connection or a detachable connection; it can be a direct connection, or it can be indirectly connected through an intermediate component, or it can be wirelessly connected through a network. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0066] The technical solution provided by the embodiment of the present invention is applied to the field of teleoperation technology. Teleoperation refers to a remote control technology that remotely monitors and controls a robot or device through a communication system and a control interface at a place far away from the machine or device, so that the robot or device operates as expected. The purpose of studying the robot's telepresence technology is to provide the operator with a more complete teleoperation immersion and real experience, laying the foundation for the robot to complete more complex and semi-autonomous operations.

[0067] The technical solution provided by the embodiments of the present invention is described and introduced below.

[0068] like Figure 1 , which is an architecture diagram of a robot teleoperation system provided by an embodiment of the present invention. The system includes: a motion capture device 100, a VR head display 200, a host computer 300 and a robot 400.

[0069] Among them, multiple IMUs (Inertial Measurement Units) are installed on the motion capture device 100. IMU is an electronic device used to measure and report specific mechanical motion characteristics of an object, which usually include acceleration, angular velocity (sometimes also angular displacement, direction or temperature, etc.). IMU is widely used in various fields that require navigation, positioning and attitude control, such as aerospace, robotics, drones, self-driving cars, virtual reality (VR) and augmented reality (AR). In the present invention, an IMU is installed on the waist, back, shoulder, upper arm, arm, back of the hand, and fingertips of the motion capture device 100 to measure and report the posture of these parts.

[0070] In addition, the motion capture device 100 is a wearable device.

[0071] VR head display 200, namely Virtual Reality (VR) head mounted display device, is a device that uses head mounted display device to block the visual and auditory senses of the user to the outside world, and guides the user to feel as if they are in a virtual 3D environment. Its display principle is that the left and right eye screens display the left and right eye images respectively, and the human eye obtains this information with differences and generates a three-dimensional sense in the mind.

[0072] The principle is that the VR head display 200 displays slightly different images through the left and right eye screens, simulating the binocular parallax of human eyes when viewing the real world, thereby creating a three-dimensional sense in the user's mind. At the same time, combined with head tracking technology, it ensures that the movement of the user's head can be reflected in the virtual environment in real time, enhancing the sense of immersion.

[0073] The host computer 300 refers to a computer or single-chip microcomputer that can directly send operation instructions, generally provides a user operation interactive interface and displays feedback data to the user. It has functions such as real-time monitoring and data acquisition, remote control and operation, alarm and fault diagnosis, data analysis and optimization, and visual interface.

[0074] In the embodiment of the present invention, the host computer 300 may be a terminal device, such as a PC, a laptop, an all-in-one machine, an intelligent terminal such as a smart phone, a tablet computer, or other forms of terminal devices, which are not limited in the embodiment of the present invention.

[0075] Robot 400 is an intelligent robot. Figure 2As shown in the figure, the head of the intelligent robot is equipped with a binocular camera. As a camera system with two lenses, the binocular camera captures images of the same scene from different perspectives through two lenses, and uses the parallax between these images to calculate the depth information of the object through computer vision technology. Compared with monocular cameras, binocular cameras can provide more accurate three-dimensional spatial coordinates, which is crucial for applications that require precise measurement and positioning.

[0076] In addition, the binocular camera also has a dynamic scene capture function. The binocular camera can capture the three-dimensional information of the dynamic scene, such as the movement and displacement of objects. This allows the binocular camera to maintain high recognition accuracy and stability in a dynamic environment. In an embodiment of the present invention, the binocular camera is installed on the head of the robot to capture the external environment image "seen" by the robot when it moves.

[0077] exist Figure 2 In the embodiment, the robot 400 is provided with 2 degrees of freedom for rotation on the neck and waist, for example, movement on the x-axis and the z-axis. The robot arm includes 7 degrees of freedom for movement, and the robot dexterous hand includes 6 degrees of freedom for movement. In addition, optionally, redundant degrees of freedom are provided for the robot arm to improve the flexibility of the robot's activities.

[0078] In the present invention, the motion capture device 100 uses multiple IMU sensors to collect the posture of the corresponding position when the operator moves. The Unity scene is run on the VR headset 200 to update the robot's perspective image, the movement of the motion capture signal on the dummy, and the movement of the robot feedback on the robot model in real time. The host computer 300 is used to run the control algorithm to convert the motion capture signal and the headset posture into a control signal of the corresponding degree of freedom of the robot. Finally, the robot 400 executes the control signal to realize the remote operation of the operator's movement to the robot's movement, and feeds back the robot's real-time binocular camera image and status to the VR headset 200.

[0079] The embodiment of the present invention provides a robot teleoperation system based on a VR head display. Figure 1 The framework structure shown can be used to implement Figure 3 A robot teleoperation method based on VR head display is shown.

[0080] like Figure 3 As shown, the method comprises the following steps:

[0081] Step S101: The motion capture device is used to collect the posture of one or more body parts of the wearer through at least one IMU.

[0082] Among them, one or more body parts include: the posture of the wearer's waist, shoulder, upper arm, arm, back of hand, and fingertips. The posture contains the position (position) and orientation (orientation) information of an object or robot relative to a reference coordinate system. Among them, the position: refers to the coordinate point of an object or robot in three-dimensional space, usually represented by numerical values ​​on the three coordinate axes of x, y, and z. These numerical values ​​describe the specific position of the object in space. The direction: also called posture, is used to describe the direction in which the object or robot is facing. Direction information can be represented in a variety of ways, such as Euler angles (including pitch angle pitch, yaw angle yaw and roll angle roll), quaternions, axis angles, and posture matrices.

[0083] Step S102: The motion capture device sends a motion capture signal carrying the posture to the VR headset.

[0084] The motion capture device packages one or more postures in the above step S101 into a motion capture signal, and sends the motion capture signal to the VR head display. The transmission medium can be transmitted via a wireless network.

[0085] Correspondingly, the VR headset receives the motion capture signal sent by the motion capture device.

[0086] Step S103: The VR head display is used to filter the posture in the motion capture signal.

[0087] Specifically, in the Unity scene of the VR headset, median filtering is performed on each posture in the motion capture signal to remove abnormal signals in the motion capture signal, and the posture of one or more body parts after filtering is obtained.

[0088] It should be noted that this step is optional. If there is no abnormality or noise in one or more of the poses in the motion capture signal, there is no need to filter it. Only the poses with abnormal signals or interference or noise signals are filtered to obtain clean and noise-free poses.

[0089] In addition, in this step, the VR headset also obtains its own posture, that is, the VR headset posture.

[0090] Step S104: The VR headset sends the filtered posture of one or more body parts and the VR headset posture to the host computer. The transmission medium is still transmitted through a wireless network.

[0091] Correspondingly, the host computer receives the posture of one or more body parts and the VR headset posture sent from the VR headset.

[0092] Step S105: After receiving the information, the host computer processes the posture and the VR head display posture according to the robot control algorithm to generate at least one control signal.

[0093] The at least one control signal is used to control the movement of at least one of the waist, arms, hands and neck of the robot. Each control signal includes an angle for controlling the movement / activity of a corresponding part of the robot, such as one or more movements of the waist, arms, fingers, neck and other robot parts.

[0094] Optionally, the at least one control signal can be generated sequentially one by one, or simultaneously. The robot control algorithm is one or more preset algorithms, including but not limited to coordinate system conversion, matrix transformation, coordinate system alignment, Euler angle conversion and extraction and other processing.

[0095] The VR headset pose is used to generate the robot neck control signal. The host computer can use this VR headset pose when generating the waist, robot arm and hand control signals.

[0096] It should be understood that the above control signal may be a control instruction or a control message, etc., and the embodiment of the present invention does not limit this.

[0097] Step S106: The host computer sends at least one control signal to the robot. The at least one control signal may be sent to the robot once it is generated, or multiple control signals may be sent to the robot at the same time.

[0098] Step S107: The robot is used to receive at least one control signal sent by the host computer, and control a designated part of the robot to perform corresponding movement according to the at least one control signal.

[0099] For example, the robot receives a first control signal sent from a host computer, and the first control signal is used to control the robot's waist movement, including the robot's waist control angle. After the robot analyzes the first control signal, it controls the robot's waist motor movement according to the analyzed waist control angle, so as to achieve the effect that the robot's waist movement angle reaches the waist control angle. For example, the robot's waist is controlled to rotate 45 degrees.

[0100] Step S108: The robot is also used to collect images of the corresponding movements and status information of the designated parts of the robot in real time through the binocular camera.

[0101] For example, after the robot's waist, robotic arm, fingers or neck are moved to a certain angle through at least one control signal, the current status information of each designated part of the robot is reported to the VR headset.

[0102] Step S109: The robot feeds back the image and status information to the VR headset.

[0103] Step S110: The VR headset is also used to receive images and status information fed back from the robot, and update the Unity scene screen in real time according to the images and status information.

[0104] In the Unity scene, the VR headset uses the binocular camera image to update the Unity scene in real time, and uses the robot joint status to update the robot model action in real time, so that the simulation scene displayed in the VR headset is consistent with the posture scene of the external robot movement.

[0105] The robot teleoperation system provided in the embodiment of the present invention collects real-time posture data of designated parts of the wearer's body through a motion capture device equipped with an IMU and a VR head display. After receiving these posture data, the host computer converts the motion capture signal into at least one control signal of the robot's waist, arms, hands and neck through the robot control algorithm, and sends it to the robot to control the relevant parts. The beneficial effects of this solution include the following:

[0106] 1. Compared with the operating handle, the use of motion capture equipment to collect the operator's arm movements can not only collect the wrist posture, but also the shoulder, upper arm, and elbow posture. The extra data can be used to control the redundant degrees of freedom of the robotic arm after kinematic conversion, effectively solving control problems such as the robotic arm "flying" in strange positions and abnormal configurations.

[0107] Second, compared with the operating arm, the operator's arm movement collected by the motion capture device is consistent with the actual movement of the robotic arm after conversion by the control algorithm, which effectively solves the problem of insufficient control accuracy when the input signal size is smaller than the output signal size.

[0108] 3. Use a VR headset to collect the operator's neck movements, and then convert them into neck motor control signals through a control algorithm, so that the operator can observe the robot's surroundings through the VR headset, overcoming the limitations of a fixed camera perspective.

[0109] In a specific implementation of the embodiment of the present invention, Figure 4 As shown, if the posture of one or more body parts collected by the motion capture device includes: the relative posture of the waist, left / right shoulder, left / right upper arm, left / right elbow and left / right wrist, then the above step S105 specifically includes:

[0110] Step S105-1: The host computer performs matrix transformation, coordinate system alignment, Euler angle conversion and extraction processing on the filtered waist relative posture according to the first control algorithm to generate a first control signal including the robot waist control angle.

[0111] Step S105-2: The host computer optimizes the relative posture of the robot's waist control angle, left / right shoulders, left / right upper arms, left / right elbows and left / right wrists according to the second control algorithm, and calculates a second control signal for the robot's mechanical arm to move in at least one degree of freedom.

[0112] The above step S106 includes:

[0113] Step S106-1: The host computer sends the first control signal and the second control signal to the robot.

[0114] The first control signal and the second control signal can be sent separately or simultaneously. After receiving the first control signal, the robot analyzes the first control signal and controls the waist of the robot; after receiving the second control signal, the robot analyzes the second control signal and controls the robot's mechanical arm.

[0115] Furthermore, the above step S105-1: the host computer performs matrix transformation, coordinate system alignment, Euler angle conversion and extraction processing on the filtered waist relative posture according to the first control algorithm to generate a first control signal containing the robot waist control angle, specifically including:

[0116] The host computer converts the filtered waist relative posture into the first homogeneous coordinate transformation matrix; the first homogeneous coordinate transformation matrix is ​​represented by T waist Then, the first homogeneous coordinate transformation matrix T waist Multiply it by a homogeneous coordinate transformation matrix T0 containing rotation and translation to make the world coordinate system of the waist relative to the posture consistent with the first origin and orientation parameters of the robot world coordinate system Alignment; the formula is:

[0117] In robotics, the world coordinate system is a fixed reference coordinate system used to describe the absolute position and orientation of a robot in space. The robot world coordinate system refers to the world coordinate system associated with the robot, which can be predefined by the robot manufacturer or set by the user according to actual needs. The relative posture of the waist refers to the position and orientation of the robot waist relative to the world coordinate system. The posture consists of two parts: position and attitude. The position is expressed by coordinates, and the attitude is expressed by rotation matrices, Euler angles or quaternions.

[0118] The first origin refers to the origin of the coordinate system, that is, the position of the coordinate (0,0,0). During the alignment process, it is necessary to ensure that the first origins of the two coordinate systems are consistent in physical space or logic. Orientation parameter: used to describe the direction or posture of the coordinate system, usually expressed by rotation matrix, Euler angle or quaternion. During the alignment process, it is necessary to ensure that the orientation parameters of the two coordinate systems are also consistent.

[0119] In this embodiment, the alignment process can be achieved by multiplying a homogeneous coordinate transformation matrix T0 including rotation and translation, so that the posture of the robot waist relative to the reference coordinate system is aligned with the first origin and orientation parameters of the robot world coordinate system.

[0120] The host computer will align the first origin and orientation parameters Converted to the first Euler angle, the first Euler angle includes multiple angles of multiple degrees of freedom; for example, the first Euler angle is expressed as (θ waist_roll ,θ waist_pitch ,θ waist_yaw ). They correspond to the roll angle roll, pitch angle pitch and yaw angle yaw respectively.

[0121] From the multiple degrees of freedom of the first Euler angle (θ waist_roll ,θ waist_pitch ,θ waist_yaw ) to extract the target angle corresponding to the target degree of freedom, and generate a first control signal containing the target angle. In the embodiment of the present invention, the waist of the robot can only move in two degrees of freedom, that is, it can only move in the pitch angle pitch and the yaw angle yaw, so the extracted target angle is (θ waist_pitch ,θ waist_yaw ), excluding the roll angle roll. The host computer generates a target angle (θ waist_pitch ,θ waist_yaw ) to obtain the first control signal. Then, step S106 may be executed, and the host computer sends the first control signal to the robot.

[0122] In the embodiment of the present invention, the host computer processes the filtered relative waist posture according to the first control algorithm based on the relative waist posture collected and reported by the IMU and after being filtered by the left side of the VR headset, to generate a first control signal for controlling the waist of the robot, thereby realizing control of the waist position of the robot.

[0123] In addition, the host computer also generates a second control signal, a third control signal and a fourth control signal based on the relative postures of other parts collected and reported by the IMU, which are used to control the movement of the robot's robotic arm, neck and dexterous hand (including each finger) of the robot respectively.

[0124] See also Figure 5 , is a schematic diagram of another process of robot teleoperation provided by an embodiment of the present invention, the method comprising:

[0125] Step S501: The VR headset filters the posture reported by the automatic capture device to obtain the filtered relative postures of the waist, left / right shoulder, left / right upper arm, left / right elbow, left / right wrist, and each finger of the hand, as well as the filtered VR headset posture.

[0126] Among them, the symbol " / " represents "and / or", for example, left / right shoulder is understood as left and / or right shoulder. For the sake of convenience of explanation, in the embodiment of the present invention, " / " can be understood as "and".

[0127] The above-mentioned postures can form a resource pool of posture parameters, and when it is necessary to generate control signals for different parts, one or more corresponding posture parameters are obtained from the resource pool. For example, in the process of generating the first control signal, step S502 is performed.

[0128] Step S502: The host computer obtains the filtered waist relative posture from the resource pool of S501, and performs a series of processing operations such as matrix transformation, coordinate system alignment, Euler angle conversion and target angle extraction on the waist relative posture to obtain the first control signal. The specific process is described in the above steps S105-1 and S105-2, which will not be repeated here in the embodiment of the present invention.

[0129] The process of the host computer generating the second control signal in the embodiment of the present invention is described in detail below.

[0130] See also Figure 5 , the method further comprises:

[0131] Step S503: The host computer optimizes the relative posture of the robot's waist control angle, left / right shoulder, left / right upper arm, left / right elbow and left / right wrist according to the second control algorithm, and calculates the second control signal for the robot's mechanical arm to move in at least one degree of freedom.

[0132] Furthermore, the waist control angle parameter in the first control signal is required when generating the second control signal. In addition, it is also necessary to obtain the filtered relative posture of the left / right shoulder, left / right upper arm, left / right elbow and left / right wrist from the resource pool of the above-mentioned posture parameters.

[0133] Specifically, the upper computer obtains the posture of the left / right wrist relative to the robot coordinate system based on the robot kinematics algorithm and coordinate transformation calculation according to the robot's waist control angle (contained in the first control signal), the relative posture of the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist.

[0134] Furthermore, first, the host computer forms two kinematic chains from the waist to the left / right wrist by combining the relative postures of the left / right shoulder, left / right upper arm, left / right elbow, and left / right wrist. Then, the posture of the left / right wrist relative to the waist coordinate system of the motion capture signal is calculated according to the robot kinematic algorithm, that is, the posture of the left / right wrist relative to the robot coordinate system. It can be expressed as in, represents the position of the left wrist relative to the robot coordinate system, Represents the position of the right wrist relative to the robot coordinate system.

[0135] Then, the host computer performs kinematic processing on the relative positions of the left / right upper arm and the left / right elbow to obtain the rotation angle of at least one DOF manipulator redundant joint; in the embodiment of the present invention, the robot manipulator may include 7 DOF activity dimensions, so the 7 DOF manipulator redundant joint rotation angle is obtained. The rotation angle is represented by the symbol θ redundant .

[0136] Finally, the left / right wrist position relative to the robot coordinate system and the rotation angle θ of at least one redundant joint of the manipulator redundant As two optimization targets, the torque of each joint of the robot arm is taken as the optimization object, and the physical performance of each joint motor of the robot arm is taken as the constraint condition, a multi-objective optimization function is constructed, and the multi-objective optimization function is solved to obtain the second control signal of the robot arm moving in at least one degree of freedom. In the embodiment of the present invention, in each control cycle, the optimization problem is solved to obtain the 7-degree-of-freedom motor torque control instruction or control signal of the robot robot arm, and the 7-degree-of-freedom motor torque is represented by the symbol (τ arm1 ,τ arm2 ,τ arm3 ,τ arm4 ,τ arm5 ,τ arm6 ,τ arm7 ).

[0137] It should be noted that the joints of the robot arm include active joints and passive joints, and the number of active joints and passive joints can be one or more. The active joint movement can drive the passive joint. In this embodiment, the rotation angle θ of the redundant joint is redundant Mainly refers to active joints.

[0138] In the embodiment of the present invention, the second control signal generated by the host computer includes the robot arm 7 degrees of freedom (τ arm1 ,τ arm2 ,τ arm3 ,τ arm4 ,τ arm5 ,τ arm6 ,τ arm7) motor torque control instruction. Then the second control signal is sent to the robot, and the second control signal is used to control the robot arm to move in 7 degrees of freedom.

[0139] Furthermore, the host computer obtains the posture of the left / right wrist relative to the robot coordinate system based on the robot kinematics algorithm and coordinate transformation calculation according to the robot waist control angle (carried by the first control signal), the relative posture of the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist, specifically including:

[0140] The host computer combines the relative postures of the left / right shoulder, left / right upper arm, left / right elbow and left / right wrist into two motion chains from the waist to the left / right wrist, and calculates the posture of the left / right wrist on the two motion chains relative to the waist coordinate system based on the robot forward kinematics algorithm.

[0141] The positions of the left / right wrists on the two kinematic chains relative to the waist coordinate system are converted into the second homogeneous coordinate transformation matrix, which can be expressed as (T l_ee ,T r_ee ); The second homogeneous coordinate transformation matrix (T l_ee ,T r_ee ) is multiplied by a homogeneous coordinate transformation matrix T0 containing rotation and translation to align the world coordinate system of the motion capture signal with the second origin and orientation data of the robot waist coordinate system. The second origin and orientation data are expressed as in The specific correspondence process between the second origin and the orientation data may refer to the alignment process between the first origin and the orientation data in the aforementioned embodiment, which will not be described in detail here.

[0142] The robot waist control angle, such as the target angle of 2 degrees of freedom in the first control signal (θ waist_pitch ,θ waist_yaw ) to perform inverse operation and obtain the corresponding third homogeneous coordinate transformation matrix The second origin and orientation data after the above alignment Multiply left by the third homogeneous coordinate transformation matrix Get the position of the left / right wrist relative to the robot coordinate system in

[0143] See also Figure 5 The method further includes: step S504: the host computer combines the filtered VR head display posture, performs posture conversion, homogeneous coordinate transformation, Euler angle extraction and other processing on the VR head display posture and one or more postures in the resource pool to generate a third control signal for controlling the movement of the robot neck.

[0144] Specifically, a method for generating a third control signal includes:

[0145] After the above step S502: the host computer generates the first control signal including the robot waist control angle, the following further includes:

[0146] The host computer performs filtering processing on the VR head display posture to obtain the filtered VR head display posture. Specifically, the filtering processing on the VR head display posture includes: firstly performing a second-order low-pass filtering on the VR head display posture to improve the smoothness of the input signal; and then performing a sliding window mean filtering on the result to suppress the low-frequency jitter in the input signal.

[0147] The filtered VR head display pose is converted into the fourth homogeneous coordinate transformation matrix (T head ), and multiplied on the left by the inverse of the homogeneous coordinate transformation matrix converted by the first control signal Get the position of the neck relative to the waist The formula is expressed as Then, adjust the position of the neck relative to the waist. The second Euler angle is converted into a corresponding second Euler angle, where the second Euler angle includes at least one angle of at least one degree of freedom.

[0148] The target angle corresponding to the target degree of freedom is extracted from at least one angle of at least one degree of freedom of the second Euler angle. For example, the target angle of 2 degrees of freedom is selected from the 3 angles of 3 degrees of freedom of the second Euler angle, which can be expressed as (θ head_pitch ,θ head_yaw ), and generate a head_pitch ,θ head_yaw )'s third control signal.

[0149] The host computer sends a third control signal to the robot, and the third control signal is used to control the movement of the neck part of the robot.

[0150] In addition, Figure 5 The method flow shown also includes: the host computer obtains the relative postures of the left / right wrist and each finger of the hand from the resource pool. Then, step S505 is executed.

[0151] Step S505: The host computer performs kinematics algorithm, Euler angle conversion and extraction processing on the relative postures of the filtered left / right wrists and each finger of the hand to generate a fourth control signal for controlling the robot hand.

[0152] The host computer sends a fourth control signal to the robot, and the fourth control signal is used to control the movement of the hand part of the robot.

[0153] Furthermore, the index finger, middle finger, ring finger, and little finger of the robot's dexterous hand each have one degree of freedom. Through the kinematic algorithm, the position (T index ,T middle ,T ring ,T pinky ) is converted into Euler angles, and the rotation angles in the corresponding directions are extracted and mapped into control instructions for the motors of the index finger, middle finger, ring finger, and little finger of the dexterous hand (P index ,P middle ,P ring ,P pinky ). The thumb of the robot's dexterous hand has two degrees of freedom. Through the kinematic algorithm, the position T of the left / right thumb relative to the left / right wrist in the motion capture signal is converted into thumb Convert to spherical polar coordinates Finally, extract the coordinate value corresponding to the spherical polar coordinate system Mapped to the control instructions of the thumb motor of the dexterous hand (P index ,P middle ).

[0154] Finally, the control instructions (P index ,P middle ) and control instructions for the motors of the index finger, middle finger, ring finger, and little finger (P index ,P middle ,P ring ,P pinky ) is sent to the robot to control the robot's dexterous hand to perform corresponding activity operations.

[0155] In addition, in the embodiment of the present invention, in the above steps S103 and S104, the VR head display is also used to perform kinematic transformation on the position and posture of one or more body parts after filtering, and to align the motion capture signal with the dummy model in the Unity scene with the origin and orientation of the coordinate system, and to update the dummy model action in the Unity scene. Thus, the operator's arm movement collected by the motion capture device is consistent with the actual movement size of the robotic arm after conversion by the control algorithm, effectively solving the problem of insufficient control accuracy when the input signal size is smaller than the output signal size.

[0156] The method provided in the embodiment of the present invention collects the real-time posture data of the operator's waist, back, neck, left / right shoulder, left / right upper arm, left / right elbow, left / right wrist, and left / right hand fingers through a motion capture device equipped with an IMU and a VR head display, and converts the motion capture signal into a motion control signal of the robot's waist, arms, hands, and neck through a control algorithm. Compared with the operating handle, this solution uses a motion capture device to collect the operator's arm movements. In addition to being able to collect the wrist posture, it can also collect the posture of the shoulder, upper arm, and elbow. The extra data can be used to control the redundant degrees of freedom of the robot arm after kinematic conversion, effectively solving the control problems such as the robot arm "flying" in a singular position and abnormal configuration.

[0157] In addition, the VR headset is used to collect the operator's neck movement, which is converted into neck motor control signals through the control algorithm, so that the operator can observe the robot's surrounding environment through the VR headset, overcoming the limitations of the fixed camera perspective. This method is effective and the process is simple and clear.

[0158] It should be noted that the first control signal to the fourth control signal generated in the above embodiment can control the robot's lower limbs, legs, chassis and other parts in addition to the waist, neck, arms and hands, and this embodiment does not limit this.

[0159] In addition, it should be understood that the second control signal generated above can be used to control the left and right arms of the robot arm, or it can be used to control one of the left and right arms, or one hand, and this embodiment does not limit this.

[0160] In another embodiment of the present invention, a robot teleoperation method based on a VR head display is also provided, which can be applied to the host computer of the above embodiment, such as Figure 6 As shown, the method includes:

[0161] Step S601: receiving the posture sent by the VR headset and the VR headset posture, wherein the posture sent by the VR headset is output after filtering the posture of one or more body parts of the wearer collected by the motion capture device by the VR headset. The VR headset posture is the posture of the headset device itself, which is generated and sent by the VR headset itself.

[0162] Step S602: Process the posture sent by the VR headset and the posture of the VR headset according to the robot control algorithm to generate at least one control signal.

[0163] Step S603: Send the at least one control signal to the robot, wherein the at least one control signal is used to control the movement of at least one of the waist, arms, hands and neck of the robot.

[0164] The specific process can be found in the description of steps S105 and S106 of the aforementioned embodiment, which will not be described in detail here.

[0165] In addition, the host computer is also used to execute the above Figure 4 or Figure 5 The method flow shown generates first to fourth control signals and sends them to the robot to control the movement of at least one of the robot's waist, robotic arm, dexterous hand and neck.

[0166] The robot teleoperation method provided by the embodiment of the present invention collects the real-time posture data of the operator's waist, arms and hands through the motion capture device equipped with IMU and transmits it to the VR head display. The Unity scene is run in the head display, and the motion capture signal is processed by kinematics to update the dummy action in the scene; the VR head display then sends the head display posture and the motion capture signal to the host computer. The host computer converts the motion capture signal and the head display posture into the motion control signal of the robot's waist, arms, hands and neck through the control algorithm; when the robot executes the corresponding motion control signal, the real-time binocular camera image and status are fed back to the VR head display, and the robot model action is updated in Unity, so that the operator can obtain the robot's perspective image in the VR head display while also seeing the real-time motion capture signal and robot status.

[0167] In the embodiment of the present invention, a robot teleoperation device is also provided, which is used to implement the above-mentioned embodiment and preferred implementation mode, and the descriptions have been made no further. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0168] The embodiment of the present invention provides a robot teleoperation device, such as Figure 7 As shown, the device includes: a receiving module 710, a generating module 720 and a sending module 730. In addition, the device may also include other more or fewer modules, such as a storage module.

[0169] Among them, the receiving module 710 is used to receive the posture sent by the VR headset and the VR headset posture, wherein the posture is output by the VR headset after filtering the posture of one or more body parts of the wearer collected by the motion capture device.

[0170] The generation module 720 is used to process the posture and the VR head display posture according to the robot control algorithm to generate at least one control signal.

[0171] The sending module 730 is used to send the at least one control signal to the robot, and the at least one control signal is used to control the movement of at least one part of the robot's waist, arms, hands and neck.

[0172] In addition, further functional descriptions of the above modules are the same as those of the above corresponding embodiments and will not be repeated here.

[0173] The robot teleoperation device in the embodiment of the present invention is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0174] The embodiment of the present invention also provides an electronic device having the above Figure 7 The robotic teleoperation device shown.

[0175] See also Figure 8 , is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, which can be used to implement part or all of the functions of the aforementioned host computer.

[0176] like Figure 8 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface).

[0177] In some optional embodiments, if desired, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0178] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0179] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the remote operation method shown in the above embodiment.

[0180] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0182] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0183] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0184] In addition, the electronic device also includes at least one communication interface for the electronic device to communicate with other devices or communication networks, such as VR headsets, robots and other devices.

[0185] An embodiment of the present invention also provides a computer-readable storage medium, and the above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented by downloading through a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware.

[0186] The storage medium may be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium may also include a combination of the above-mentioned types of memories. It is understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the robot teleoperation method based on a VR head display shown in the above embodiment is implemented.

[0187] The present invention may be implemented in many ways, including as: a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on a memory coupled to the processor and / or provided by the memory. In this specification, these embodiments or any other form that the present invention may take may be referred to as techniques. In general, the order of steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, a component described as being configured to perform a task (such as a processor or memory) may be implemented as a general component that is temporarily configured to perform a task at a given time, or as a specific component manufactured to perform a task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data (such as computer program instructions).

[0188] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A robot teleoperation system based on VR head display, characterized in that: The system comprises: a motion capture device, a VR head display, a host computer and a robot, wherein the motion capture device is a wearable device and is equipped with at least one inertial measurement unit IMU; The motion capture device is used to collect the posture of one or more body parts of the wearer through the at least one IMU, and send a motion capture signal carrying the posture to the VR head display; The VR head display is used to filter the posture in the motion capture signal, and send the filtered posture of one or more body parts and the VR head display posture to the host computer; The host computer is used to receive the filtered posture of one or more body parts and the posture of the VR head display sent by the VR head display, process the posture and the posture of the VR head display according to the robot control algorithm, generate at least one control signal, and send the at least one control signal to the robot, wherein the at least one control signal is used to control the movement of at least one part of the waist, arms, hands and neck of the robot; The robot is used to receive at least one control signal sent by the host computer, and control a designated part of the robot to perform corresponding movement according to the at least one control signal.

2. The system according to claim 1, characterized in that The postures of one or more body parts collected by the motion capture device include: the relative postures of the waist, left / right shoulders, left / right upper arms, left / right elbows, and left / right wrists; The host computer processes the posture and the VR head display posture according to the robot control algorithm, generates at least one control signal, and sends the at least one control signal to the robot, including: The host computer performs matrix transformation, coordinate system alignment, Euler angle conversion and extraction processing on the filtered waist relative posture according to the first control algorithm to generate a first control signal including the robot waist control angle; The host computer optimizes the relative positions of the robot's waist control angle, left / right shoulder, left / right upper arm, left / right elbow and left / right wrist according to the second control algorithm, and calculates a second control signal for the robot's mechanical arm to move in at least one degree of freedom; The host computer sends the first control signal and the second control signal to the robot.

3. The system according to claim 2, characterized in that The host computer performs matrix transformation, coordinate system alignment, Euler angle conversion and extraction processing on the filtered waist relative posture according to the first control algorithm to generate a first control signal containing the robot waist control angle, including: Converting the filtered waist relative posture into a first homogeneous coordinate transformation matrix; Multiplying the first homogeneous coordinate transformation matrix by a homogeneous coordinate transformation matrix including rotation and translation so as to align the world coordinate system of the waist relative position with the first origin and orientation parameters of the robot world coordinate system; Converting the aligned first origin and the orientation parameter into a first Euler angle, wherein the first Euler angle includes a plurality of angles with a plurality of degrees of freedom; A target angle corresponding to a target degree of freedom is extracted from a plurality of angles of a plurality of degrees of freedom of the first Euler angle, and a first control signal including the target angle is generated.

4. The system according to claim 2, characterized in that The host computer optimizes the relative positions of the robot's waist control angle, left / right shoulder, left / right upper arm, left / right elbow and left / right wrist according to the second control algorithm, and calculates a second control signal for the robot's mechanical arm to move in at least one degree of freedom, including: According to the relative postures of the robot waist control angle, the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist, based on the robot kinematics algorithm and coordinate transformation calculation, the posture of the left / right wrist relative to the robot coordinate system is obtained; Performing kinematic processing on the relative posture of the left / right upper arm and the left / right elbow to obtain the rotation angle of the redundant joint of the manipulator with at least one degree of freedom; The posture of the left / right wrist relative to the robot coordinate system and the rotation angle of the redundant joint of the at least one degree of freedom robotic arm are taken as optimization targets, the torque of each joint of the robotic arm is taken as optimization object, and the physical performance of the motor of each joint of the robotic arm is taken as constraint conditions, a multi-objective optimization function is constructed, and the multi-objective optimization function is solved to obtain the second control signal for the robotic arm to move in at least one degree of freedom.

5. The system according to claim 4, characterized in that The method comprises: obtaining the posture of the left / right wrist relative to the robot coordinate system based on the robot kinematics algorithm and coordinate transformation calculation according to the relative posture of the robot waist control angle, the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist, including: The relative postures of the left / right shoulder, the left / right upper arm, the left / right elbow and the left / right wrist form two kinematic chains from the waist to the left / right wrist, and the postures of the left / right wrist on the two kinematic chains relative to the waist coordinate system are calculated based on the robot forward kinematics algorithm, and the postures are converted into a second homogeneous coordinate transformation matrix; Multiplying the second homogeneous coordinate transformation matrix by a homogeneous coordinate transformation matrix including rotation and translation, so as to align the world coordinate system of the motion capture signal with the second origin and orientation data of the robot waist coordinate system; Perform an inverse operation on the robot waist control angle to obtain a corresponding third homogeneous coordinate transformation matrix; The aligned second origin and the orientation data are multiplied by the third homogeneous coordinate transformation matrix to obtain the position and posture of the left / right wrist relative to the robot coordinate system.

6. The system according to claim 2, characterized in that After the host computer generates the first control signal including the robot waist control angle, the method further includes: Filtering the VR head display posture to obtain a filtered VR head display posture; Convert the filtered VR head display posture into a fourth homogeneous coordinate transformation matrix, and multiply it by the inverse of the homogeneous coordinate transformation matrix converted by the first control signal to obtain the posture of the neck relative to the waist; Converting the posture of the neck relative to the waist into a corresponding second Euler angle, wherein the second Euler angle includes at least one angle of at least one degree of freedom; extracting a target angle corresponding to a target degree of freedom from at least one angle of at least one degree of freedom of the second Euler angle, and generating a third control signal including the target angle; The third control signal is sent to the robot, and the third control signal is used to control the movement of the neck part of the robot.

7. The system according to claim 1, characterized in that The postures of one or more body parts collected by the motion capture device include: the relative postures of the left / right wrist and each finger of the hand; The host computer processes the posture and the VR head display posture according to the robot control algorithm, generates at least one control signal, and sends the at least one control signal to the robot, including: The host computer performs kinematics algorithm, Euler angle conversion and extraction processing on the relative postures of the left / right wrist and each finger of the hand after filtering, and generates a fourth control signal containing each finger of the robot hand in at least one degree of freedom; The host computer is also used to send the fourth control signal to the robot, and the fourth control signal is used to control the movement of the hand part of the robot.

8. The system according to any one of claims 1 to 7, characterized in that: The head of the robot is equipped with a binocular camera; The robot is also used to collect images of the corresponding movements and status information of the designated parts of the robot in real time through the binocular camera, and feed the images and status information back to the VR head display; The VR head display is also used to receive the image and the status information fed back from the robot, and update the Unity scene screen in real time according to the image and the status information.

9. The system according to any one of claims 1 to 7, characterized in that: The VR head display is also used to perform kinematic transformation on the position and posture of one or more filtered body parts, align the motion capture signal with the coordinate system origin and orientation of the dummy model in the Unity scene, and update the dummy model movement in the Unity scene.

10. A robot teleoperation method based on VR head display, characterized in that: Applied to a host computer, the method includes: Receiving a posture sent by a VR head display and a posture of the VR head display, wherein the posture sent by the VR head display is output after filtering the posture of one or more body parts of the wearer collected by a motion capture device by the VR head display; Processing the posture sent by the VR head display and the posture of the VR head display according to a robot control algorithm to generate at least one control signal; The at least one control signal is sent to the robot, and the at least one control signal is used to control the movement of at least one part of the waist, arms, hands and neck of the robot.

11. An electronic device, characterized in that: comprising a memory and a processor, wherein the memory and the processor are connected; The memory stores computer instructions, and the processor executes the robot teleoperation method based on VR head display as described in claim 10 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer readable storage medium has computer instructions stored thereon; The computer instructions are used to enable a computer to execute the robot teleoperation method based on a VR head display as described in claim 10.

Citation Information

Patent Citations

  • Teleoperation man-machine interaction terminal and method for flexible mechanical arm

    CN110834330A

  • Redundant mechanical arm teleoperation combined interaction device and redundant mechanical arm teleoperation system

    CN112828916A

  • Human body posture determination method and device, equipment and storage medium

    CN115560750A

  • Method for remotely operating humanoid robot by identifying hand postures through virtual reality glasses

    CN117746494A

Cited By

  • Teleoperation method and device of robot, terminal equipment and storage medium

    CN119347781A

  • Robot remote control system and method, storage medium and computer program product

    CN119427365A

  • Remote operation system and control method for two mechanical arms based on VR head-mounted display

    CN120326596A

  • Robot teleoperation control method and system, medium, equipment and program product

    CN120395842A

  • Robotic teleoperation control method and system, medium, device and program product

    CN120395842B