Humanoid robot remote control system and method based on virtual reality technology

Abstract

The present application relates to a humanoid robot remote control system and method based on virtual reality technology. The system includes a VR device and a humanoid robot. The VR device includes a head-mounted device, a hand device, a visual information sampling unit, a hand device sampling unit, and a visual information imaging unit. The humanoid robot includes a robot system communication unit, a robot system planning control unit, a visual sensor, and a joint motor execution unit. The present application solves the technical problem that the existing robot control mainly relies on workers' on-site teaching operations, which makes the control threshold of the robot arm high and the site limitations large. The present application has the advantages of low threshold, strong interactivity, low latency, high security, and high flexibility.

Description

Technical Field

[0001] The present application relates to the field of robots, and in particular, to a remote control system and method for a humanoid robot based on virtual reality technology. Background Art

[0002] In recent years, with the continuous development of robotics technology, robots have been more and more widely used in production and manufacturing, medical surgery, education and other fields. For the traditional dual-arm robot control system, professional operators are required to have a certain technical threshold, so it is difficult for ordinary people to operate the robot. The remote control dual-arm robot system and method based on virtual reality technology eliminates this problem. Using virtual reality technology, the operator can control the robot to follow the user's hand movements through the hand actuator without having relevant technical knowledge of robotic arm control. This greatly reduces the control threshold of the robot. More people can easily control the robot, which means expanding the scope of robot application. At the same time, through virtual reality technology, the operator can feel the real operation experience while remotely controlling the robot, which further improves the control accuracy of the robot.

[0003] In the field of virtual reality (VR) technology, current VR teleoperation systems face severe real-time and stability challenges. Communication delays and sensor data processing speeds limit the system's real-time response performance, hampering users' immediate control of the virtual environment. At the same time, the system's stability is threatened by many aspects, including system crashes, anomalies, and data synchronization issues, which significantly weaken the stability of the user experience.

[0004] In terms of communication delay, the existing system has failed to effectively overcome the lag in data transmission and processing time, resulting in a significant impact on the real-time performance of user operations in the virtual environment. The insufficient processing speed of sensor data makes it difficult for the system to quickly and accurately capture and respond to user actions, further reducing the real-time and accuracy of operations. In addition, the stability of the system is also a difficult problem that needs to be solved urgently. The frequent occurrence of system crashes and abnormal situations reduces the reliability of the entire system and brings unnecessary troubles to users. Data synchronization problems cause screen jitter and unsmoothness, affecting the user's perception in the virtual environment. Summary of the invention

[0005] In order to overcome at least one deficiency in the prior art, the present application provides a humanoid robot remote control system and method based on virtual reality technology.

[0006] In a first aspect, a humanoid robot remote control system based on virtual reality technology is provided, comprising: a VR device and a humanoid robot, wherein the VR device comprises a head-mounted device, a hand device, a visual information sampling unit, a hand device sampling unit and a visual information imaging unit; the humanoid robot comprises a robot system communication unit, a robot system planning control unit, a visual sensor, and a joint motor execution unit;

[0007] The visual information sampling unit collects environmental information around the user, and the visual sensor collects environmental information around the humanoid robot and sends it to the visual information sampling unit through the robot system communication unit; the visual information sampling unit sends the acquired environmental information to the visual information imaging unit for imaging processing to obtain images around the user and the humanoid robot; the images around the user and the humanoid robot are transmitted to the head-mounted device and displayed in the visual interface;

[0008] The visual interface of the head-mounted device also has a control interface, and the user drags the control interface through the hand device to switch the control mode of the humanoid robot;

[0009] If the control mode of the humanoid robot is the arm following mode, when the hand device moves, the hand device sampling unit obtains the posture information of the hand device, and sends the posture information to the robot system planning control unit through the robot system communication unit; the robot system planning control unit performs position planning and posture planning of the humanoid robot arm according to the received posture information, obtains the position planning results and posture planning results, and sends the position planning results and posture planning results to the joint motor execution unit to control the arm movement of the humanoid robot.

[0010] In one embodiment, the hand device sampling unit obtains the posture information of the hand device, and sends the posture information to the robot system planning control unit through the robot system communication unit, including:

[0011] After the hand device sampling unit obtains the posture information of the hand device, the posture information is stored in a double-ended FIFO queue, and the posture information is filtered using a sliding mean filter, a Kalman filter, and a Butterworth low-pass filter to obtain the filtered posture information, which is the absolute coordinate in the virtual coordinate system;

[0012] The absolute coordinates in the virtual coordinate system are converted into the absolute coordinates in the actual coordinate system of the robot through the homogeneous transformation matrix, and then the absolute coordinates in the actual coordinate system of the robot are converted into the relative coordinates in the actual coordinate system of the robot through differential operation, which are recorded as the converted posture information;

[0013] The double-ended FIFO queue sends the converted posture information to the humanoid robot at a fixed period; the VR device does not access the position return frame of the humanoid robot and only performs the sending operation.

[0014] In one embodiment, when an abnormality occurs in a humanoid robot, the abnormality information is sent to the VR device, and the abnormal state of the humanoid robot is displayed in the head mounted device; a robot stop signal is sent and the communication between the VR device and the humanoid robot is interrupted.

[0015] In one embodiment, the robot system planning control unit performs position planning and posture planning of the humanoid robot arm according to the received position and posture information, and obtains position planning results and posture planning results, including:

[0016] The posture information includes displacement information and posture information;

[0017] The displacement information is used as a displacement vector, and the posture rotation angle is determined according to the posture information. The magnitudes of the posture rotation angle and the displacement vector are compared. If the posture rotation angle is greater than the magnitude of the displacement vector, the posture planning is determined as the main axis planning, and the position planning is determined as the secondary axis planning. If the posture rotation angle is less than the magnitude of the displacement vector, the position planning is determined as the main axis planning, and the posture planning is determined as the secondary axis planning.

[0018] If the main axis planning is position planning and the secondary axis planning is attitude planning, then:

[0019] The spindle planning is as follows: the displacement information is used as the displacement vector, and the quintic spline function is used to perform speed planning based on the displacement vector to obtain the spindle planning speed; the quintic spline function is used to introduce a direction constraint on the spindle planning speed to obtain the spindle planning trajectory;

[0020] The secondary axis planning is: Determine the secondary axis planning ratio :

[0021]

[0022] Set the main axis speed to the ratio of the secondary axis speed Scale down in proportion to obtain the secondary axis planning speed;

[0023] If the main axis planning is posture planning and the secondary axis planning is position planning, then:

[0024] The main axis planning is: using the quintic spline function, the speed planning is performed based on the posture rotation angle to obtain the main axis planning speed;

[0025] The secondary axis planning is: Determine the secondary axis planning ratio :

[0026]

[0027] Set the main axis speed to the ratio of the secondary axis speed The secondary axis planning speed is obtained by scaling down in proportion; the quintic spline function is used to introduce directional constraints on the secondary axis planning speed to obtain the secondary axis planning trajectory.

[0028] In one embodiment, the posture rotation angle is a quaternion rotation angle or an Euler angle.

[0029] In one embodiment, the humanoid robot also includes: a force control sensor, which collects contact torque information of the gripper at the end of the arm of the humanoid robot, and sends the contact torque information to the visual information sampling unit through the robot system communication unit, the visual information sampling unit transmits the contact torque information to the visual information imaging unit to generate a current waveform, and sends the current waveform to the head-mounted device for display in a visual interface; the current waveform enables the user to adjust the gripping force of the gripper at the end of the arm of the humanoid robot.

[0030] In one embodiment, the humanoid robot control mode also includes a chassis movement mode, a head rotation mode and a waist rotation mode; after switching to the corresponding humanoid robot control mode, the humanoid robot is made to perform corresponding movements through the operation buttons on the hand device.

[0031] In one embodiment, the humanoid robot is provided with a mobile base. When the control mode of the humanoid robot is a chassis movement mode, the VR device can control the movement of the mobile base through a hand device.

[0032] In one embodiment, the humanoid robot is provided with a rotatable head, and when the control mode of the humanoid robot is a head rotation mode, the VR device can control the rotation of the head through a hand device.

[0033] In a second aspect, a humanoid robot remote control method based on virtual reality technology is provided, which is applied to the humanoid robot and includes:

[0034] Receiving posture information; the posture information includes displacement information and posture information;

[0035] The displacement information is used as a displacement vector, and the posture rotation angle is determined according to the posture information. The magnitudes of the posture rotation angle and the displacement vector are compared. If the posture rotation angle is greater than the magnitude of the displacement vector, the posture planning is determined as the main axis planning, and the position planning is determined as the secondary axis planning. If the posture rotation angle is less than the magnitude of the displacement vector, the position planning is determined as the main axis planning, and the posture planning is determined as the secondary axis planning.

[0036] If the main axis planning is position planning and the secondary axis planning is attitude planning, then:

[0037] The spindle planning is as follows: the displacement information is used as the displacement vector, and the quintic spline function is used to perform speed planning based on the displacement vector to obtain the spindle planning speed; the quintic spline function is used to introduce a direction constraint on the spindle planning speed to obtain the spindle planning trajectory;

[0038] The secondary axis planning is: Determine the secondary axis planning ratio :

[0039]

[0040] Set the main axis speed to the ratio of the secondary axis speed Scale down in proportion to obtain the secondary axis planning speed;

[0041] If the main axis planning is posture planning and the secondary axis planning is position planning, then:

[0042] The main axis planning is: using the quintic spline function, the speed planning is performed based on the posture rotation angle to obtain the main axis planning speed;

[0043] The secondary axis planning is: Determine the secondary axis planning ratio :

[0044]

[0045] Set the main axis speed to the ratio of the secondary axis speed The secondary axis planning speed is obtained by scaling down in proportion; the quintic spline function is used to introduce directional constraints on the secondary axis planning speed to obtain the secondary axis planning trajectory.

[0046] In one embodiment, the posture rotation angle is a quaternion rotation angle or an Euler angle.

[0047] In a third aspect, a method for remotely controlling a humanoid robot based on virtual reality technology is provided, which is applied to VR equipment, comprising:

[0048] Obtain the posture information of the hand device, store the posture information in a double-ended FIFO queue, filter the posture information using a sliding mean filter, a Kalman filter, and a Butterworth low-pass filter to obtain the filtered posture information, which is the absolute coordinate in the virtual coordinate system;

[0049] The absolute coordinates in the virtual coordinate system are converted into the absolute coordinates in the actual coordinate system of the robot through the homogeneous transformation matrix, and then the absolute coordinates in the actual coordinate system of the robot are converted into the relative coordinates in the actual coordinate system of the robot through differential operation, which are recorded as the converted posture information;

[0050] The double-ended FIFO queue sends the converted posture information to the humanoid robot at a fixed period; the VR device does not access the position return frame of the humanoid robot and only performs the sending operation.

[0051] Compared with the prior art, this application has the following beneficial effects:

[0052] Low threshold: Compared with traditional robot programming and control methods, this application uses hand devices for dynamic tracking control, which does not require complex programming knowledge and experience, lowers the usage threshold, and enables more people to easily use robots for operation.

[0053] Strong interactivity: The robot and the end of the arm are equipped with visual sensors. Users can open the window to observe the actual operating space while using VR. At the same time, the force-controlled gripper at the end of the robotic arm can provide real-time feedback of the current torque current, allowing the operator to achieve real-time visual and force interaction with the environment.

[0054] Low latency: The speed fusion planning algorithm based on time cycle is adopted to make the robot highly sensitive to time cycle when following the hand movements, which reduces the control delay during remote operation to a certain extent.

[0055] Speed ​​smoothing: This application performs multiple filtering processes and speed re-planning algorithms to address situations such as communication packet loss and uneven position data sampling, to ensure smooth position when the joint motor is executed.

[0056] High safety: By limiting the actual hardware activity space of the robot arm and implementing multi-order smoothing of the terminal trajectory based on the multi-order differential constraints of the spline function, the path and speed are smoothed while limiting the actual hardware activity space of the robot arm. This can effectively avoid collisions or out-of-range activities during movement, thus ensuring the safety and stability of the robot.

[0057] High flexibility: The humanoid robot used has dual seven-axis arms with 14 degrees of freedom (including redundant degrees of freedom). The head, neck, and waist are equipped with servo motors for rotation. The base is an AGV cart that supports forward, backward, left, and right movement and rotation. Users can switch the arms, head, waist, neck, and base AGV control in the VR control interface, which has extremely high flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings:

[0059] Figure 1 The structure diagram of the humanoid robot remote control system based on virtual reality technology is shown;

[0060] Figure 2 A schematic diagram of a visualization interface in a head mounted device is shown;

[0061] Figure 3 A schematic diagram of master-slave communication is shown. DETAILED DESCRIPTION

[0062] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.

[0063] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.

[0064] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.

[0065] The present application embodiment provides a humanoid robot remote control system based on virtual reality technology. Figure 1 The structure diagram of the humanoid robot remote control system based on virtual reality technology is shown in Figure 1 The system includes: VR equipment and humanoid robot. The VR equipment includes head-mounted equipment, hand equipment, visual information sampling unit, hand equipment sampling unit and visual information imaging unit; the humanoid robot includes robot system communication unit, robot system planning control unit, visual sensor and joint motor execution unit. The specific implementation functions of each unit are introduced in detail below.

[0066] The visual information sampling unit collects environmental information around the user, and the visual sensor collects environmental information around the humanoid robot and sends it to the visual information sampling unit through the robot system communication unit; the visual information sampling unit sends the acquired environmental information to the visual information imaging unit for imaging processing to obtain images around the user and the humanoid robot; the images around the user and the humanoid robot are transmitted to the head-mounted device and displayed in the visual interface;

[0067] Here, the visual sensor may be a 3D camera mounted on the robot head, used to collect environmental information around the humanoid robot, and the visual information sampling unit may include a visual sensor mounted on a head-mounted device, used to collect environmental information around the user. There is a visualization interface in the head-mounted device, and the visualization interface may include multiple windows, which are respectively used to display images around the user and images around the humanoid robot. When the user observes through the visualization interface that the user and the surrounding environment of the humanoid robot are in a safe state, the movement of the humanoid robot can be controlled through the hand device. Figure 2 A schematic diagram of a visualization interface in a head mounted device is shown.

[0068] A control interface is also provided in the visual interface of the head mounted device. The user can switch the control mode of the humanoid robot by dragging the control interface through the hand device.

[0069] Here, the humanoid robot control modes include arm following mode, chassis moving mode, head rotation mode and waist rotation mode; after switching to the corresponding humanoid robot control mode, the operation buttons on the hand device are used to make the humanoid robot perform corresponding movements. Users can switch the arm, head, waist, neck and base AGV control in the VR control interface, which has extremely high flexibility. The head, neck and waist are equipped with servo motors for rotation.

[0070] Specifically, the humanoid robot has a mobile base, which is an AGV cart. It supports movement and rotation forward, backward, left and right, and can flexibly meet various working scenarios when performing remote robot control operations. In the control mode designed for the head-mounted device, there is a control interface for controlling the movement of the robot chassis. The user can switch to this mode through the hand device and control the robot base to move forward and backward, and rotate left and right.

[0071] Specifically, the humanoid robot has a rotatable head. In a more complex working environment, it is necessary to ensure that the robot is not disturbed by environmental obstacles when operating. There is a control interface for rotating the robot camera in the control mode designed for the head-mounted device. The user can use the hand device to select the robot head control in the control mode on the right side of the field of view to achieve 270° field of view information collection.

[0072] Specifically, the humanoid robot has a mechanical arm with 7 degrees of freedom on each side. When the control mode is switched to the arm following mode, the arms of the humanoid robot will follow the user's hand device for synchronous control.

[0073] If the control mode of the humanoid robot is the arm following mode, when the hand device moves, the hand device sampling unit obtains the position and posture information of the hand device, and sends the position and posture information to the robot system planning control unit through the robot system communication unit; the robot system planning control unit performs position planning and posture planning of the humanoid robot arm according to the received posture information, obtains the position planning results and posture planning results, and sends the position planning results and posture planning results to the joint motor execution unit to control the arm movement of the humanoid robot, thereby achieving the purpose of the humanoid robot following the synchronous movement of the hand device.

[0074] Specifically, the VR device is based on the Oculus Quest 3 platform, uses Unity3D's official VR framework XRInteractive Toolkit, and calls the OpenXR interface and UMP (Universal MediaPlayer) plug-in under the platform characteristics of Oculus to implement RTSP video stream transmission. Considering the user's usage habits when remotely controlling the robot, this embodiment is designed with a perspective mode (Passthrough) so that the center area of ​​the head display field of view is the visual information collected by the robot's head camera, the upper left corner displays the image information of the head-mounted device's camera, and the right side is the control interface for switching the robot's control mode.

[0075] Furthermore, the humanoid robot also includes: a force control sensor, which collects contact torque information of the gripper at the end of the arm of the humanoid robot, and sends the contact torque information to the visual information sampling unit through the robot system communication unit, the visual information sampling unit transmits the contact torque information to the visual information imaging unit to generate a current waveform, and sends the current waveform to the head-mounted device for display in a visual interface; the current waveform enables the user to adjust the gripping force of the gripper at the end of the arm of the humanoid robot.

[0076] In this embodiment, the end of the robot arm is equipped with a force-controlled gripper. The user can identify the current waveform of the current gripper gripping the object in the visual interface of the head-mounted device, thereby avoiding damage to the equipment due to excessive load during manual operations. The user achieves force interaction with the robot during operation based on the current waveform.

[0077] In one embodiment, the classic server model C / S structure is adopted. The VR device sends different commands to the humanoid robot by accessing a specific URL. The humanoid robot uses Flask as the RPC (Remote Procedure Call) framework. Receive and process the control instructions sent by the VR device. Accelerated GStreamer is a set of GStreamer plug-ins provided by NVIDIA for its Jetson platform. It aims to accelerate multimedia processing tasks, improve performance and reduce latency by using GPU (graphics processing unit). Accelerated GStreamer plug-ins accelerate the processing of video data by making full use of the computing power of GPU. This technical means improves the efficiency of video encoding and decoding while optimizing the utilization efficiency of network bandwidth. In terms of video, GStreamer is used as a multimedia framework. Thanks to the ecological software Accelerated GStreamer provided by the robot main control platform Jetson Xavier, a high-efficiency acquisition, encoding and streaming process is achieved. At the same time, the GStreamer pipeline is packaged as an RTSP server using C language to facilitate remote programs to pull video streams. Figure 3 A schematic diagram of master-slave communication is shown.

[0078] Specifically, the hand device sampling unit obtains the posture information of the hand device, and sends the posture information to the robot system planning control unit through the robot system communication unit, including:

[0079] First, after the hand device sampling unit obtains the posture information of the hand device, the posture information is stored in a double-ended FIFO queue, and the posture information is filtered using a sliding mean filter, a Kalman filter, and a Butterworth low-pass filter to obtain the filtered posture information. The filtered posture information is the absolute coordinate in the virtual coordinate system.

[0080] Then, the absolute coordinates in the virtual coordinate system are converted into the absolute coordinates in the actual coordinate system of the robot through the homogeneous transformation matrix, and then the absolute coordinates in the actual coordinate system of the robot are converted into the relative coordinates in the actual coordinate system of the robot through differential operation, which is recorded as the converted posture information;

[0081] Here, the absolute coordinates in the virtual coordinate system are obtained by sampling the virtual coordinate system in Unity3D. This coordinate system and the actual coordinate system of the robot need to be converted. In Unity3D, the virtual coordinate system is defined as a left-handed coordinate system, in which the x direction is the horizontal forward direction of the head display device, the z direction is vertically upward, and the y direction is obtained according to the definition of the left-handed coordinate system. The robot coordinate system used in this embodiment is a right-handed coordinate system, in which the x direction is the horizontal forward direction of the robot body, the z direction is vertically upward, and the y direction is obtained according to the definition of the right-handed coordinate system.

[0082] Then, the double-ended FIFO queue sends the converted posture information to the humanoid robot at a fixed period; the VR device does not access the position return frame of the humanoid robot and only performs the sending operation.

[0083] In the above embodiment, multiple filtering processes are performed to address situations such as communication packet loss and uneven position data sampling, to ensure smooth position of the joint motor during execution.

[0084] In one embodiment, when an abnormality occurs in a humanoid robot, the abnormality information is sent to the VR device, and the abnormal state of the humanoid robot is displayed in the head mounted device; a robot stop signal is sent and the communication between the VR device and the humanoid robot is interrupted.

[0085] In one embodiment, the robot system planning control unit performs position planning and posture planning of the humanoid robot arm according to the received position and posture information, and obtains position planning results and posture planning results, including:

[0086] The posture information includes displacement information and attitude information; here, the posture information received by the robot system planning control unit is the relative coordinates in the actual coordinate system of the robot. The displacement information is a vector, including displacement and direction, and the displacement unit is mm. The attitude information is a scalar under the quaternion rule, including the attitude rotation angle, and the attitude rotation angle unit is degree.

[0087] The displacement information is used as a displacement vector, and the attitude rotation angle is determined according to the attitude information. The attitude rotation angle is a quaternion rotation angle or an Euler angle. The magnitude of the attitude rotation angle and the magnitude of the displacement vector are compared. If the attitude rotation angle is greater than the magnitude of the displacement vector, the attitude planning is determined as the main axis planning, and the position planning is determined as the secondary axis planning. If the attitude rotation angle is less than the magnitude of the displacement vector, the position planning is determined as the main axis planning, and the attitude planning is determined as the secondary axis planning.

[0088] If the main axis planning is position planning and the secondary axis planning is attitude planning, then:

[0089] The spindle planning is as follows: the displacement information is used as the displacement vector, and the quintic spline function is used to perform speed planning based on the displacement vector to obtain the spindle planning speed; the quintic spline function is used to introduce a direction constraint on the spindle planning speed to obtain the spindle planning trajectory;

[0090] Here, the quintic spline function is used for planning. Due to the high-order continuous nature of the quintic spline function, when planning the vector position, after introducing constraints on the displacement and direction of the quintic spline function, the main axis planning speed, that is, the speed curve, and the direction planning result, that is, the secondary axis planning trajectory, can be obtained at the same time.

[0091] The secondary axis planning is: Determine the secondary axis planning ratio :

[0092]

[0093] Set the main axis speed to the ratio of the secondary axis speed Scale down in proportion to obtain the secondary axis planning speed, that is, the speed planning curve of the attitude angle;

[0094] If the main axis planning is posture planning and the secondary axis planning is position planning, then:

[0095] The main axis planning is: using the quintic spline function, speed planning is performed based on the attitude rotation angle to obtain the main axis planning speed, that is, the speed planning curve of the attitude angle;

[0096] The secondary axis planning is: Determine the secondary axis planning ratio :

[0097]

[0098] Set the main axis speed to the ratio of the secondary axis speed By scaling down the speed in equal proportion, the planned speed of the secondary axis is obtained, that is, the speed planning curve of the displacement; the quintic spline function is used to introduce direction constraints on the planned speed of the secondary axis to obtain the planned trajectory of the secondary axis.

[0099] The embodiment of the present application also provides a humanoid robot remote control method based on virtual reality technology, which is applied to the humanoid robot, comprising:

[0100] Receiving posture information; the posture information includes displacement information and posture information;

[0101] The displacement information is used as a displacement vector, and the posture rotation angle is determined according to the posture information. The magnitudes of the posture rotation angle and the displacement vector are compared. If the posture rotation angle is greater than the magnitude of the displacement vector, the posture planning is determined as the main axis planning, and the position planning is determined as the secondary axis planning. If the posture rotation angle is less than the magnitude of the displacement vector, the position planning is determined as the main axis planning, and the posture planning is determined as the secondary axis planning.

[0102] If the main axis planning is position planning and the secondary axis planning is attitude planning, then:

[0103] The spindle planning is as follows: the displacement information is used as the displacement vector, and the quintic spline function is used to perform speed planning based on the displacement vector to obtain the spindle planning speed; the quintic spline function is used to introduce a direction constraint on the spindle planning speed to obtain the spindle planning trajectory;

[0104] The secondary axis planning is: Determine the secondary axis planning ratio :

[0105]

[0106] Set the main axis speed to the ratio of the secondary axis speed Scale down in proportion to obtain the secondary axis planning speed;

[0107] If the main axis planning is posture planning and the secondary axis planning is position planning, then:

[0108] The main axis planning is: using the quintic spline function, the speed planning is performed based on the posture rotation angle to obtain the main axis planning speed;

[0109] The secondary axis planning is: Determine the secondary axis planning ratio :

[0110]

[0111] Set the main axis speed to the ratio of the secondary axis speed The secondary axis planning speed is obtained by scaling down in proportion; the quintic spline function is used to introduce directional constraints on the secondary axis planning speed to obtain the secondary axis planning trajectory.

[0112] The embodiment of the present application also provides a humanoid robot remote control method based on virtual reality technology, which is applied to VR equipment, including:

[0113] Obtain the posture information of the hand device, store the posture information in a double-ended FIFO queue, filter the posture information using a sliding mean filter, a Kalman filter, and a Butterworth low-pass filter to obtain the filtered posture information, which is the absolute coordinate in the virtual coordinate system;

[0114] The absolute coordinates in the virtual coordinate system are converted into the absolute coordinates in the actual coordinate system of the robot through the homogeneous transformation matrix, and then the absolute coordinates in the actual coordinate system of the robot are converted into the relative coordinates in the actual coordinate system of the robot through differential operation, which are recorded as the converted posture information;

[0115] The double-ended FIFO queue sends the converted posture information to the humanoid robot at a fixed period; the VR device does not access the position return frame of the humanoid robot and only performs the sending operation.

[0116] In summary, this application has the following technical effects:

[0117] Low threshold: Compared with traditional robot programming and control methods, this application uses hand devices for dynamic tracking control, which does not require complex programming knowledge and experience, lowers the usage threshold, and enables more people to easily use robots for operation.

[0118] Strong interactivity: The robot and the end of the arm are equipped with visual sensors. Users can open the window to observe the actual operating space while using VR. At the same time, the force-controlled gripper at the end of the robotic arm can provide real-time feedback of the current torque current, allowing the operator to achieve real-time visual and force interaction with the environment.

[0119] Low latency: The speed fusion planning algorithm based on time cycle is adopted to make the robot highly sensitive to time cycle when following the hand movements, which reduces the control delay during remote operation to a certain extent.

[0120] Speed ​​smoothing: This application performs multiple filtering processes and speed re-planning algorithms to address situations such as communication packet loss and uneven position data sampling, to ensure smooth position when the joint motor is executed.

[0121] High safety: By limiting the actual hardware activity space of the robot arm and implementing multi-order smoothing of the terminal trajectory based on the multi-order differential constraints of the spline function, the path and speed are smoothed while limiting the actual hardware activity space of the robot arm. This can effectively avoid collisions or out-of-range activities during movement, thus ensuring the safety and stability of the robot.

[0122] High flexibility: The humanoid robot used has dual seven-axis arms with 14 degrees of freedom (including redundant degrees of freedom). The head, neck, and waist are equipped with servo motors for rotation. The base is an AGV cart that supports forward, backward, left, and right movement and rotation. Users can switch the arms, head, waist, neck, and base AGV control in the VR control interface, which has extremely high flexibility.

[0123] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A humanoid robot remote control system based on virtual reality technology, characterized in that: include: VR equipment and a humanoid robot, wherein the VR equipment includes a head-mounted device, a hand device, a visual information sampling unit, a hand device sampling unit, and a visual information imaging unit; the humanoid robot includes a robot system communication unit, a robot system planning control unit, a visual sensor, and a joint motor execution unit; The visual information sampling unit collects environmental information around the user, and the visual sensor collects environmental information around the humanoid robot and sends it to the visual information sampling unit through the robot system communication unit; the visual information sampling unit sends the acquired environmental information to the visual information imaging unit for imaging processing to obtain images around the user and the humanoid robot; the images around the user and the humanoid robot are transmitted to the head mounted device and displayed in a visual interface; The visual interface of the head mounted device is also provided with a control interface, and the user drags the control interface through the hand device to switch the control mode of the humanoid robot; If the control mode of the humanoid robot is the arm following mode, when the hand device moves, the hand device sampling unit obtains the posture information of the hand device, and sends the posture information to the robot system planning control unit through the robot system communication unit; the robot system planning control unit performs position planning and posture planning of the humanoid robot arm according to the received posture information, obtains the position planning result and the posture planning result, and sends the position planning result and the posture planning result to the joint motor execution unit to control the arm movement of the humanoid robot; The robot system planning control unit performs position planning and posture planning of the humanoid robot arm according to the received position and posture information, and obtains position planning results and posture planning results, including: The position and posture information includes displacement information and posture information; Taking the displacement information as a displacement vector, determining a posture rotation angle according to the posture information, comparing the magnitudes of the posture rotation angle and the displacement vector, if the posture rotation angle is greater than the magnitude of the displacement vector, determining the posture planning as the main axis planning, and determining the position planning as the secondary axis planning, if the posture rotation angle is less than the magnitude of the displacement vector, determining the position planning as the main axis planning, and determining the posture planning as the secondary axis planning; If the main axis planning is position planning and the secondary axis planning is attitude planning, then: The spindle planning is as follows: taking the displacement information as a displacement vector, using a quintic spline function to perform speed planning based on the displacement vector, and obtaining a spindle planning speed; using a quintic spline function to introduce a direction constraint on the spindle planning speed, and obtaining a spindle planning trajectory; The secondary axis planning is: Determine the secondary axis planning ratio r: Set the main axis planning speed to the ratio of the secondary axis planning Scale down in proportion to obtain the secondary axis planning speed; If the main axis planning is posture planning and the secondary axis planning is position planning, then: The main axis planning is as follows: using a quintic spline function, performing speed planning based on the posture rotation angle, and obtaining the main axis planning speed; The secondary axis planning is: Determine the secondary axis planning ratio : Set the main axis planning speed to the ratio of the secondary axis planning The secondary axis planning speed is obtained by scaling down in proportion; the quintic spline function is used to introduce directional constraints on the secondary axis planning speed to obtain the secondary axis planning trajectory.

2. The system according to claim 1, characterized in that in, The hand device sampling unit acquires the posture information of the hand device, and sends the posture information to the robot system planning control unit through the robot system communication unit, including: After the hand device sampling unit acquires the posture information of the hand device, the posture information is stored in a double-ended FIFO queue, and the posture information is filtered using a sliding mean filter, a Kalman filter, and a Butterworth low-pass filter to obtain filtered posture information, where the filtered posture information is an absolute coordinate in a virtual coordinate system; The absolute coordinates in the virtual coordinate system are converted into the absolute coordinates in the actual coordinate system of the robot through a homogeneous transformation matrix, and then the absolute coordinates in the actual coordinate system of the robot are converted into the relative coordinates in the actual coordinate system of the robot through a differential operation, which are recorded as the converted posture information; The double-ended FIFO queue sends the converted posture information to the humanoid robot at a fixed period; the VR device does not access the position return frame of the humanoid robot and only performs the sending operation.

3. The system according to claim 1, characterized in that When the humanoid robot is abnormal, the abnormal information is sent to the VR device, and the head mounted device displays that the humanoid robot is currently in an abnormal state; Send a robot stop signal and interrupt the communication between the VR device and the humanoid robot.

4. The system according to claim 1, characterized in that The posture rotation angle is a quaternion rotation angle or an Euler angle.

5. The system according to claim 1, wherein: The humanoid robot also includes: a force control sensor, which collects contact torque information of the gripper at the end of the arm of the humanoid robot, and sends the contact torque information to the visual information sampling unit through the robot system communication unit; the visual information sampling unit transmits the contact torque information to the visual information imaging unit to generate a current waveform, and sends the current waveform to the head-mounted device for display in a visual interface; the current waveform enables the user to adjust the gripping force of the gripper at the end of the arm of the humanoid robot.

6. The system according to claim 1, characterized in that The humanoid robot control modes also include a chassis movement mode, a head rotation mode and a waist rotation mode; when switching to the corresponding humanoid robot control mode, the humanoid robot is made to perform corresponding movements through the operation buttons on the hand device.

7. The system according to claim 6, characterized in that The humanoid robot is provided with a mobile base. When the control mode of the humanoid robot is a chassis movement mode, the VR device can control the movement of the mobile base through a hand device.

8. The system according to claim 6, characterized in that The humanoid robot is provided with a rotatable head. When the control mode of the humanoid robot is a head rotation mode, the VR device can control the rotation of the head through a hand device.

9. A humanoid robot remote control method based on virtual reality technology, applied to a humanoid robot, characterized in that: include: Receiving position and posture information; the position and posture information includes displacement information and posture information; Taking the displacement information as a displacement vector, determining a posture rotation angle according to the posture information, comparing the magnitudes of the posture rotation angle and the displacement vector, if the posture rotation angle is greater than the magnitude of the displacement vector, determining the posture planning as the main axis planning, and determining the position planning as the secondary axis planning, if the posture rotation angle is less than the magnitude of the displacement vector, determining the position planning as the main axis planning, and determining the posture planning as the secondary axis planning; If the main axis planning is position planning and the secondary axis planning is attitude planning, then: The spindle planning is as follows: taking the displacement information as a displacement vector, using a quintic spline function to perform speed planning based on the displacement vector, and obtaining a spindle planning speed; using a quintic spline function to introduce a direction constraint on the spindle planning speed, and obtaining a spindle planning trajectory; The secondary axis planning is: Determine the secondary axis planning ratio : Set the main axis planning speed to the ratio of the secondary axis planning Scale down in proportion to obtain the secondary axis planning speed; If the main axis planning is posture planning and the secondary axis planning is position planning, then: The main axis planning is as follows: using a quintic spline function, performing speed planning based on the posture rotation angle, and obtaining the main axis planning speed; The secondary axis planning is: Determine the secondary axis planning ratio : Set the main axis planning speed to the ratio of the secondary axis planning The secondary axis planning speed is obtained by scaling down in proportion; the quintic spline function is used to introduce directional constraints on the secondary axis planning speed to obtain the secondary axis planning trajectory.

10. The method according to claim 9, characterized in that The posture rotation angle is a quaternion rotation angle or an Euler angle.

Images