Remote operation system and control method for two mechanical arms based on VR head-mounted display

Abstract

According to the double-arm mechanical arm teleoperation system based on the VR head-mounted display and the control method, high-precision and flexible teleoperation is achieved by accurately capturing movement tracks of the two hands and mapping the movement tracks to the double-arm mechanical arm, meanwhile, the content shot by the scene camera is transmitted to the VR head-mounted display in a streaming mode, and the real-time performance and immersion of operation are enhanced. In addition, the intelligent obstacle avoidance strategy designed by the invention effectively prevents the movement of the mechanical arm from being out of control in the operation process, and ensures the safety and reliability of the operation. The system has the remarkable advantages of high-precision operation, real-time performance, immersion, multi-scene adaptability, safety, reliability and the like, is suitable for interaction and task execution of a universal robot in industrial, household and other scenes, replaces human beings to execute precise operation in a high-risk environment, and has wide application prospects and important practical significance.

Description

Technical Field

[0001] The present invention relates to the technical field of robot teleoperation, and particularly relates to a two-arm manipulator teleoperation system and control method based on a VR headset. Background Art

[0002] In the fields of industrial production, home service, and high-risk environment operations, etc., the application of robot technology is becoming more and more extensive. Traditional robot operation methods mostly rely on manual programming or simple remote control devices, with limited operation accuracy and flexibility, and it is difficult to achieve precise operation in high-risk environments.

[0003] With the development of virtual reality (VR) technology, the combination of VR and robot operation has become a new trend. However, most current related technologies can only achieve single-arm operation, and there are deficiencies in aspects such as gesture recognition accuracy, motion control smoothness, and scene adaptability. Summary of the Invention

[0004] To achieve the above-mentioned objects and other advantages of the present invention, the first object of the present invention is to provide a two-arm manipulator teleoperation control method based on a VR headset, including the following steps:

[0005] Receive the user's two-handed gesture data sent by the VR headset;

[0006] Process the gesture data to obtain the user's two-handed motion trajectory data;

[0007] Convert the motion trajectory data into motion instructions for the two-arm manipulator;

[0008] Send the motion instructions to the robot configured with the two-arm manipulator to achieve real-time teleoperation of the two-arm manipulator.

[0009] Further, the step of processing the gesture data to obtain the user's two-handed motion trajectory data includes:

[0010] Extract the key feature points of the hand from the gesture data;

[0011] Calculate the velocity vector of the hand motion trajectory by differential calculation of the position of the hand in three-dimensional space. The formula is:

[0012]

[0013] P(t) = (x(t), y(t), z(t))

[0014] where V(t) is the velocity vector, P(t) is the position of the hand in three-dimensional space, t is the time stamp, and Δt is the time interval;

[0015] Filter the velocity vector to obtain a smoothed velocity vector.

[0016] Further, the step of filtering the velocity vector to obtain a smoothed velocity vector includes:

[0017] Initialize the state vector and covariance matrix of the Kalman filter;

[0018] At each time step, iteratively calculate the state vector according to the prediction equation and update equation to obtain smoothed motion trajectory data. The formula is:

[0019] V′(t)=K(t)·Z(t)+(1-K(t))·V(t)

[0020] where K(t) is the Kalman gain and Z(t) is the observed value.

[0021] Further, the step of converting the motion trajectory data into motion commands for a dual-arm manipulator includes:

[0022] Set the motion model and dynamic parameters of the dual-arm manipulator, and calculate the target positions and velocities of the joints according to the smoothed velocity vector;

[0023] Use a time-based interpolation method to generate a smooth motion trajectory curve to ensure the smoothness of the manipulator's motion;

[0024] Convert the processed data into motion commands recognizable by the dual-arm manipulator.

[0025] Further, it also includes the step:

[0026] Receive the real-time environmental information of the operation scenario of the dual-arm manipulator sent by the robot;

[0027] Construct an environmental map according to the real-time environmental information;

[0028] According to the environmental map, the current position and motion trajectory of the dual-arm manipulator, determine in real time whether the manipulator is approaching a singularity or an obstacle;

[0029] When the manipulator is approaching a singularity or an obstacle, adjust the motion speed and direction of the manipulator to make it move away from the singularity or the obstacle.

[0030] Further, when the manipulator is approaching a singularity or an obstacle, the specific adjustment of the motion speed of the manipulator is:

[0031] When the distance between the manipulator and the singularity or the obstacle is less than the safety distance threshold, adjust the motion speed of the manipulator. The formula is:

[0032]

[0033] Among them, v(t) is the movement speed of the robotic arm, d(t) is the distance between the robotic arm and the singularity or obstacle, and d threshold is the safety distance threshold between the robotic arm and the singularity or obstacle.

[0034] Furthermore, it also includes the steps of:

[0035] Converting the real-time environmental information into data in streaming media format;

[0036] Transmitting the data in streaming media format to the VR headset to realize the real-time display of the operation scenario on the VR headset.

[0037] The second object of the present invention is to provide a dual-arm robotic arm teleoperation system based on a VR headset. Applying the above method, it includes a VR headset, a robot configured with dual-arm robotic arms and a scene camera, and a host computer; among them,

[0038] The VR headset is used to capture the user's two-handed gesture data, and restore and display in real time the data in streaming media format transmitted by the host computer;

[0039] The scene camera is used to collect the real-time environmental information of the dual-arm robotic arm operation scenario and send it to the host computer;

[0040] The host computer is used to process the gesture data to obtain the user's two-handed movement trajectory data, convert the movement trajectory data into movement instructions for the dual-arm robotic arm, send the movement instructions to the robot, convert the environmental information into data in streaming media format and transmit it to the VR headset, and judge in real time according to the environmental information whether the robotic arm is approaching a singularity or an obstacle to adjust the movement speed and direction of the robotic arm;

[0041] The robot is used to realize the real-time teleoperation of the dual-arm robotic arm according to the movement instructions.

[0042] The third object of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0043] The fourth object of the present invention is to provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] The present invention provides a two - armed robotic arm tele - operation system and control method based on a VR headset. By accurately capturing the motion trajectories of both hands and mapping them to the two - armed robotic arm, high - precision and flexible tele - operation are achieved. At the same time, the content captured by the scene camera is streamed to the VR headset, enhancing the real - time nature and immersion of the operation. In addition, the intelligent obstacle - avoidance strategy designed in the present invention effectively avoids the out - of - control movement of the robotic arm during operation, ensuring the safety and reliability of the operation. This system has significant advantages such as high - precision operation, real - time nature and immersion, multi - scenario adaptability, and safety and reliability. It is applicable to the interaction and task execution of general robots in industrial, household and other scenarios, as well as replacing humans to perform precision operations in high - risk environments, having broad application prospects and important practical significance.

[0046] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly and implement it in accordance with the content of the specification, the following takes the preferred embodiments of the present invention and combines with the attached drawings to elaborate in detail as follows. The specific implementation manners of the present invention are given in detail by the following embodiments and their attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present invention and form a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0048] Figure 1 is a schematic diagram of a two - armed robotic arm tele - operation system based on a VR headset;

[0049] Figure 2 is a signaling flow chart of a two - armed robotic arm tele - operation based on a VR headset;

[0050] Figure 3 is a flow chart of a control method for a two - armed robotic arm tele - operation based on a VR headset;

[0051] Figure 4 is a flow chart of gesture data processing;

[0052] Figure 5 is a flow chart of speed vector filtering processing;

[0053] Figure 6 is a flow chart of motion trajectory data conversion;

[0054] Figure 7 is a flow chart of robotic arm adjustment;

[0055] Figure 8 is a flow chart of real - time display of the operation scene on the VR headset;

[0056] Figure 9 is a flow chart of a two - armed robotic arm tele - operation based on a VR headset;

[0057] Figure 10 is a flowchart for gesture recognition and motion mapping;

[0058] Figure 11 is a flowchart for obstacle avoidance and singularity avoidance control;

[0059] Figure 12 is a flowchart for scene camera streaming;

[0060] Figure 13 is a schematic diagram of a computer device;

[0061] Figure 14 is a schematic diagram of a computer-readable storage medium. Specific Embodiments

[0062] Next, in combination with the accompanying drawings and specific embodiments, the present invention will be further described. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, on the premise of no conflict, the following-described embodiments or technical features can be arbitrarily combined to form new embodiments.

[0063] All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0064] In this application, the attached drawing numbers are only used to distinguish each step in the solution and are not used to limit the execution order of each step. The specific execution order shall be subject to the description in the specification.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0066] Embodiment 1

[0067] A two-armed robotic arm teleoperation system based on a VR headset, as Figures 1 - 2 , Figure 9 shown, includes a VR headset 100, a host computer 200, and a robot 300 configured with two-armed robotic arms and a scene camera; wherein,

[0068] The VR headset is used to capture the gesture data of the user's hands, and restore and display in real time the streaming media format data transmitted by the host computer;

[0069] Among them, a VR headset refers to a virtual reality head-mounted display device that closes a person's vision and hearing to the outside world and guides the user to have a feeling of being in a virtual environment. Its display principle is that the left and right eye screens respectively display the images of the left and right eyes. After the human eye obtains this different information, a three-dimensional sense is generated in the mind. Optionally, the VR headset adopts a VR glasses, a VR helmet, a VR eye mask, etc. In this embodiment, a high-performance VR glasses is preferably used, which has a good display effect and a comfortable wearing experience, and is built-in with a high-precision vision sensor for capturing the gesture movement information of the user. At the same time, a wireless communication module is equipped to transmit data to the host computer through a Wi-Fi or 5G network.

[0070] The VR glasses not only display the content captured by the scene camera to the operating user, but also capture the user's gestures and movements based on visual information, realizing real-time interaction between the operator and the robotic arm. This interactive design enables the operator to control the robotic arm more naturally and intuitively, improving the convenience and efficiency of operation, which is an important innovation and improvement to the traditional teleoperation user interface.

[0071] The scene camera is used to collect the real-time environmental information of the dual-arm robotic arm operation scene and send it to the host computer;

[0072] In this embodiment, a binocular camera is adopted, which has the shooting capabilities of high resolution and high frame rate, and can clearly capture the RGB information and depth information of the operation scene. The camera can communicate with the host computer through a wired or wireless (Wi-Fi or 5G) method to realize the real-time transmission of the captured content. The depth information assists the host computer and the robotic arm to complete obstacle avoidance and singularity avoidance operations, improving the safety and reliability of the system.

[0073] The host computer is used to process the gesture data to obtain the motion trajectory data of the user's both hands, convert the motion trajectory data into the motion instructions of the dual-arm robotic arm, send the motion instructions to the robot, convert the environmental information into streaming media format data and transmit it to the VR headset, and judge in real time according to the environmental information whether the robotic arm is approaching a singularity or an obstacle to adjust the motion speed and direction of the robotic arm;

[0074] In this embodiment, a high-performance industrial computer is adopted, which is equipped with professional robot control software and motion planning algorithms. It has powerful data processing capabilities and real-time control capabilities, can quickly receive and process the data from the VR glasses, and accurately control the motion of the robotic arm.

[0075] The robot is used to realize real-time teleoperation of the dual-arm robotic arm according to the motion instructions.

[0076] In this embodiment, an industrial-grade dual-arm robot with multiple joints is selected, which has high-precision motion control capabilities and good load performance, and can meet the operation requirements in different scenarios.

[0077] In this embodiment, the VR glasses are used to capture the motion trajectories of the user's left and right hands respectively. The host computer calculates the corresponding velocity vectors based on the motion positions and timestamps, and then calculates the joint motion trajectories and maps them to the corresponding robot arms through the velocity control mode. This precise capture and mapping of the motion trajectories of both hands achieve high-precision teleoperation of the dual-arm robot, enabling it to complete more complex and delicate operation tasks, such as the actions of dual-arm cooperation for assembly, etc., which are relatively rare in the prior art, and significantly improving the precision and flexibility of teleoperation.

[0078] A dual-arm robot teleoperation system based on a VR headset provided in this embodiment can be used to execute a Figure 2 dual-arm robot teleoperation control method based on a VR headset as shown. The method includes the following steps:

[0079] Step S101: The VR headset captures the gesture data of the user's both hands;

[0080] Step S102: The VR headset captures and sends the gesture data to the host computer; optionally, the VR headset sends the data to the host computer through a wireless network.

[0081] Step S103: The host computer receives the gesture data, processes the gesture data to obtain the motion trajectory data of the user's both hands, and converts the motion trajectory data into the motion instructions of the dual-arm robot.

[0082] In some embodiments, as Figure 10 shown, the processing process of the host computer for the gesture data includes:

[0083] Extract the key feature points of the hand, such as the position information of the fingertips, wrists, etc. through an image processing algorithm. Set the position of the hand in the three-dimensional space as P(t) = (x(t), y(t), z(t)), and the timestamp as t, then the velocity vector V(t) can be obtained through differential calculation:

[0084]

[0085] where Δt is the time interval.

[0086] After smoothing the velocity vector, map it to the end of the robot arm. The specific method is as follows:

[0087] Filter the velocity vector V(t) through the Kalman filter algorithm to obtain the smoothed velocity vector V′(t):

[0088] V′(t) = K(t)·Z(t) + (1 - K(t))·V(t)

[0089] Where K(t) is the Kalman gain and Z(t) is the observed value.

[0090] Combined with the Kalman filtering algorithm, the captured motion trajectory (velocity vector) is pre-filtered to smooth the trajectory curve and reduce noise interference. The specific implementation steps are as follows:

[0091] Initialize the state vector and covariance matrix of the Kalman filter.

[0092] At each time step, the state vector is iteratively calculated according to the prediction equation and update equation to obtain the smoothed motion trajectory data.

[0093] Convert the processed data into motion commands recognizable by the robotic arm and send them to the robot.

[0094] Calculate the joint motion trajectory according to the smoothed velocity vector V′(t), and use the inverse kinematics method to solve the robotic arm joint angle θ(t):

[0095] θ(t) = f -1 (V′(t))

[0096] Where f -1 is the inverse kinematics function.

[0097] Optionally, according to the preprocessed data, use the inverse kinematics method to calculate the target position and velocity of the robotic arm joints. Combine the specific requirements of this system for motion trajectory planning and control. The specific implementation steps are as follows:

[0098] Set the motion model and dynamic parameters of the robotic arm, and calculate the target position and velocity of the joints according to the gesture motion trajectory (velocity vector).

[0099] Use the time-based interpolation method to generate a smooth motion trajectory curve to ensure the smoothness of the robotic arm movement.

[0100] S104: The host computer sends the motion command to the robot, that is, sends the calculated motion command to the robotic arm motion control module to drive the robotic arm to move according to the planned trajectory.

[0101] Based on the above steps, the operator's hand motion trajectory can be mapped to the robotic arm to achieve real-time teleoperation.

[0102] In this embodiment, when capturing the hand movement trajectory, the Kalman filtering algorithm is introduced to smooth and predict the data, effectively reducing the noise interference, further improving the accuracy and stability of gesture recognition, and ensuring the smoothness and accuracy of the robotic arm movement. This is an important improvement and innovation to the existing teleoperation technology.

[0103] S105: The robot receives the motion instruction and realizes real-time teleoperation of the dual-arm robotic arm according to the motion instruction.

[0104] The system will also monitor the motion state and environmental information of the robotic arm in real time in the form of an independent thread. When detecting approaching a singularity or an obstacle, it automatically adjusts the motion speed and direction to achieve the functions of obstacle avoidance and singularity avoidance. At the same time, corresponding prompt information is displayed in the VR visualization interface to remind the operator to pay attention to safety.

[0105] S106: The robot collects the real-time environmental information of the operation scene of the dual-arm robotic arm through the scene camera;

[0106] S107: The robot sends the real-time environmental information to the host computer.

[0107] In some embodiments, as Figure 11 shown, the environmental perception and obstacle avoidance principle of the host computer is as follows:

[0108] The scene binocular camera installed on the robotic arm can obtain the three-dimensional environmental information of the operation scene in real time. Then, S108: The host computer receives the real-time environmental information, constructs an environmental map according to the real-time environmental information; and based on the environmental map and the current position and motion trajectory of the dual-arm robotic arm, it judges in real time whether the robotic arm is approaching a singularity or an obstacle;

[0109] S109: When the robotic arm is approaching a singularity or an obstacle, it adjusts the motion speed and direction of the robotic arm to make it move away from the singularity or the obstacle. The specific adjustment strategy is as follows:

[0110] Set the safety distance threshold d threshold between the robotic arm and the singularity or the obstacle. When the distance d(t) between the robotic arm and the singularity or the obstacle is less than d threshold , adjust the motion speed v(t) of the robotic arm:

[0111]

[0112] In the VR visualization interface, in ways such as highlighting the obstacle area and displaying the obstacle avoidance path, it prompts the operator of the obstacle avoidance state of the robotic arm in real time, enhancing the safety and reliability of the operation.

[0113] When the robotic arm approaches a singularity or an obstacle, the host computer automatically displays a warning message in the VR visualization interface to remind the operator to pay attention to safety. The warning message can include text prompts, highlighting the obstacle area, displaying the obstacle avoidance path, etc., to ensure that the operator can take timely measures to avoid collision accidents.

[0114] When the robotic arm encounters a singularity or approaches an obstacle, the system can automatically reduce the speed until it stops, or move away from the singularity or the obstacle, and display corresponding prompt information in the VR visualization interface. This intelligent obstacle avoidance strategy effectively avoids the collision of the robotic arm with obstacles during operation or the loss of motion control caused by getting stuck in a singularity, ensuring the safety and reliability of the operation.

[0115] In some embodiments, as Figure 12 shown, the principle of the streaming transmission of the operation scene image is as follows:

[0116] S110: The host computer converts the real-time environment information into data in a streaming media format. Specifically, the scene camera captures the operation scene and streams the captured content to the VR glasses through the HLS protocol to ensure the stability and real-time of the transmission. The specific steps are as follows:

[0117] Use a binocular scene camera to capture the real-time RGB image and point cloud image of the operation scene, and convert the image data into a streaming media format supported by the HLS protocol through an encoder.

[0118] S111: The host computer transmits the streaming media format data to the VR headset. Specifically, an efficient network transmission protocol, such as Wi-Fi or 5G network, is used to transmit the streaming media data to the VR glasses. During the transmission, the resolution and frame rate of the image are dynamically adjusted according to the network bandwidth and latency to ensure the stability and real-time of the transmission.

[0119] S112: The VR headset receives the streaming media format data, restores the streaming media format data and displays it in real time. Specifically, after the VR glasses receive the streaming media data, it is restored to an image through a decoder and displayed in real time on the display screen, enabling the operator to clearly observe the operation scene, further enhancing the safety and reliability of the operation.

[0120] By displaying the operation scene captured by the scene camera in real time through the VR glasses, the operator can observe the operation environment as if he were on the spot, improving the immersion and real-time of the operation. The operator can adjust the viewing angle in the VR interface by turning the head or making gesture operations to observe the operation scene more comprehensively. The application of this streaming transmission technology optimizes the efficiency and stability of data transmission, reduces operation latency, improves the accuracy and efficiency of the operation, and provides a more intuitive and realistic operation experience for the operator.

[0121] In some embodiments, the control of the robotic arm gripper is determined by the pinching and releasing of the fingers. The specific control logic is as follows:

[0122] Set the finger pinching state as C(t). When C(t) = 1, the gripper closes; when C(t) = 0, the gripper opens. The pinching action of the fingers is captured by the VR glasses, and the host computer converts it into a gripper control signal to control the action of the robotic arm gripper.

[0123] In some embodiments, the operation process of the dual-arm robotic arm teleoperation system based on the VR headset is as follows:

[0124] System initialization: Turn on the dual-arm robotic arm and the scene camera, perform device self-check and calibration to ensure that all components are working properly;

[0125] Start the VR glasses, enter the system main interface, perform user login and device connection operations, and establish a communication connection with the host computer;

[0126] The host computer loads the preset motion planning model and control parameters, initializes algorithm modules such as the Kalman filter, and prepares to receive gesture data and control the movement of the robotic arm.

[0127] Operation preparation: The operator wears the VR glasses and performs gesture calibration and initialization operations through the gesture recognition module to ensure that the VR glasses can accurately capture the hand movement information;

[0128] In the display interface of the VR glasses, observe the operation scene captured by the scene camera, adjust the position and posture of oneself to place both hands within the appropriate operation area;

[0129] Through the virtual buttons or voice commands in the VR interface, start the standby state of the robotic arm and wait for the operator's gesture commands.

[0130] Teleoperation execution: The operator performs various gesture actions such as grasping, moving, and rotating with both hands in the virtual space. The VR glasses capture the hand movement trajectory in real time, send the processed data to the host computer, and perform smoothing processing through the Kalman filtering algorithm;

[0131] After receiving the gesture data, the host computer calculates the joint movement trajectory of the robotic arm according to the preset motion planning model and control algorithm, and sends it to the robotic arm motion control module to drive the robotic arm to move according to the planned trajectory and complete the corresponding operation tasks;

[0132] During operation, the scene camera captures the operation scene in real time and transmits the images to the VR glasses through the streaming module, enabling the operator to observe the motion state and operation effect of the robotic arm in real time. Meanwhile, the host computer monitors the environmental information and the motion state of the robotic arm in real time. When it detects approaching singularities or obstacles, it automatically adjusts the motion speed to achieve the functions of obstacle avoidance and singularity avoidance, and displays corresponding prompt information in the VR visualization interface to remind the operator to pay attention to safety.

[0133] Operation end: After completing the operation task, the operator stops the motion of the robotic arm through preset gestures and makes the robotic arm return to the initial position or a safe position;

[0134] After receiving the stop command, the host computer controls the robotic arm to stop smoothly and performs reset and calibration operations on the motion state;

[0135] The operator removes the VR glasses, and the system automatically saves the motion data and log information of this operation for subsequent analysis and optimization.

[0136] In some embodiments, the application of the dual-arm robotic arm teleoperation system based on a VR headset in an industrial assembly scenario is as follows:

[0137] Task description: In the factory, it is necessary to complete the assembly task of components. The operator controls the dual-arm robotic arm to accurately install the components to the specified position through the teleoperation system of the present invention.

[0138] Operation process: The operator wears VR glasses, enters the main interface of the system, and starts the standby state of the robotic arm;

[0139] The operator simulates the action of grasping components with both hands. The VR glasses capture the hand movement trajectory, and after Kalman filtering processing, it is sent to the host computer; the host computer calculates the robotic arm joint movement trajectory and drives the robotic arm to grasp the components;

[0140] The operator moves both hands to the target assembly position. The VR glasses capture the movement trajectory in real time, and the host computer controls the robotic arm to move according to the planned trajectory and accurately place the components to the specified position;

[0141] During the operation process, the scene camera captures the assembly scene in real time and transmits it to the VR glasses through streaming. The operator can clearly observe the assembly effect; meanwhile, the host computer monitors the environmental information. When it detects that the robotic arm is approaching other equipment or obstacles, it automatically adjusts the motion speed and direction to avoid collisions, and displays prompt information in the VR visualization interface to remind the operator to pay attention to safety.

[0142] Effect evaluation: Through the teleoperation system of the present invention, the operator can complete the assembly task of components with high precision and efficiency, and no collision accidents occur during the operation process, effectively ensuring production safety and product quality.

[0143] The present invention provides a teleoperation system for a two-armed robotic arm based on a VR headset. The system obtains the movement trajectories of the user's two hands through the VR glasses, and maps these trajectories to the two-armed robotic arm to achieve high-precision teleoperation. At the same time, the system also includes streaming the content captured by the scene camera (binocular camera) to the VR glasses to enhance the real-time performance and immersion of the operation. The system also designs a special intelligent obstacle avoidance strategy for the teleoperation task to avoid situations that may damage the equipment or the operation object during remote operation. This system is applicable to the interaction and task execution of general robots in industrial, household and other scenarios, and to perform precise operations on behalf of humans in high-risk environments. This design with multi-scenario adaptability enables the system to have a broader application prospect and practical significance, can meet the operation requirements in different scenarios, and provides the possibility for the application of robot technology in more fields.

[0144] Compared with traditional teleoperation solutions (such as joysticks, teach pendants, motion capture devices, etc.), the present invention only requires a VR headset to complete the monitoring and control of the robot operation environment and its body. This design greatly simplifies the operation process, reduces the operation complexity, and makes the system easier to use and promote.

[0145] By capturing natural gestures through the VR headset for control, the operator can operate with both hands as naturally as in the real world without having to learn complex control commands or operation interfaces. This natural gesture control method further improves the naturalness and intuitiveness of the operation, and significantly enhances the user-friendliness and market competitiveness of the system.

[0146] Embodiment 2

[0147] A teleoperation control method for a two-armed robotic arm based on a VR headset, which is applied to the above-mentioned teleoperation system for a two-armed robotic arm based on a VR headset. For a detailed description of the teleoperation system for a two-armed robotic arm based on a VR headset, reference can be made to the corresponding description in the above-mentioned embodiment of the teleoperation system for a two-armed robotic arm based on a VR headset, which will not be elaborated here. As Figures 2 - 3 、 Figure 9 shown, the method includes the following steps:

[0148] S210. Receive the user's two-handed gesture data sent by the VR headset;

[0149] Among them, after the VR headset captures the user's two-handed gesture data, it sends the user's two-handed gesture data to the VR headset. Optionally, the VR headset sends the data to the host computer through a wireless network. Then, the host computer receives the user's two-handed gesture data sent by the VR headset.

[0150] In this embodiment, the VR headset refers to a virtual reality head-mounted display device that closes a person's vision and hearing from the outside world and guides the user to have a feeling of being in a virtual environment. Its display principle is that the left and right eye screens respectively display the images of the left and right eyes. After the human eye obtains this different information, a stereoscopic feeling is generated in the mind. Optionally, the VR headset adopts VR glasses, VR helmets, VR eye masks, etc. In this embodiment, high-performance VR glasses are preferably used, which have good display effects and comfortable wearing experiences, and are built-in with high-precision visual sensors for capturing the gesture motion information of the user. At the same time, a wireless communication module is equipped to transmit data to the host computer through Wi-Fi or 5G network.

[0151] The VR glasses not only display the content captured by the scene camera to the operating user, but also capture the user's gestures and movements based on visual information, realizing real-time interaction between the operator and the robotic arm. This interactive design enables the operator to control the robotic arm more naturally and intuitively, improving the convenience and efficiency of operation, which is an important innovation and improvement to the traditional teleoperation user interface.

[0152] In this embodiment, a high-performance industrial computer is adopted, which is equipped with professional robot control software and motion planning algorithms. It has powerful data processing capabilities and real-time control capabilities, can quickly receive and process data from the VR glasses, and accurately control the movement of the robotic arm.

[0153] S220. Process the gesture data to obtain the user's two-handed motion trajectory data;

[0154] Further, as Figure 4 , Figure 10 shown, the step of processing the gesture data to obtain the user's two-handed motion trajectory data includes:

[0155] S221. Extract the key feature points of the hand from the gesture data, such as the position information of the fingertips, wrists, etc.

[0156] S222. Calculate the velocity vector of the hand motion trajectory by taking the difference of the position of the hand in the three-dimensional space. The formula is:

[0157]

[0158] P(t) = (x(t), y(t), z(t))

[0159] where V(t) is the velocity vector, P(t) is the position of the hand in the three-dimensional space, t is the timestamp, and Δt is the time interval;

[0160] S223. Perform filtering processing on the velocity vector to obtain a smoothed velocity vector.

[0161] Then, after smoothing the velocity vector, it is mapped to the end of the robotic arm. In this embodiment, the Kalman filter algorithm is used to filter the velocity vector V(t) to obtain the smoothed velocity vector V′(t). Further, as Figure 5 shown, the step of filtering the velocity vector to obtain the smoothed velocity vector includes:

[0162] S2231. Initialize the state vector and covariance matrix of the Kalman filter;

[0163] S2232. At each time step, perform iterative calculations on the state vector according to the prediction equation and update equation to obtain the smoothed motion trajectory data. The formula is:

[0164] V′(t) = K(t)·Z(t) + (1 - K(t))·V(t)

[0165] where K(t) is the Kalman gain and Z(t) is the observed value.

[0166] S230. Convert the motion trajectory data into motion commands for the dual-arm robotic arm;

[0167] Combined with the Kalman filter algorithm, perform filtering preprocessing on the captured motion trajectory (velocity vector), smooth the trajectory curve, and reduce noise interference.

[0168] According to the preprocessed data, use the inverse kinematics method to calculate the target positions and velocities of the robotic arm joints. Combine the specific requirements of this system to perform motion trajectory planning and control. Further, as Figure 6 shown, the step of converting the motion trajectory data into motion commands for the dual-arm robotic arm includes:

[0169] S231. Set the motion model and dynamic parameters of the dual-arm robotic arm, and calculate the target positions and velocities of the joints according to the smoothed velocity vector;

[0170] Optionally, calculate the joint motion trajectory according to the smoothed velocity vector V′(t), and use the inverse kinematics method to solve the robotic arm joint angle θ(t):

[0171] θ(t) = f -1 (V′(t))

[0172] where f -1 is the inverse kinematics function.

[0173] S232. Use the time-based interpolation method to generate a smooth motion trajectory curve to ensure the smoothness of the robotic arm motion;

[0174] S233. Convert the processed data into motion instructions recognizable by the dual-arm robot arm. Then send the calculated motion instructions to the robot arm motion control module to drive the robot arm to move along the planned trajectory.

[0175] In this embodiment, the VR glasses are used to capture the motion trajectories of the user's left and right hands respectively. The host computer calculates the corresponding velocity vectors based on the motion positions and timestamps, and then calculates the joint motion trajectories through the velocity control mode and maps them to the corresponding robot arm. This precise capture and mapping of the motion trajectories of both hands achieve high-precision teleoperation of the dual-arm robot arm, enabling more complex and delicate operation tasks to be completed, such as the actions of dual-arm collaboration for assembly, etc., which are relatively rare in the prior art, significantly improving the accuracy and flexibility of teleoperation.

[0176] S240. Send the motion instructions to the robot equipped with the dual-arm robot arm to achieve real-time teleoperation of the dual-arm robot arm.

[0177] Among them, the robot is equipped with a dual-arm robot arm, and the robot realizes real-time teleoperation of the dual-arm robot arm according to the motion instructions. In this embodiment, an industrial-grade dual-arm robot arm with multiple joints is selected, which has high-precision motion control capabilities and good load performance, and can meet the operation requirements in different scenarios.

[0178] Based on the above steps, the motion trajectory of the operator's hand can be mapped to the robot arm to achieve real-time teleoperation.

[0179] In this embodiment, when capturing the hand motion trajectory, the Kalman filter algorithm is introduced to smooth and predict the data, effectively reducing noise interference, further improving the accuracy and stability of gesture recognition, and ensuring the smoothness and accuracy of the robot arm motion. This is an important improvement and innovation to the existing teleoperation technology.

[0180] The robot receives the motion instructions and realizes real-time teleoperation of the dual-arm robot arm according to the motion instructions.

[0181] In some embodiments, the control of the robot arm gripper is determined by the pinching and releasing of the fingers. The specific control logic is as follows:

[0182] Set the finger pinching state as C(t). When C(t) = 1, the gripper closes; when C(t) = 0, the gripper opens. The VR glasses are used to capture the pinching action of the fingers, and the host computer converts it into a gripper control signal to control the action of the robot arm gripper.

[0183] The system will also monitor the motion state and environmental information of the robotic arm in real time in the form of an independent thread. When approaching a singularity or an obstacle is detected, it will automatically adjust the motion speed and direction to achieve the functions of obstacle avoidance and singularity avoidance. At the same time, corresponding prompt information will be displayed in the VR visualization interface to remind the operator to pay attention to safety.

[0184] The robot acquires the real-time environmental information of the dual-arm robotic arm operation scene through a scene camera and sends the real-time environmental information to the host computer. Further, as Figure 7 、 Figure 11 shown, it also includes the steps:

[0185] S250. Receive the real-time environmental information of the dual-arm robotic arm operation scene sent by the robot;

[0186] The robot is also equipped with a scene camera. The scene binocular camera installed on the robotic arm can acquire the three-dimensional environmental information of the operation scene in real time and send it to the host computer; in this embodiment, a binocular camera is used, which has the shooting capabilities of high resolution and high frame rate, and can clearly capture the RGB information and depth information of the operation scene. The camera can communicate with the host computer through wired or wireless (wifi or 5G) means to achieve real-time transmission of the shooting content. The depth information assists the host computer and the robotic arm to complete obstacle avoidance and singularity avoidance operations, improving the safety and reliability of the system.

[0187] S260. Construct an environmental map according to the real-time environmental information;

[0188] S270. According to the environmental map, the current position and motion trajectory of the dual-arm robotic arm, determine in real time whether the robotic arm is approaching a singularity or an obstacle;

[0189] S280. When the robotic arm is approaching a singularity or an obstacle, adjust the motion speed and direction of the robotic arm to make it move away from the singularity or the obstacle;

[0190] Further, when the robotic arm is approaching a singularity or an obstacle, the adjustment of the motion speed of the robotic arm specifically is:

[0191] Set a safety distance threshold d threshold between the robotic arm and the singularity or the obstacle. When the distance d(t) between the robotic arm and the singularity or the obstacle is less than the safety distance threshold d threshold , adjust the motion speed v(t) of the robotic arm. The formula is:

[0192]

[0193] where v(t) is the motion speed of the robotic arm, d(t) is the distance between the robotic arm and the singularity or the obstacle, and d threshold is the safety distance threshold between the robotic arm and the singularity or the obstacle.

[0194] S290. Normally execute the subsequent program.

[0195] In the VR visualization interface, the obstacle avoidance status of the robotic arm is real-time prompted to the operator by highlighting the obstacle area, displaying the obstacle avoidance path, etc., enhancing the safety and reliability of the operation.

[0196] When the robotic arm approaches a singularity or an obstacle, the host computer automatically displays a warning message in the VR visualization interface to remind the operator to pay attention to safety. The warning message can include text prompts, highlighting the obstacle area, displaying the obstacle avoidance path, etc., ensuring that the operator can take timely measures to avoid collision accidents.

[0197] When the robotic arm encounters a singularity or approaches an obstacle, the system can automatically reduce the speed until it stops, or move away from the singularity or the obstacle, and display the corresponding prompt information in the VR visualization interface. This intelligent obstacle avoidance strategy effectively avoids the collision of the robotic arm with obstacles during operation or the loss of motion control caused by getting stuck in a singularity, ensuring the safety and reliability of the operation.

[0198] Furthermore, as Figure 8 , Figure 12 shown, it further includes the steps:

[0199] S201. Convert the real-time environment information into data in a streaming media format;

[0200] Specifically, use a binocular scene camera to capture the real-time RGB image and point cloud image of the operation scene, and convert the image data into a streaming media format supported by the HLS protocol through an encoder.

[0201] S202. Transmit the streaming media format data to the VR headset to achieve real-time display of the operation scene on the VR headset.

[0202] Specifically, adopt an efficient network transmission protocol, such as Wi-Fi or 5G network, to transmit the streaming media data to the VR glasses. During the transmission process, dynamically adjust the resolution and frame rate of the image according to the network bandwidth and latency to ensure the stability and real-time nature of the transmission.

[0203] The VR headset receives the streaming media format data, restores the streaming media format data and displays it in real time. Specifically, after receiving the streaming media data, the VR glasses restore it to an image through a decoder and display it on the display screen in real time, enabling the operator to clearly observe the operation scene, further enhancing the safety and reliability of the operation.

[0204] The scene camera captures the operation scene and streams the captured content to the VR glasses through the HLS protocol, ensuring the stability and real-time nature of the transmission.

[0205] The operation scene captured by the scene camera is displayed in real time through the VR glasses, enabling the operator to observe the operation environment immersive and improving the immersion and real-time performance of the operation. The operator can adjust the viewing angle in the VR interface by turning the head or gesture operation to observe the operation scene more comprehensively. The application of this streaming technology optimizes the efficiency and stability of data transmission, reduces operation latency, improves the accuracy and efficiency of operation, and provides a more intuitive and realistic operation experience for the operator.

[0206] The present invention provides a method for remotely operating and controlling a two-armed robotic arm based on a VR headset. This method obtains the motion trajectories of the user's hands through the VR glasses and maps these trajectories to the two-armed robotic arm to achieve high-precision remote operation. At the same time, this method also includes streaming the content captured by the scene camera (binocular camera) to the VR glasses to enhance the real-time performance and immersion of the operation. This method also designs a special intelligent obstacle avoidance strategy for the remote operation task to avoid situations where equipment or the operation object may be damaged during remote operation. This method is applicable to the interaction and task execution of general robots in industrial, household and other scenarios, and to perform precise operations on behalf of humans in high-risk environments. This design with multi-scenario adaptability makes this method have a broader application prospect and practical significance, can meet the operation requirements in different scenarios, and provides the possibility for the application of robotic technology in more fields.

[0207] Compared with traditional remote operation solutions (such as joysticks, teaching pendants, motion capture devices, etc.), the present invention only requires a VR glasses to complete the monitoring and control of the robot operation environment and its body. This design greatly simplifies the operation process, reduces the operation complexity, and is easier to use and promote.

[0208] By capturing natural gestures through the VR glasses for control, the operator can operate with both hands as naturally as in the real world without having to learn complex control commands or operation interfaces. This natural gesture control method further enhances the naturalness and intuitiveness of the operation, significantly improving the user-friendliness and market competitiveness.

[0209] Embodiment 3

[0210] A computer device 400, as Figure 13 shown, includes a memory 410, a processor 420, and a computer program 430 stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a method for remotely operating and controlling a two-armed robotic arm based on a VR headset. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiments and will not be elaborated here.

[0211] Embodiment 4

[0212] A computer-readable storage medium, asFigure 14 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, it implements the steps of a method for remotely operating and controlling a two-armed robotic arm based on a VR headset. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiments, and details will not be repeated here.

[0213] The number of devices and the scale of processing described here are used to simplify the description of the present invention. Applications, modifications, and variations of the present invention will be apparent to those skilled in the art.

[0214] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those skilled in the art, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to specific details and the illustrated and described examples here.

[0215] The device, computer device, non-volatile computer storage medium provided in the embodiments of this specification correspond to the method. Therefore, the device, computer device, and non-volatile computer storage medium also have beneficial technical effects similar to those of the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding device, computer device, and non-volatile computer storage medium will not be repeated here.

[0216] Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, the method steps can be logically programmed to enable the controller to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, such a controller can be regarded as a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software units for implementing the method and structures within the hardware component.

[0217] The systems, devices, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. For the convenience of description, when describing the above devices, they are described as various units according to functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0218] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, the embodiments of this specification can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0219] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of this specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0220] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0222] It should also be noted that the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0223] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program units may be located in local and remote computer storage media including storage devices.

[0224] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference may be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and reference may be made to the corresponding parts of the method embodiments for the relevant content.

[0225] The above description is only for the embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, various changes and modifications can be made to one or more embodiments of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A teleoperation control method for a two-arm manipulator based on a VR headset, characterized in that, It includes the following steps: Receive the user's hand gesture data sent by the VR headset; Process the gesture data to obtain the user's hand movement trajectory data; Convert the movement trajectory data into movement instructions for the dual-arm manipulator; Send the movement instructions to the robot equipped with the dual-arm manipulator to achieve real-time teleoperation of the dual-arm manipulator.

2. The method for remotely operating and controlling a two-arm robot based on a VR headset according to claim 1, wherein, The step of processing the gesture data to obtain the user's hand movement trajectory data includes: Extract the key feature points of the hand from the gesture data; Calculate the velocity vector of the hand movement trajectory by taking the difference of the hand's position in three-dimensional space. The formula is: where V(t) is the velocity vector, P(t) is the hand's position in three-dimensional space, t is the timestamp, and Δt is the time interval; Perform filtering processing on the velocity vector to obtain a smoothed velocity vector.

3. The method for remotely operating and controlling a two-arm manipulator based on a VR headset according to claim 2, wherein The step of performing filtering processing on the velocity vector to obtain a smoothed velocity vector includes: Initialize the state vector and covariance matrix of the Kalman filter; At each time step, perform iterative calculation on the state vector according to the prediction equation and update equation to obtain the smoothed movement trajectory data. The formula is: V′(t)=K(t)·Z(t)+(1-K(t))·V(t) where K(t) is the Kalman gain and Z(t) is the observed value.

4. The method for remotely operating and controlling a two-arm robotic arm based on a VR headset according to claim 2, wherein, The step of converting the movement trajectory data into movement instructions for the dual-arm manipulator includes: Set the movement model and dynamic parameters of the dual-arm manipulator, and calculate the target position and velocity of the joints according to the smoothed velocity vector; Adopt a time-based interpolation method to generate a smooth movement trajectory curve to ensure the smooth movement of the manipulator; Convert the processed data into movement instructions recognizable by the dual-arm manipulator.

5. The method for remotely operating and controlling a two-armed robotic arm based on a VR headset according to claim 4, wherein, It also includes the steps: Receive the real-time environment information of the operation scenario of the dual-arm manipulator sent by the robot; Construct an environment map according to the real-time environment information; According to the environment map, the current position and movement trajectory of the dual-arm manipulator, judge in real time whether the manipulator is approaching a singularity or an obstacle; When the manipulator is approaching a singularity or an obstacle, adjust the movement speed and direction of the manipulator to make it move away from the singularity or the obstacle.

6. The method for remotely operating and controlling a two-arm robot based on a VR headset according to claim 5, wherein The specific method of adjusting the movement speed of the manipulator when it is approaching a singularity or an obstacle is: When the distance between the manipulator and the singularity or the obstacle is less than the safety distance threshold, adjust the movement speed of the manipulator. The formula is: Among them, v(t) is the motion speed of the robotic arm, d(t) is the distance between the robotic arm and the singularity or obstacle, and d threshold is the safety distance threshold between the robotic arm and the singularity or obstacle.

7. The method for remotely operating and controlling a two-armed robotic arm based on a VR headset according to claim 5, characterized in that, It also includes the steps: Convert the real-time environment information into data in streaming media format; Transmit the data in streaming media format to the VR headset to achieve real-time display of the operation scenario on the VR headset.

8. A two-arm robotic arm teleoperation system based on a VR headset, which applies the method according to any one of claims 1 to 7, characterized in that: It includes a VR headset, a robot equipped with a dual-arm manipulator and a scene camera, and a host computer. Among them, The VR headset is used to capture the user's hand gesture data, and restore and display in real time the data in streaming media format transmitted by the host computer; The scene camera is used to collect the real-time environment information of the operation scenario of the dual-arm manipulator and send it to the host computer; The host computer is used to process the gesture data to obtain the motion trajectory data of the user's two hands, convert the motion trajectory data into motion instructions for the dual-arm manipulator, send the motion instructions to the robot, convert the environmental information into streaming media format data and transmit it to the VR headset, and determine in real time whether the manipulator is approaching a singularity or an obstacle according to the environmental information, so as to adjust the motion speed and direction of the manipulator; The robot is used to realize real-time teleoperation of the dual-arm manipulator according to the motion instructions.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images