Robotic teleoperation system and method based on visual positioning and virtual reality technology

Abstract

The application discloses a kind of robot teleoperation systems based on visual positioning and virtual reality technology, including robot located in remote space, depth camera and scene reconstruction system, robot includes robot control system and robot execution system;Virtual reality device and virtual engine located in local operating space;Scene reconstruction system generates remote robot work scene;Virtual engine is connected with virtual reality head-mounted display and handheld controller, it generates virtual robot according to actual robot size, and generates initial virtual robot work space based on the three-dimensional point cloud information of remote robot work scene and the initial pose data of robot work object;It changes virtual robot pose and virtual robot work space according to the control signal of handheld controller.The application realizes the three-dimensional reconstruction of remote environment based on point cloud and 6D pose estimation algorithm, and feeds back to operator by virtual reality device, avoids the delay of point cloud continuous transmission.

Description

Technical Field

[0001] This invention relates to a robot teleoperation system and method, and particularly to a robot teleoperation system and method based on visual positioning and virtual reality technology. Background Technology

[0002] Robot teleoperation refers to an operator remotely controlling the movement of a robot to complete tasks. Fully autonomous intelligent robots operating in unknown environments are a goal that is difficult to achieve now and for the next few years. Therefore, "human-robot interactive teleoperated robots" are one of the important development directions for robots. A "human-robot interactive teleoperated robot" means that the robot operates remotely while a human is locally positioned. Various sensors measure the human's limb movement information and use this as control commands to control the robot's movement and operation. Simultaneously, the robot's visual, force, and tactile senses are fed back to the human, achieving deep coupling between human and robot.

[0003] Currently, teleoperation technology has been widely applied in various fields with complex work requirements and potential dangers to humans, such as mine clearance and explosive ordnance disposal, search and rescue, high-altitude operations, underwater operations, and space exploration. One of the key factors for achieving intuitive and efficient teleoperation in practical applications is providing the operator with a good sense of presence. Presence refers to enabling the operator to fully perceive the remote robot environment, providing them with immersive sensory assistance. Presence can be further divided into visual presence and force / tactile presence. The importance of visual presence in teleoperation has been widely recognized and extensively studied, as visual feedback is the most crucial sensory feedback in teleoperation.

[0004] Traditional teleoperation visual feedback primarily relies on multi-angle 2D video from the remote environment. While this offers the advantage of simple, direct, and easily accessible information, its limitations include the operator's need for strong coordinate and spatial transformation capabilities when switching between different video angles, resulting in a heavy workload and low efficiency. Furthermore, the lack of depth information often leads to misperceptions of depth and distance. Additionally, 2D video-based visual feedback is prone to delays in information transmission and graphics computation. These two factors combined fail to provide the operator with a good sense of presence, significantly reducing teleoperation efficiency and impacting system safety. Summary of the Invention

[0005] This invention provides a robot teleoperation system and method based on visual positioning and virtual reality technology to solve the technical problems existing in the prior art.

[0006] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows:

[0007] A robot teleoperation system based on visual positioning and virtual reality technology includes a robot located in a remote robot workspace, the robot including a robot control system and a robot execution system; a depth camera and scene reconstruction system located in the remote robot workspace; a virtual reality device and virtual engine located in a local operating space; and a communication module for enabling communication between the device in the remote robot workspace and the device in the local operating space.

[0008] A depth camera acquires RGB and depth images of the remote robot's workspace; it then sends the acquired image data to a scene reconstruction system.

[0009] The scene reconstruction system fuses RGB images and depth images from a depth camera to generate 3D point cloud data of the remote robot's working scene. It sends the generated 3D point cloud data to the virtual engine through a communication module. The scene reconstruction system includes a 6D pose estimation model. The 6D pose estimation model is used to identify the initial pose of the robot's working object in the remote robot's working scene based on the RGB image. It sends the initial pose data of the robot's working object to the virtual engine through a communication module.

[0010] Virtual reality devices include virtual reality headsets and handheld controllers; the virtual reality headset is used to provide operators with visual information about the virtual robot's workspace; the handheld controller is used by the operator to control the virtual robot's pose.

[0011] The virtual engine connects to both the virtual reality headset and the handheld controller. It generates a virtual robot by modeling it to scale based on the actual robot size, and creates an initial virtual robot workspace based on the 3D point cloud information of the remote robot's working scene and the initial pose data of the robot's work object. It changes the virtual robot's pose and workspace according to the control signals from the handheld controller. It sends the virtual robot's pose data to the robot control system in real time and sends the virtual robot workspace data to the virtual reality headset.

[0012] The robot control system sends signals to control the robot to perform the system's work based on the virtual robot's pose data, so that the robot's pose in the remote robot workspace follows the virtual robot's pose.

[0013] Furthermore, the virtual reality device selected is the HTC VIVE Pro virtual reality device.

[0014] Furthermore, the virtual engine uses the Unity virtual engine.

[0015] Furthermore, the depth camera selected is the Realsense D435i depth camera.

[0016] The present invention also provides a robot teleoperation method based on visual positioning and virtual reality technology, utilizing the above-mentioned robot teleoperation system based on visual positioning and virtual reality technology. The method includes the following steps:

[0017] Step 1: Train the 6D pose estimation model.

[0018] Step 2: Use the eye-outside-hand extrinsic calibration method to obtain the coordinate transformation matrix N between the robot coordinate system and the depth camera coordinate system; The scene reconstruction system registers the acquired RGB image and depth image based on the depth camera intrinsic parameters, transforms the depth camera coordinate system to the robot coordinate system, generates 3D point cloud data of the remote robot working scene and sends it to the virtual engine.

[0019] Step 3: The trained 6D pose estimation model identifies the initial pose of the robot's work object based on the RGB image and multiplies it with the coordinate transformation matrix N. The initial pose data of the robot's work object after coordinate transformation is then sent to the virtual engine.

[0020] Step four, the virtual engine, generates a virtual robot workspace visible to the operator in the virtual reality headset based on 3D point cloud data from the scene reconstruction system, and matches and overlaps the 3D point cloud information of the virtual robot workspace with that of the remote robot work scene; based on the initial pose data of the robot work object from the 6D pose estimation model, it replaces the point cloud data of the robot work object in the virtual robot workspace with the virtual work object, while retaining the point cloud of the stationary object, thus completing the 3D reconstruction of the remote robot work scene.

[0021] Step 5: After the virtual engine completes the 3D reconstruction, the operator controls the virtual robot to interact with the virtual work object in real time through a handheld controller. The virtual engine generates the motion trajectory of the virtual robot based on the control signals from the handheld controller, and changes the virtual robot's pose and workspace accordingly. It sends the virtual robot's pose data to the robot control system in real time and sends the virtual robot's workspace data to the virtual reality headset.

[0022] Step six: The robot control system sends signals to control the robot to perform the system's work based on the virtual robot pose data, so that the robot pose in the remote robot workspace follows the virtual robot pose.

[0023] Furthermore, the 6D pose estimation model is constructed using the Gen6D pose estimation algorithm. Step one includes the following sub-steps:

[0024] Step 1-1: Use a depth camera to capture a video around the robot's work object as a reference sequence.

[0025] Steps 1-2 involve splitting the reference sequence into images.

[0026] Steps 1-3 involve recovering the pose of each reference image using COLMAP SfM, and simultaneously reconstructing the sparse point cloud of the unknown object to define its size.

[0027] Steps 1-4 define the coordinate system of the reconstructed point cloud.

[0028] Steps 1-5: Run the Gen6D pose estimation algorithm program to estimate the 6D pose of the robot's work object based on the pre-trained weights.

[0029] Furthermore, in step two, the camera coordinate system T is set. Realsense Calibration plate coordinate system T Board and robot base coordinate system T Robot If the calibration plate is fixed to the robot's end effector, then the matrix from the calibration plate to the robot's base coordinate system is... Given that the transformation matrix from the camera coordinate system to the calibration board is determined through a recognition program, the process can be accelerated. The transformation matrix N from the depth camera coordinate system to the robot coordinate system is:

[0030] Furthermore, in step five, the handheld controller simulates the robot's end effector, and the handheld controller's end effector pose maps to the virtual robot's end effector pose. The method for the virtual engine to generate the virtual robot's motion trajectory based on the control signals from the handheld controller includes the following steps:

[0031] Let V t+1 With V t This corresponds to the end-effector pose of the virtual robot at frames t+1 and t, U t+1 with U t The corresponding handheld controller end-effector poses are at frames t+1 and t, where k is the mapping ratio between the handheld controller pose and the virtual robot end-effector pose, such that:

[0032] V t+1 =k·(U t+1 -U t )+V t

[0033] According to robot inverse kinematics, V t+1 This is converted into the angles of each joint of the virtual robot, enabling the end effector of the virtual robot to move proportionally in accordance with the handheld controller.

[0034] Furthermore, taking the virtual work object as the center, two spheres with radii r0 and r1 are set to divide the virtual robot's workspace into three workspaces, named S1, S2, and S3 from the outside to the inside. For each of the three workspaces, an adaptive position incremental mapping algorithm is used to adjust k so that the mapping ratio coefficients k correspond to k1, k2, and k3 respectively. The calculation formula for the adaptive position incremental mapping algorithm is as follows:

[0035]

[0036] In the formula: r is the distance from the end effector of the virtual robot to the virtual work object, c1 and c2 are constants less than 1, and c2 < c1.

[0037] Furthermore, r0 and r1 are determined by the size of the target, where r0 is the radius of the minimum spherical bounding box of the virtual work object, and r1 is 1.5 to 2 times the radius of the minimum spherical bounding box of the virtual work object. The method for the operator to control the virtual robot to interact with the virtual work object in real time through a handheld controller includes the following steps:

[0038] When the operator presses the start button on the handheld controller, the end-effector pose V of the virtual robot in that frame is recorded. t and the pose U of the handheld controller t When there are no interactive virtual work objects in the virtual environment, the mapping ratio is k1; when there are interactive virtual work objects in the virtual environment, the virtual space where the virtual robot's end effector is located is determined. If it is in space S1, the mapping ratio is k1; if it is in space S2, the mapping ratio is k2; if it is in space S3, the mapping ratio is k3. Subsequently, the operator moves the handheld controller, and the virtual engine calculates the end effector pose V of the virtual robot in frame t+1 based on the pose of the handheld controller. t+1 This allows the virtual robot to follow the movement of the handheld controller.

[0039] The advantages and positive effects of this invention are as follows: Based on point cloud and 6D pose estimation algorithms, this invention achieves 3D reconstruction of a remote environment and provides feedback to the operator through a virtual reality device, enhancing the operator's sense of presence, avoiding the pressure of continuous point cloud transmission, and improving adaptability to unstructured environments. The operator controls the virtual robot and interacts with virtual objects through a handheld controller, achieving truly latency-free remote operation. The adaptive position incremental mapping algorithm based on virtual reality space provided by this invention improves operational efficiency and accuracy, and endows the virtual reality teleoperation system with obstacle avoidance capabilities, thereby enhancing the safety of the virtual reality teleoperation system. Attached Figure Description

[0040] Figure 1This is a schematic diagram of a robot teleoperation system based on visual positioning and virtual reality technology according to the present invention.

[0041] Figure 2 This is a schematic diagram of the workflow of a scene reconstruction system in a robot teleoperation system based on visual positioning and virtual reality technology according to the present invention.

[0042] Figure 3 This is a schematic diagram of the virtual robot workspace in the virtual reality scene of this invention.

[0043] Figure 4 This is a schematic diagram of the adaptive position increment mapping process based on virtual reality space according to the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0045] In the description of this invention, the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," and "bottom," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and do not require the invention to be constructed and operated in a specific orientation; therefore, they should not be construed as limitations on the invention. The terms "connected" and "linked" used in this invention should be interpreted broadly. For example, they can refer to a fixed connection or a detachable connection; they can refer to a direct connection or an indirect connection through intermediate components. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0046] Please see Figures 1 to 4 A robot teleoperation system based on visual positioning and virtual reality technology includes a robot located in a remote robot workspace, the robot including a robot control system and a robot execution system; a depth camera and scene reconstruction system located in the remote robot workspace; a virtual reality device and virtual engine located in a local operating space; and a communication module that enables communication between the device in the remote robot workspace and the device in the local operating space.

[0047] A depth camera acquires RGB and depth images of the remote robot's workspace; it then sends the acquired image data to a scene reconstruction system.

[0048] The scene reconstruction system fuses RGB images and depth images from a depth camera to generate 3D point cloud data of the remote robot's working scene. It sends the generated 3D point cloud data to the virtual engine through a communication module. The scene reconstruction system includes a 6D pose estimation model. The 6D pose estimation model is used to identify the initial pose of the robot's working object in the remote robot's working scene based on the RGB image. It sends the initial pose data of the robot's working object to the virtual engine through a communication module.

[0049] Virtual reality devices include virtual reality headsets and handheld controllers; the virtual reality headset is used to provide operators with visual information about the virtual robot's workspace; the handheld controller is used by the operator to control the virtual robot's pose.

[0050] The virtual engine connects to both the virtual reality headset and the handheld controller. It generates a virtual robot by modeling it to scale based on the actual robot size, and creates an initial virtual robot workspace based on the 3D point cloud information of the remote robot's working scene and the initial pose data of the robot's work object. It changes the virtual robot's pose and workspace according to the control signals from the handheld controller. It sends the virtual robot's pose data to the robot control system in real time and sends the virtual robot workspace data to the virtual reality headset.

[0051] The robot control system sends signals to control the robot to perform the system's work based on the virtual robot's pose data, so that the robot's pose in the remote robot workspace follows the virtual robot's pose.

[0052] Preferably, the virtual reality device can be the HTC VIVE Pro virtual reality device.

[0053] Preferably, the virtual engine can be the Unity virtual engine.

[0054] Preferably, the depth camera can be a Realsense D435i depth camera.

[0055] Preferably, the robot can be an R-Sv5 serial collaborative robot, and the robot motion controller can be a Power PMAC motion controller from DeltaTau.

[0056] The present invention also provides an embodiment of a robot teleoperation method based on visual positioning and virtual reality technology using the robot teleoperation system based on visual positioning and virtual reality technology as described above, the method comprising the following steps:

[0057] Step 1: Train the 6D pose estimation model.

[0058] Step 2: Use the eye-outside-hand extrinsic calibration method to obtain the coordinate transformation matrix N between the robot coordinate system and the depth camera coordinate system; The scene reconstruction system registers the acquired RGB image and depth image based on the depth camera intrinsic parameters, transforms the depth camera coordinate system to the robot coordinate system, generates 3D point cloud data of the remote robot working scene and sends it to the virtual engine.

[0059] Step 3: The trained 6D pose estimation model identifies the initial pose of the robot's work object based on the RGB image and multiplies it with the coordinate transformation matrix N. The initial pose data of the robot's work object after coordinate transformation is then sent to the virtual engine.

[0060] Step four, the virtual engine, generates a virtual robot workspace visible to the operator in the virtual reality headset based on 3D point cloud data from the scene reconstruction system, and matches and overlaps the 3D point cloud information of the virtual robot workspace with that of the remote robot work scene; based on the initial pose data of the robot work object from the 6D pose estimation model, it replaces the point cloud data of the robot work object in the virtual robot workspace with the virtual work object, while retaining the point cloud of the stationary object, thus completing the 3D reconstruction of the remote robot work scene.

[0061] Step 5: After the virtual engine completes the 3D reconstruction, the operator controls the virtual robot to interact with the virtual work object in real time through a handheld controller. The virtual engine generates the motion trajectory of the virtual robot based on the control signals from the handheld controller, and changes the virtual robot's pose and workspace accordingly. It sends the virtual robot's pose data to the robot control system in real time and sends the virtual robot's workspace data to the virtual reality headset.

[0062] Step six: The robot control system sends signals to control the robot to perform the system's work based on the virtual robot pose data, so that the robot pose in the remote robot workspace follows the virtual robot pose.

[0063] Preferably, the 6D pose estimation model can be constructed using the Gen6D pose estimation algorithm, and step one may include the following sub-steps:

[0064] Step 1-1: Use a depth camera to capture a video around the robot's work object as a reference sequence.

[0065] Steps 1-2 involve splitting the reference sequence into images.

[0066] Steps 1-3 involve recovering the pose of each reference image using COLMAP SfM, and simultaneously reconstructing the sparse point cloud of the unknown object to define its size.

[0067] Steps 1-4 define the coordinate system of the reconstructed point cloud.

[0068] Steps 1-5: Run the Gen6D pose estimation algorithm program to estimate the 6D pose of the robot's work object based on the pre-trained weights.

[0069] Preferably, in step two, the camera coordinate system T can be set. Realsense Calibration plate coordinate system T Board and robot base coordinate system T Robot The calibration plate can be fixed to the robot's end effector, then the matrix from the calibration plate to the robot's base coordinate system... Given that the transformation matrix from the camera coordinate system to the calibration board is determined through a recognition program, the process can be accelerated. The transformation matrix N from the depth camera coordinate system to the robot coordinate system can be:

[0070]

[0071] Preferably, in step five, the handheld controller can simulate the robot's end effector, and the handheld controller's end effector pose maps to the virtual robot's end effector pose. The method for the virtual engine to generate the virtual robot's motion trajectory based on the control signals from the handheld controller may include the following steps:

[0072] V can be set t+1 With V t This corresponds to the end-effector pose of the virtual robot at frames t+1 and t, U t+1 with U t This corresponds to the end effector pose of the handheld controller at frame t+1 and frame t, where k is the mapping ratio between the handheld controller pose and the virtual robot end effector pose, which allows:

[0073] V t+1 =k·(U t+1 -U t )+V t

[0074] Based on the robot's inverse kinematics, V t+1 This is converted into the angles of each joint of the virtual robot, enabling the end effector of the virtual robot to move proportionally in accordance with the handheld controller.

[0075] Preferably, with the virtual work object as the center, two spheres with radii r0 and r1 can be set to divide the virtual robot's workspace into three workspaces. These three workspaces can be named S1, S2, and S3 from the outside to the inside. For each of the three workspaces, an adaptive position incremental mapping algorithm can be used to adjust k so that the mapping ratio coefficients k correspond to k1, k2, and k3, respectively. The calculation formula for the adaptive position incremental mapping algorithm is as follows:

[0076]

[0077] In the formula: r is the distance from the end effector of the virtual robot to the virtual work object, c1 and c2 are constants less than 1, and c2 < c1.

[0078] Preferably, r0 and r1 are determined by the size of the target, where r0 is the radius of the minimum spherical bounding box of the virtual task object, and r1 is 1.5 to 2 times the radius of the minimum spherical bounding box of the virtual task object, preferably 1.7 times.

[0079] A method for an operator to control a virtual robot to interact with a virtual work object in real time via a handheld controller may include the following steps:

[0080] When the operator presses the start button on the handheld controller, the end-effector pose V of the virtual robot in that frame is recorded. t and the pose U of the handheld controller t When there are no interactive virtual work objects in the virtual environment, the mapping ratio is k1; when there are interactive virtual work objects in the virtual environment, the virtual space where the virtual robot's end effector is located is determined. If it is in space S1, the mapping ratio is k1; if it is in space S2, the mapping ratio is k2; if it is in space S3, the mapping ratio is k3. Subsequently, the operator moves the handheld controller, and the virtual engine calculates the end effector pose V of the virtual robot in frame t+1 based on the pose of the handheld controller. t+1 This allows the virtual robot to follow the movement of the handheld controller.

[0081] The results, workflow, and working principle of the present invention are further illustrated below with a preferred embodiment of the invention:

[0082] like Figure 1 As shown, a robot teleoperation system based on visual positioning and virtual reality technology includes a robot located in a remote robot workspace, the robot including a robot control system and a robot execution system; it also includes a depth camera and scene reconstruction system located in the remote robot workspace; a virtual reality device and virtual engine located in the local operating space; and a communication module that enables communication between the device in the remote robot workspace and the device in the local operating space.

[0083] A depth camera acquires RGB and depth images of the remote robot's workspace; it then sends the acquired image data to a scene reconstruction system.

[0084] The scene reconstruction system fuses RGB images and depth images from a depth camera to generate 3D point cloud data of the remote robot's working scene. It sends the generated 3D point cloud data to the virtual engine through a communication module. The scene reconstruction system includes a 6D pose estimation model. The 6D pose estimation model is used to identify the initial pose of the robot's working object in the remote robot's working scene based on the RGB image. It sends the initial pose data of the robot's working object to the virtual engine through a communication module.

[0085] Virtual reality devices include virtual reality headsets and handheld controllers; the virtual reality headset is used to provide operators with visual information about the virtual robot's workspace; the handheld controller is used by the operator to control the virtual robot's pose.

[0086] The virtual engine connects to both the virtual reality headset and the handheld controller. It generates a virtual robot by modeling it to scale based on the actual robot size, and creates an initial virtual robot workspace based on the 3D point cloud information of the remote robot's working scene and the initial pose data of the robot's work object. It changes the virtual robot's pose and workspace according to the control signals from the handheld controller. It sends the virtual robot's pose data to the robot control system in real time and sends the virtual robot workspace data to the virtual reality headset.

[0087] The robot control system sends signals to control the robot to perform the system's work based on the virtual robot's pose data, so that the robot's pose in the remote robot workspace follows the virtual robot's pose.

[0088] The virtual reality device chosen is the HTC VIVE Pro. The HTC VIVE Pro is a virtual reality headset jointly developed by HTC and Valve Corporation.

[0089] The virtual engine uses the Unity Virtual Engine. The Unity Engine is a cross-platform game engine developed by Unity Technologies.

[0090] The depth camera used is the Realsense D435i. The Realsense D435i is a consumer-grade depth camera from Intel. It mainly consists of an RGB camera, two infrared cameras, an infrared transmitter, and an IMU unit.

[0091] The 6D pose estimation model adopts the Gen6D pose estimation algorithm, which is used to estimate the pose of the robot's working object. Gen6D has three advantages over other pose estimation algorithms: 1) Gen6D does not require knowledge of the object's CAD model or renderable model; it only needs to capture a video around the object to estimate its pose. 2) It is generalizable; given a new object, it can predict the object's pose in an image without needing to train on the new object. 3) It only requires RGB images; Gen6D only needs RGB images and does not require knowledge of the object's mask or depth for pose estimation.

[0092] The robot selected is the R-Sv5 serial collaborative robot, and the robot control system uses the PowerPMAC motion controller from Delta Tau.

[0093] The robot's end effector is equipped with a mechanical gripper, which is used to grasp the object the robot is working on. The PowerPMAC motion controller transmits data with the Unity virtual engine through a TCP communication module or a wireless communication module.

[0094] The Unity Virtual Engine also connects to collaborative robots and scene reconstruction systems via TCP or wireless communication modules.

[0095] like Figure 2 As shown, a preferred embodiment of a robot teleoperation method based on visual positioning and virtual reality technology, utilizing the aforementioned robot teleoperation system, includes the following steps:

[0096] Step 1: Train the Gen6D pose estimation model;

[0097] Step two: To accurately locate the point cloud model in the Unity virtual engine so that the virtual scene seen by the operator matches the real-world scene, we need to obtain the transformation matrix between the robot's base coordinate system and the RealSense camera coordinate system. We use the "eye-outside-hand" extrinsic calibration method to find the relative pose transformation between the robot and RealSense. We then use this method to obtain the coordinate transformation matrix N between the robot coordinate system and the depth camera coordinate system. The scene reconstruction system registers the acquired RGB and depth images based on the depth camera's intrinsic parameters, transforms the depth camera coordinate system to the robot coordinate system, generates 3D point cloud data of the remote robot's working scene, and sends it to the virtual engine.

[0098] Step 3: The trained 6D pose estimation model identifies the initial pose of the robot's work object based on the RGB image and multiplies it with the coordinate transformation matrix N. The initial pose data of the robot's work object after coordinate transformation is then sent to the virtual engine.

[0099] Step four involves the virtual engine, which generates a virtual robot workspace visible to the operator in a virtual reality headset based on 3D point cloud data from the scene reconstruction system. This virtual robot workspace matches and overlaps with the 3D point cloud information of the remote robot's work scene. Based on the initial pose data of the robot's work objects from the 6D pose estimation model, it replaces the point cloud data of the robot's work objects in the virtual robot workspace with the virtual work objects, while retaining the point clouds of stationary objects, thus completing the 3D reconstruction of the remote robot's work scene. The Unity virtual engine uses mesh filter and mesh renderer components to visualize the point cloud. The operator can see the 3D point cloud of the remote robot's work scene in the virtual reality headset, and the 3D point cloud of the virtual robot workspace should completely match and overlap with the 3D point cloud of the remote robot's work scene.

[0100] Step five: After completing the 3D reconstruction, the virtual engine no longer transmits point cloud data in real time. Instead, it uses the Unity virtual engine's built-in physics engine to simulate and predict the motion trajectory of the replaced virtual object. The operator controls the virtual robot to interact with the virtual work object in real time using a handheld controller. The virtual engine generates the virtual robot's motion trajectory based on the control signals from the handheld controller and accordingly changes the virtual robot's pose and workspace. It sends the virtual robot's pose data to the robot control system in real time and sends the virtual robot's workspace data to the virtual reality headset.

[0101] Step six: The robot control system sends signals to control the robot to perform the system's work based on the virtual robot pose data, so that the robot pose in the remote robot workspace follows the virtual robot pose.

[0102] By adopting the above system, the burden of continuous point cloud transmission is avoided, and the advantages of the Unity physics engine are fully utilized. Operators can control virtual robots and virtual objects to interact in real time through handheld controllers, realizing truly immersive, time-delay-free remote operation.

[0103] In step five, the handheld controller simulates the robot's end effector, and the handheld controller's end effector pose maps to the virtual robot's end effector pose. The method for the virtual engine to generate the virtual robot's motion trajectory based on the control signals from the handheld controller includes the following steps:

[0104] Let V t+1 With V tThis corresponds to the end-effector pose of the virtual robot at frames t+1 and t, U t+1 with U t The corresponding handheld controller end-effector poses are at frames t+1 and t, where k is the mapping ratio between the handheld controller pose and the virtual robot end-effector pose, such that:

[0105] V t+1 =k·(U t+1 -U t )+V t

[0106] According to robot inverse kinematics, V t+1 This is converted into the angles of each joint of a virtual robot, and the V-shape is then calculated based on the robot inverse kinematics program written in the Unity virtual engine. t+1 This is converted into joint angles θ = [θ1, θ2, θ3, θ4, θ5, θ6], enabling the virtual robot's end effector to move proportionally to follow the handheld controller.

[0107] like Figure 3 As shown, with the virtual object as the center, two spheres with radii r0 and r1 are set to divide the virtual robot's workspace into three workspaces, named S1, S2, and S3 respectively.

[0108] Centered on the virtual work object, two spheres with radii r0 and r1 are used to divide the virtual robot's workspace into three workspaces, named S1, S2, and S3 from the outside in. An adaptive position incremental mapping algorithm is used to adjust k so that the mapping ratio coefficients k correspond to k1, k2, and k3 respectively. The calculation formula for the adaptive position incremental mapping algorithm is as follows:

[0109]

[0110] In the formula: r is the distance from the virtual robot's end effector to the virtual work object, c1 and c2 are constants less than 1, and c2 < c1.

[0111] c1 = 0.8, c2 = 0.3.

[0112] r0 and r1 are determined by the size of the target. r0 is the radius of the minimum spherical bounding box of the virtual task object, and r1 is 1.7 times the radius of the minimum spherical bounding box of the virtual task object.

[0113] S1, S2, and S3 can represent three safety levels of distance from the virtual robot's end effector to the virtual work object. S1 is the safest, S2 is the next safest, and S3 is the least safe. The smaller the distance from the robot's end effector to the virtual work object, the easier it is to collide with the virtual work object. Therefore, k should be adjusted using an adaptive position incremental mapping algorithm to correspond to the three workspaces of S1, S2, and S3.

[0114] The method for an operator to control a virtual robot to interact with a virtual work object in real time via a handheld controller includes the following steps:

[0115] When the operator presses the start button on the handheld controller, the end-effector pose V of the virtual robot in that frame is recorded. t and the pose U of the handheld controller t When there are no interactive virtual work objects in the virtual environment, the mapping ratio is k1; when there are interactive virtual work objects in the virtual environment, the virtual space where the virtual robot's end effector is located is determined. If it is in space S1, the mapping ratio is k1; if it is in space S2, the mapping ratio is k2; if it is in space S3, the mapping ratio is k3. Subsequently, the operator moves the handheld controller, and the virtual engine calculates the end effector pose V of the virtual robot in frame t+1 based on the pose of the handheld controller. t+1 This allows the virtual robot to follow the movement of the handheld controller.

[0116] Because if the scaling factor fluctuates wildly within the same task phase, the operator's experience will be poor. In practical applications, the process of the robot approaching the target object and interacting with the environment requires consideration not only of the motion accuracy of teleoperation control, but also of the safety of the teleoperation process. S3 is similar to a repulsive field in an artificial potential field. When the virtual robot's end effector enters the S3 workspace, it means that the end effector is too close to the target object, posing a risk of collision. Therefore, the "repulsive force" of S3 will drive k3 to continuously decrease until it reaches 0, preventing collision with the target object and enabling the operator to perform teleoperation tasks more safely. The adaptive position incremental mapping algorithm based on virtual reality space provided by this invention improves operational efficiency and accuracy, and endows the virtual reality teleoperation system with obstacle avoidance capabilities.

[0117] The above embodiments are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The patent scope of the present invention should not be limited by these embodiments. That is, all equivalent changes or modifications made in accordance with the spirit of the present invention still fall within the patent scope of the present invention.

Claims

1. A robot teleoperation system based on visual positioning and virtual reality technology, comprising a robot located in a remote robot workspace, the robot including a robot control system and a robot execution system; characterized in that, It also includes a depth camera and scene reconstruction system located in a remote robotic workspace; a virtual reality device and virtual engine located in a local operating space; and a communication module that enables communication between the device in the remote robotic workspace and the device in the local operating space. A depth camera acquires RGB and depth images of the remote robot's workspace; it then sends the acquired image data to a scene reconstruction system. The scene reconstruction system fuses RGB images and depth images from a depth camera to generate 3D point cloud data of the remote robot's working scene. It then sends the generated 3D point cloud data to the virtual engine via a communication module. The scene reconstruction system includes a 6D pose estimation model, which is used to identify the initial pose of the robot's working object in the remote robot's working scene based on the RGB images. It sends the initial pose data of the robot's work object to the virtual engine through the communication module; Virtual reality devices include virtual reality headsets and handheld controllers; virtual reality headsets are used to provide operators with visual information about a virtual robotic workspace; The handheld controller is used by the operator to control the pose of the virtual robot; The virtual engine connects to both the virtual reality headset and the handheld controller. It generates a virtual robot by modeling it to scale based on the actual robot size, and creates an initial virtual robot workspace based on the 3D point cloud information of the remote robot's working scene and the initial pose data of the robot's work object. According to the control signals from the handheld controller, it changes the virtual robot's pose and virtual robot workspace accordingly, and generates the virtual robot's motion trajectory. It sends the virtual robot's pose data to the robot control system in real time, and sends the virtual robot workspace data to the virtual reality headset. Based on the initial pose data of the robot working object from the 6D pose estimation model, it replaces the point cloud data of the robot working object in the virtual robot workspace with the virtual working object, while retaining the point cloud of the stationary object, thus completing the 3D reconstruction of the remote robot working scene. The robot control system sends signals to control the robot to perform the system's work based on the virtual robot's pose data, so that the robot's pose in the remote robot's workspace follows the virtual robot's pose. The handheld controller simulates the robot's end effector, and the handheld controller's end effector pose maps to the virtual robot's end effector pose. The method for the virtual engine to generate the virtual robot's motion trajectory based on the control signals from the handheld controller includes the following steps: set up and The corresponding end-effector poses of the virtual robot are at frame t+1 and frame t. and The corresponding end-effector poses of the handheld controller at frame t+1 and frame t are... To establish the mapping ratio between the handheld controller pose and the virtual robot end effector pose, such that: ； According to robot inverse kinematics, This is converted into the angles of each joint of the virtual robot, enabling the end effector of the virtual robot to move proportionally following the handheld controller. Centered on the virtual task object, set the radius as follows: , The two spheres divide the virtual robot's workspace into three workspaces, which are named from the outside in as follows: ,correspond Three workspaces are adjusted using an adaptive position incremental mapping algorithm. Make the mapping scaling factor They correspond to as The calculation formula for the adaptive position increment mapping algorithm is as follows: ； In the formula: This refers to the distance from the virtual robot's end effector to the virtual work object. A constant less than 1 ; , It depends on the size of the target. Let be the radius of the smallest spherical bounding box of the virtual task object. It is 1.5 to 2 times the radius of the smallest spherical bounding box of the virtual task object.

2. The robot teleoperation system based on visual positioning and virtual reality technology according to claim 1, characterized in that, The virtual reality device selected is the HTC VIVE Pro.

3. The robot teleoperation system based on visual positioning and virtual reality technology according to claim 1, characterized in that, The virtual engine uses the Unity virtual engine.

4. The robot teleoperation system based on visual positioning and virtual reality technology according to claim 1, characterized in that, The depth camera used is the Realsense D435i.

5. A robot teleoperation method based on visual positioning and virtual reality technology using a robot teleoperation system based on any one of claims 1 to 4, characterized in that, The method includes the following steps: Step 1: Train the 6D pose estimation model; Step 2: Use the eye-outside-hand extrinsic calibration method to obtain the coordinate transformation matrix N between the robot coordinate system and the depth camera coordinate system; The scene reconstruction system registers the acquired RGB image and depth image based on the depth camera intrinsic parameters, transforms the depth camera coordinate system to the robot coordinate system, generates 3D point cloud data of the remote robot working scene, and sends it to the virtual engine. Step 3: The trained 6D pose estimation model identifies the initial pose of the robot object based on the RGB image and multiplies it with the coordinate transformation matrix N. The initial pose data of the robot object after coordinate transformation is then sent to the virtual engine. Step four, the virtual engine, generates a virtual robot workspace visible to the operator in the virtual reality headset based on 3D point cloud data from the scene reconstruction system, and matches and overlaps the 3D point cloud information of the virtual robot workspace with the remote robot work scene; based on the initial pose data of the robot work object from the 6D pose estimation model, it replaces the point cloud data of the robot work object in the virtual robot workspace with the virtual work object, while retaining the point cloud of the stationary object, thus completing the 3D reconstruction of the remote robot work scene. Step 5: After the virtual engine completes the 3D reconstruction, the operator controls the virtual robot to interact with the virtual work object in real time through the handheld controller. The virtual engine generates the motion trajectory of the virtual robot according to the control signal of the handheld controller, and changes the virtual robot pose and virtual robot workspace accordingly. It sends the virtual robot pose data to the robot control system in real time and sends the virtual robot workspace data to the virtual reality headset. Step six: The robot control system sends signals to control the robot to perform the system's work based on the virtual robot pose data, so that the robot pose in the remote robot workspace follows the virtual robot pose.

6. The robot teleoperation method based on visual positioning and virtual reality technology according to claim 5, characterized in that, The 6D pose estimation model is constructed using the Gen6D pose estimation algorithm. Step one includes the following sub-steps: Step 1-1: Use a depth camera to capture a video around the robot's work object as a reference sequence; Steps 1-2: Split the reference sequence into images; Steps 1-3: The pose of each reference image is recovered using COLMAP SfM, and the sparse point cloud of the unknown object is reconstructed to define the size of the object. Steps 1-4: Define the coordinate system of the reconstructed point cloud; Steps 1-5: Run the Gen6D pose estimation algorithm program to estimate the 6D pose of the robot's work object based on the pre-trained weights.

7. The robot teleoperation method based on visual positioning and virtual reality technology according to claim 5, characterized in that, In step two, the camera coordinate system is set. Calibration plate coordinate system and robot base coordinate system If the calibration plate is fixed to the robot's end effector, then the matrix from the calibration plate to the robot's base coordinate system is... Given that the transformation matrix from the camera coordinate system to the calibration board is determined through a recognition program, the process can be accelerated. The transformation matrix N from the depth camera coordinate system to the robot coordinate system is: 。 8. The robot teleoperation method based on visual positioning and virtual reality technology according to claim 5, characterized in that, The method for an operator to control a virtual robot to interact with a virtual work object in real time via a handheld controller includes the following steps: When the operator presses the start button on the handheld controller, the end-effector pose of the virtual robot in that frame is recorded. and the pose of the handheld controller When there are no interactive virtual job objects in the virtual environment, the mapping ratio is... When interactive virtual work objects exist in the virtual environment, determine the virtual space where the virtual robot's end effector is located. If it is in the space... The mapping ratio is If in space The mapping ratio is If in space The mapping ratio is ; The operator then moves the handheld controller, and the virtual engine calculates the end-effector pose of the virtual robot in frame t+1 based on the pose of the handheld controller. This allows the virtual robot to follow the movement of the handheld controller.

Images