Robot teleoperation method based on fusion of virtual reality and mixed reality

Abstract

The invention discloses a robot teleoperation method based on fusion of virtual reality and mixed reality. The robot teleoperation method comprises the following steps: sensing a far-end environment by observing a robot; constructing a near-end interactive virtual environment, and presenting the virtual environment to a near-end operator through a virtual reality device; constructing three-dimensional point cloud data based on a far-end environment through three-dimensional reconstruction; in the virtual environment, pose estimation is carried out on the far-end operation robot and the target object, and three-dimensional registration is carried out in the virtual environment; working space boundary condition constraints and space operation pose change constraints of the working robot are determined, path planning and trajectory planning are conducted on the working robot, and visualization is conducted in the virtual environment; and generating a control program of the operation robot to move along the planned track. By constructing the interactive virtual environment, the working space range and the operation pose of the operation robot are visualized in the interactive virtual environment, and the trajectory is planned and visualized in the interactive virtual environment, so that the safety of teleoperation is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of robot teleoperation technology, and more specifically to a robot teleoperation method based on the integration of virtual reality and mixed reality. Background Technology

[0002] Robotic teleoperation is a special human-robot collaborative mode that requires operators to remotely control robots to perform tasks in complex and hazardous environments. It can be applied to many high-risk operational fields such as nuclear energy maintenance, deep space exploration, and hazardous materials processing. Taking explosives production as an example, its production process involves many complex and stringent procedures such as nitration and drying. The raw materials themselves are highly sensitive and toxic, and are extremely prone to violent combustion and explosion accidents due to factors such as runaway reactions, mechanical friction, and static sparks. Such tasks are not only difficult and dangerous, but also require extremely high precision, placing higher demands on robotic teleoperation technology. However, during robotic teleoperation, there is a natural physical isolation between the operator and the work environment. The operator lacks on-site perception of the remote work environment and necessary information assistance, making it difficult to make timely judgments and decisions regarding the task status. This greatly affects the completion quality and operational safety of the teleoperation task.

[0003] Virtual reality (VR) technology, with its powerful environment-building capabilities, has immense application potential in the field of robotic teleoperation. However, as an emerging technology, VR still has many shortcomings. The operating environment in VR is typically constructed from virtual models, which struggle to accurately reflect the characteristics and real-time status of the actual teleoperation scenario. This makes it difficult for operators to precisely perceive the teleoperation environment. Furthermore, current VR technology focuses on reconstructing the environment while neglecting to extract task-related auxiliary information from it. This auxiliary information is crucial for operators to complete tasks accurately and efficiently.

[0004] Mixed reality, as a visualization and information enhancement technology, has significant application value in providing auxiliary information to operators. Defined as a technology that seamlessly integrates the virtual and real worlds, it can often be seen as a deeper extension of augmented reality. Mixed reality technology overlays computer-generated 3D models, dynamic data, and other virtual information onto physical objects in the real environment in real time, providing operators with more accurate environmental perception and more comprehensive information assistance. This allows users to interact and operate naturally in a space that blends virtual and reality. However, mixed reality requires operators and robots to be in the same manufacturing environment, while in remote robot operation scenarios, physical isolation between humans and robots is unavoidable. Therefore, mixed reality technology is difficult to directly apply to remote operation scenarios.

[0005] While virtual reality technology can visualize distant scenes, it lacks important auxiliary information and struggles to reflect the real-time production status in actual scenarios. Mixed reality technology, on the other hand, can directly map auxiliary information into actual scenarios, but it requires operators to be physically present in the current production environment, thus failing to meet the requirements for remote operation.

[0006] Therefore, providing a robot teleoperation method based on the integration of virtual reality and mixed reality is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a robot teleoperation method based on the integration of virtual reality and mixed reality. It combines the advantages of virtual reality in terms of on-site perception with the advantages of mixed reality in terms of auxiliary information enhancement. An interactive virtual environment is constructed on the host device, in which the working space range and operation posture of the robot are visualized, the planned trajectory is visualized, and the safety of teleoperation is greatly improved. It can meet the needs of robot teleoperation in many special scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a robot teleoperation method based on the fusion of virtual reality and mixed reality, comprising the following steps: S1. The robot perceives the remote environment; an interactive virtual environment located at the near end is constructed, and the virtual environment is presented to the near-end operator through a virtual reality device; three-dimensional point cloud data based on the remote environment is constructed through three-dimensional reconstruction. S2. In the interactive virtual environment, the pose of the remote operation robot and the target object is estimated, and the virtual models of the operation robot and the target object are registered in three dimensions in the virtual environment. S3. Determine the workspace boundary condition constraints and spatial operation pose change constraints of the robot, and visualize the boundary condition constraints and pose change constraints in the virtual environment; S4. Perform path planning and trajectory planning on the robot, and visualize the planned path and trajectory in the virtual environment; S5. Based on the perceived remote environment and the planned path and trajectory, generate a collision-free motion offline control program for the work robot, and control the work robot to move along the planned trajectory.

[0010] Further, step S1 specifically includes: By observing the distant environment through the binocular vision system on the robot, left and right eye images containing scene parallax and depth information are obtained. The computer processing unit renders the left and right images onto the left and right displays of the virtual reality headset through two rendering channels, presenting a three-dimensional interactive virtual environment to the operator. The computer processing unit performs three-dimensional reconstruction on the observed left and right eye images to obtain three-dimensional point cloud data based on the remote environment.

[0011] Furthermore, when observing the distant environment through the binocular vision system mounted on the observation robot, the formula for calculating the depth information of the image is as follows:

[0012] in, Representing a spatial point Q Depth value in camera coordinate system and Representing spatial points Q The horizontal coordinate of the left and right camera image coordinate systems, Indicates the camera's focal length; Indicates the baseline distance.

[0013] Furthermore, step S2 specifically includes: For remote operation robots, the pose estimation method based on ArUco markers is used to calculate the pose of the robot in the base coordinate system, and the virtual model of the robot is registered in three dimensions in the virtual environment. For the target object, a pose estimation method based on rendering comparison is adopted, combined with a residual neural network model, to estimate the pose of the target object; by rendering the 3D model of the target object from different perspectives, the rendered image and the actual scene image are input into the residual neural network model for comparison and learning, thereby realizing the pose estimation of the target object, and the virtual model of the target object is registered in the virtual environment.

[0014] Furthermore, in step S3, the process of determining and visualizing the workspace boundary condition constraints of the robot includes: The random pose of the end effector of the robot in the joint space is generated by Monte Carlo random sampling method, and then transformed into Cartesian space by forward kinematics to form a point cloud distribution. An envelope sphere model is constructed based on the point cloud distribution to represent the reachable workspace of the robot, and the envelope sphere model is rendered as a semi-transparent or wireframe form in the virtual environment.

[0015] Furthermore, in step S3, the process of determining the spatial operation pose change constraint includes: Iterate through the rotation angle and height of the end effector of the robot relative to the target object; The desired pose of the robot end effector relative to the base is solved based on traversal parameters. Verify whether the desired pose is reachable by inverse kinematics solution; if the inverse kinematics solution exists, output the operation pose; otherwise, continue traversing until all traversal parameters have been verified.

[0016] Furthermore, step S4 specifically includes: A path tree is constructed by randomly sampling from the 3D point cloud data, and an optimized Informed-RRT is used. The algorithm performs path planning for the robot to obtain collision-free discrete path points; Cubic spline interpolation is used to allocate time to the discrete path points and render them as continuous motion trajectory lines; the expected trajectory and actual motion trajectory of the end effector of the work robot are displayed in real time in the virtual environment.

[0017] Furthermore, step S5 specifically includes: A simplified collision envelope model for the operation robot is constructed. Combined with the three-dimensional point cloud data of the remote environment, the operation robot and the environment are subjected to real-time collision detection. The collision risk of the original path is corrected, and a collision-free motion path of the operation robot is obtained. Based on the collision-free motion path, an offline control program for the work robot is generated to control the work robot to move along the planned trajectory; and the expected trajectory and actual motion trajectory of the work robot's end effector are displayed in real time through a virtual environment.

[0018] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a robot teleoperation method based on the fusion of virtual reality and mixed reality, which has the following advantages: This invention integrates the immersive perception advantages of virtual reality with the augmented information advantages of mixed reality. On the one hand, it provides operators with a more immersive immersive perception of the remote environment, enabling them to monitor the status of the remote environment in real time and make timely judgments and decisions. On the other hand, it provides operators with more comprehensive augmented information, allowing them to more intuitively and efficiently monitor the real-time progress of the robot's teleoperation system.

[0019] Secondly, by constructing an interactive virtual environment, this invention visualizes the workspace and operational posture of the robot, plans the trajectory and visualizes it, greatly improving the safety of remote operation and meeting the remote operation needs of robots in many special scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 A flowchart of a robot teleoperation method based on the fusion of virtual reality and mixed reality provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the transformation relationship from the world coordinate system to the pixel coordinate system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the binocular camera positioning principle provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the remote environment construction structure provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] This invention discloses a robot teleoperation method based on the fusion of virtual reality and mixed reality, referring to... Figure 1 As shown, it includes the following steps: S1. The robot perceives the remote environment; an interactive virtual environment located at the near end is constructed, and the virtual environment is presented to the near-end operator through a virtual reality device; three-dimensional point cloud data based on the remote environment is constructed through three-dimensional reconstruction. S2. In the interactive virtual environment, the pose of the remote operation robot and the target object is estimated, and the virtual models of the operation robot and the target object are registered in three dimensions in the virtual environment. S3. Determine the workspace boundary condition constraints and spatial operation pose change constraints of the robot, and visualize the boundary condition constraints and pose change constraints in the virtual environment; S4. Perform path planning and trajectory planning on the robot, and visualize the planned path and trajectory in the virtual environment; S5. Based on the perceived remote environment and the planned path and trajectory, generate a collision-free motion offline control program for the work robot, and control the work robot to move along the planned trajectory.

[0024] This embodiment applies to the raw material addition scenario in the nitration process of explosives. During the nitration process, the raw materials need to be added to the nitrator precisely and steadily. This environment is filled with flammable and explosive materials, requiring extremely high precision and safety in operation, and direct personnel presence is not permitted.

[0025] In this embodiment, the robot first uses its binocular vision system to perceive the distant environment from all angles. Based on this perception data, an immersive, interactive virtual environment reflecting the real scene of the distant nitration workshop is constructed for the operator at the near end.

[0026] Secondly, within the constructed virtual environment, precise pose estimation is performed on the remotely executed task robot and the target object. Subsequently, high-precision virtual models of the task robot and the target object are overlaid onto their corresponding positions in the virtual environment, achieving virtual-real fusion. Simultaneously, this embodiment rigorously defines the safe operating range of the task robot, determines the boundary constraints of its task planning, and analyzes the spatial pose changes of the robot during operations such as grasping, moving, and feeding. All these critical boundary conditions and pose constraints are clearly visualized in the virtual environment in the form of semi-transparent envelopes, warning lines, or dynamic range boxes, intuitively alerting the operator and preventing collision risks caused by the robot exceeding its limits.

[0027] Path and trajectory planning are performed on the robot, and the resulting paths and trajectories are visualized in the virtual environment. A real-time collision detection program between the robot and the environment is designed and run to obtain a collision-free motion path for the robot. Then, an offline control program is written to control the robot to move along the planned trajectory.

[0028] Finally, collision-free path planning and smooth trajectory planning are performed for the robot based on the task objectives. The planned globally optimized path and fine motion trajectory are rendered in real time in the virtual environment. Based on the 3D point cloud model of the workshop, a real-time collision detection program is designed and run to detect collisions between the robot and the complex environment of the nitration workshop, ultimately obtaining an absolutely safe collision-free motion path. After confirming that all plans are safe and correct, an offline control program for the robot is generated; this program controls the robot in the remote nitration workshop to move strictly along the verified trajectory, accurately and smoothly completing the raw material addition task, with complete human-machine isolation throughout the process, maximizing the protection of life and production safety.

[0029] The implementation process of this embodiment is described in detail below: Step one involves observing the distant environment through the robot's binocular vision system. After camera calibration and image correction, left and right eye images of the distant environment, containing scene parallax and depth information, are acquired. The computer processing unit then renders these images onto the left and right displays of the virtual reality headset via two rendering channels, presenting a three-dimensional interactive virtual environment to the operator. Simultaneously, the computer processing unit performs 3D reconstruction of the acquired images to obtain 3D point cloud data of the distant environment, which supports subsequent path and trajectory planning for the robot.

[0030] First, the binocular camera is calibrated to obtain its intrinsic parameters, extrinsic parameters, and distortion coefficients. Based on the calibration results, distortion correction and stereo correction are performed on the acquired left and right views to ensure that the two images meet the epipolar constraints.

[0031] This embodiment performs precise calibration and correction of a stereo camera. Stereo camera calibration involves experimentally calculating the camera's intrinsic, extrinsic, and distortion parameters. First, approximately 30 images of a checkerboard calibration board are captured using the stereo camera. During capture, the checkerboard calibration board is ensured to appear in all areas of the image as much as possible. Then, left and right single-target calibration is performed by extracting the corner information of the calibration board, solving for the extrinsic and extrinsic parameter matrices, and obtaining the distortion coefficients using the least squares method. Finally, the calibration parameters are optimized through stereo joint calibration.

[0032] In this embodiment, the stereo camera is calibrated using the MATLAB calibration toolbox. A 27mm×27mm checkerboard pattern is used to calibrate the ZED Mini stereo camera. The camera is moved to different poses to capture images of the calibration board. The calibration results of the stereo camera's intrinsic and extrinsic parameters are shown in Table 1. The intrinsic parameters include the focal length, principal point coordinates, and image size of the left and right cameras, while the extrinsic parameters include the pose of the left and right camera coordinate systems relative to the world coordinate system. Since the world coordinate system coincides with the left camera coordinate system, the rotation and translation matrices of the left camera are both identity matrices. The calibration results of this embodiment are shown in Table 1 below: Table 1. Calibration results of the intrinsic and extrinsic parameters of the ZED Mini binocular camera.

[0033] Based on the intrinsic and extrinsic parameters and distortion coefficients of the binocular camera obtained from calibration, distortion correction and stereo correction are performed on the camera images. Correction enables geometric epipolar alignment between the left and right images, ensuring that the same spatial point is imaged on the same horizontal line in both images, simplifying the stereo matching process.

[0034] Subsequently, stereo matching is performed using the corrected stereo images to calculate pixel-level disparity maps. These disparities are then converted into depth information based on camera parameters, thereby generating 3D point cloud data representing the geometry of the distant environment. This embodiment establishes a transformation model between pixel coordinates and actual 3D spatial coordinates to achieve spatial positioning of camera image points.

[0035] This embodiment analyzes the imaging and positioning principles of a binocular camera. (Refer to...) Figure 2 As shown, this illustrates the transformation relationship from the world coordinate system to the pixel coordinate system. Figure 2 middle O w -X w Y w Z w Using the world coordinate system, O c -X c Y c Z c For the camera coordinate system, O p -xy For the image coordinate system, o-uv In pixel coordinate system, the focal length is... Origin of the camera coordinate system O c To the origin of the image coordinate system O p The distance, for a point in the world coordinate system Q(x w ,y w ,z w ) The corresponding coordinates in the image coordinate system are ( x,y ), the coordinates in the pixel coordinate system are ( u,v ). Figure 2 The medium blue line is used for highlighting.

[0036] This embodiment divides the transformation from the world coordinate system to the pixel coordinate system into three steps: First, the transformation from the world coordinate system to the camera coordinate system. For ease of analysis and calculation, this embodiment defines the world coordinate system as coinciding with the robot's base coordinate system, and the camera coordinate system as fixed at the optical center of the left eye camera. The above transformation can then be expressed as a rotation matrix R and a translation matrix T. Second, the transformation from the camera coordinate system to the image coordinate system. This transformation maps three-dimensional points to two dimensions, and the mapping relationship can be obtained through... Figure 2The similarity of triangles in the image is used to represent this. Finally, the image coordinate system is transformed into the pixel coordinate system. The image and pixel coordinate systems are located on the same imaging plane, but their units of measurement and origin positions are different. The transformation between the two coordinate systems can be completed by establishing the positional relationship between the image coordinates and the pixel coordinates.

[0037] For a certain point in space Q Establish world coordinates ( x w ,y w, z w ) to pixel coordinates ( u,v The transformation matrix of ) is:

[0038] in,( , ( ) represents the camera's focal length, expressed in pixels; , ) represents the coordinates of the camera principal point, indicating the coordinates of the origin of the image coordinate system in the pixel coordinate system; the two matrices on the right side of the equation are the camera's intrinsic and extrinsic parameters, respectively.

[0039] This embodiment uses parallel binocular vision, and its positioning principle is as follows: Figure 3 As shown, two parallel monocular cameras form an image based on the pinhole imaging principle. For the same target point, its imaging position in the left and right images is different, which will produce parallax. The positioning principle of the binocular camera is to obtain the parallax value of the matching point through stereo matching, and then calculate the depth information of the image through the principle of triangular similarity. Figure 3 The blue lines are also used for highlighting.

[0040] Let the optical centers of the left and right cameras be respectively O L and O R spatial point Q The x-coordinates of the left and right camera image coordinate systems are respectively and The depth information is then calculated using the following formula:

[0041] in, This refers to the camera's focal length. Baseline distance; For spatial points Q The depth value in the camera coordinate system can be obtained by reorganizing the above formula. By combining the transformation matrix from world coordinates to pixel coordinates mentioned above, the three-dimensional spatial coordinates of this point can be solved. x c ,y c ,z c This process generates the original 3D point cloud of the remote environment.

[0042] The key to obtaining high-precision environmental stereo features lies in the calibration of camera parameters, image distortion correction, and the accuracy of binocular stereo matching.

[0043] Finally, after image correction is completed, stereo matching is performed on the left and right images on the same horizontal line to find corresponding points, and then the disparity is calculated to obtain the accurate depth information of the corresponding points.

[0044] The stereo matching process comprises four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization post-processing. First, the matching cost is calculated by measuring the similarity of corresponding pixel regions in the left and right images; different cost functions significantly impact matching accuracy and noise resistance. Then, the matching cost is weighted and summed by considering pixel features and neighborhood information to enhance local consistency and improve the accuracy of disparity calculation. Next, disparity is calculated based on the aggregated cost map, searching for each pixel within a preset disparity range. The disparity value corresponding to the position that minimizes the aggregated cost is taken as the optimal matching disparity. Finally, disparity optimization post-processing is performed, generating a high-quality disparity map by smoothing the disparity map and reducing noise, and then calculating depth information.

[0045] The ZED Mini binocular camera used in this embodiment provides a relatively mature stereo matching optimization algorithm. This algorithm combines adaptive window size selection, dynamic cost calculation, and other optimization techniques to maintain high matching accuracy and real-time performance in different scenarios. The frame rate of the stereo matching algorithm update can typically reach 100 FPS. Therefore, this embodiment is based on this stereo matching algorithm for system development, which can reduce the workload of algorithm development and optimization while ensuring high accuracy and real-time performance of stereo matching, thereby improving the overall performance of the robot's teleoperation system.

[0046] Based on this, this embodiment utilizes the binocular vision system mounted on the observation robot to perceive the distant environment.

[0047] This embodiment constructs a virtual reality-based remote environment through three steps: binocular visual perception, computer processing, and operator perception. (Refer to...) Figure 4As shown, the visual perception unit acquires two-dimensional images of the distant environment using binocular cameras and transmits these images to the near-end computer processing unit. The computer processing unit renders the left and right eye images onto the left and right displays of the virtual reality headset using two rendering channels, and obtains environmental point clouds through 3D reconstruction to support subsequent steps. The operator perception unit projects the image information from the dual rendering channels onto the operator's left and right eyes.

[0048] In this embodiment, cross-platform development is achieved using the Unity3D image rendering engine and the Visual Studio integrated development platform to build a remote environment based on binocular vision.

[0049] In the aforementioned software platform, the Unity3D rendering engine establishes communication with the near-end virtual reality headset via the SteamVR protocol, rendering the remote environment onto the virtual reality headset. Simultaneously, the Visual Studio integrated development platform uses C++ programming and the TCP protocol to achieve real-time acquisition of images from the binocular cameras.

[0050] In the image rendering process of the virtual reality headset, the left and right images from the binocular camera are rendered onto two screens respectively. The two screens are then rendered onto the left and right displays of the VR headset through two rendering channels in Unity3D, thereby realizing the construction of the remote virtual environment.

[0051] Step two involves estimating the pose of the remote robot and the target object, completing 3D registration, and visualizing them in the constructed virtual environment.

[0052] This embodiment employs a 3D registration method for operational robots based on ArUco markers. By placing markers in the robot's working area, the robot's pose in its base coordinate system is calculated. Each component of the robot is modeled individually, and an assembly model of the robot is created using SolidWorks, then converted to STEP format. The robot model is simplified and processed using 3ds Max, and the processed model is saved as an FBX file. This file is then imported into the Unity3D environment to construct the assembly relationships and kinematic pairs of the parts. Based on the robot's pose estimation results, a visualization platform is built in Unity3D to perform 3D registration of the virtual robot model.

[0053] In this embodiment, a target object pose estimation method based on rendering comparison is adopted, combined with a ResNet-34 residual neural network model. The 3D model of the target object is rendered from different viewpoints, and the rendered images are compared with the actual scene images input into the neural network for learning, thereby achieving pose estimation of the target object. Based on the pose estimation results, the OBJ format model of the target object is imported into Unity3D to achieve 3D registration of the target object.

[0054] After completing the 3D registration, operators can see the virtual robot model overlapping with the real robot, as well as the virtual target object superimposed with the real target object, through a virtual reality headset.

[0055] Step 3: Determine the boundary condition constraints and spatial operation pose of the workspace of the robot, and visualize the boundary condition constraints in the virtual environment.

[0056] In this embodiment, the workspace boundary condition constraints are determined by establishing a kinematic model of the robot to quantify its workspace, then simplifying the workspace of the robot through geometric constraints, and finally rendering the constraints to the operator's field of vision in the virtual simulation environment established above, thus completing the visualization of the workspace range of the robot.

[0057] In this embodiment, Monte Carlo random sampling is used: the random pose of the robot's end effector in joint space is generated and transformed into Cartesian space through forward kinematics to form a point cloud distribution. Based on the point cloud distribution, an envelope sphere model is constructed to represent the robot's reachable workspace. In Unity3D, this sphere model is rendered as a semi-transparent or wireframe and overlaid on a remote real or virtual environment. The operator can intuitively see the boundary of the robot's workspace through a virtual reality headset, assisting in determining whether the target location is reachable.

[0058] In robot grasping tasks, the different placement postures of the target object determine the grasping pose of the robot's end effector. Therefore, it is necessary to analyze the robot's operational pose based on the target object's pose. In this embodiment, the spatial operational pose is determined by solving the robot's spatial operational pose based on the 3D registration information of the target object and the robot.

[0059] In this embodiment, firstly, the angles of the robot's end effector relative to the target object are traversed. θ and height h The desired pose of the robot's end effector relative to the base is calculated based on the traversed parameters. Then, the inverse kinematics of the robot's desired pose is obtained analytically. If the inverse kinematics exists, the pose is output as the robot's operational pose. If the inverse kinematics does not exist, the rotation angles of the robot's end effector are traversed again. θand height h Continue this process until all possible values ​​have been traversed; finally, output the solution for the robot's operational pose.

[0060] Step four: Perform path planning and trajectory planning for the robot, and visualize the resulting path and trajectory in a virtual environment.

[0061] Among them, path planning mainly solves the problem of how a robot can avoid obstacles and reach the target location, while trajectory planning further refines the path and generates a motion trajectory that meets dynamic constraints and task requirements.

[0062] In this embodiment, for a six-degree-of-freedom robot, collision-free global path planning is performed based on 3D point cloud data of the remote environment, provided that the workspace boundary constraints and the feasible end-effector pose constraints are met. The specific process is as follows: First, random sampling is performed within the feasible configuration space from the obstacle environment represented by the 3D point cloud environment, and a path tree is gradually constructed.

[0063] Secondly, the optimized Informed-RRT is adopted. The algorithm performs path search and optimization. This algorithm is based on the traditional Informed-RRT... Based on this, adaptations were made to suit the motion characteristics of the work robot: The robot's configuration (i.e., end-effector pose) is represented by joint angle vectors. For any two vector types, the joint angle distance is defined as the L¹ norm (i.e., the vector 1 norm) of the difference between the two vectors, and this is used as the metric for path cost. During the path tree expansion process, this cost metric is used to dynamically evaluate and optimize the connections between new nodes and existing nodes, prioritizing the retention of lower-cost and smoother path segments.

[0064] Finally, through the above sampling, expansion, and connection optimization mechanisms, a collision-free, globally optimized motion trajectory composed of discrete path points is generated while ensuring obstacle avoidance, which is used to guide the robot to complete the task.

[0065] Based on cubic spline interpolation, trajectory planning is performed on the robot. By analyzing the robot's maximum speed and acceleration constraints, time allocation is performed on discrete path points, and then trajectory interpolation is performed on the robot's joint space, enabling the robot to move smoothly along the predetermined trajectory.

[0066] In Unity3D, the planned sequence of path points is rendered as a continuous motion trajectory line. The expected and actual trajectories of the robot's end effector are displayed in real time, assisting operators in monitoring the motion process.

[0067] Step 5: Preprocess the 3D point cloud data reconstructed in Step 1, and construct a simplified collision envelope model for the robot. Combining the environmental 3D point cloud and the robot's collision envelope model, design and run a real-time collision detection program between the robot and the environment.

[0068] In this embodiment, noise, missing data, and redundant points are removed from the reconstructed 3D point cloud data. A hierarchical envelope model is adopted, with different envelope models established for different components to save computational resources. To achieve real-time collision detection between the robot and its environment, an octree is first constructed by processing the 3D point cloud. Then, starting from the root node of the octree, each layer of nodes is traversed sequentially, and it is determined whether the node boundary interferes with the robot's envelope model. If interference occurs, it indicates a collision at that node. This process is iteratively deepened, continuously increasing the node hierarchy until all discrete points involved in collisions are finally found from the leaf nodes.

[0069] After obtaining the collision-free motion path of the robot, an offline control program is written to control the remote robot to move along the planned trajectory.

[0070] This invention uses an observation robot equipped with a binocular vision system to perceive the environment, and on this basis, constructs an interactive virtual environment located at the near end.

[0071] Subsequently, pose estimation is performed on the remote robot and target object within the constructed near-end virtual environment, and the 3D virtual models of the robot and target object are registered in the virtual environment. The workspace of the robot is defined, and the boundary conditions and constraints for task planning are determined. The robot's operational pose is analyzed, and its spatial pose change constraints during operation are determined. The aforementioned boundary condition constraints and pose change constraints are visualized in the virtual environment.

[0072] Secondly, path and trajectory planning are performed on the robot, and the resulting paths and trajectories are visualized in the virtual environment. A real-time collision detection program between the robot and the environment is designed and run. After obtaining the collision-free motion path of the robot, an offline control program is written to control it to move along the planned trajectory.

[0073] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0074] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot teleoperation method based on the fusion of virtual reality and mixed reality, characterized in that, Includes the following steps: S1. The robot perceives the remote environment; an interactive virtual environment located at the near end is constructed, and the virtual environment is presented to the near-end operator through a virtual reality device; three-dimensional point cloud data based on the remote environment is constructed through three-dimensional reconstruction. S2. In the virtual environment, the pose of the remote operation robot and the target object is estimated, and the virtual models of the operation robot and the target object are registered in three dimensions in the virtual environment. S3. Determine the workspace boundary condition constraints and spatial operation pose change constraints of the robot, and visualize the boundary condition constraints in the virtual environment; S4. Perform path planning and trajectory planning on the robot, and visualize the planned path and trajectory in the virtual environment; S5. Based on the perceived remote environment and the planned path and trajectory, generate a collision-free motion offline control program for the work robot, and control the work robot to move along the planned trajectory.

2. The robot teleoperation method based on the fusion of virtual reality and mixed reality as described in claim 1, characterized in that, Step S1 specifically includes: By observing the distant environment through the binocular vision system on the robot, left and right eye images containing scene parallax and depth information are obtained. The computer processing unit renders the left and right images onto the left and right displays of the virtual reality headset through two rendering channels, presenting a three-dimensional interactive virtual environment to the operator. The computer processing unit performs three-dimensional reconstruction on the observed left and right eye images to obtain three-dimensional point cloud data based on the remote environment.

3. The robot teleoperation method based on the fusion of virtual reality and mixed reality as described in claim 2, characterized in that, When observing the distant environment using the binocular vision system mounted on the observation robot, the formula for calculating the depth information of the image is: in, This represents the depth value of a point in space within the camera coordinate system. and Representing spatial points The horizontal coordinate of the left and right camera image coordinate systems, Indicates the camera's focal length; Indicates the baseline distance.

4. The robot teleoperation method based on the fusion of virtual reality and mixed reality as described in claim 1, characterized in that, Step S2 specifically includes: For remote operation robots, the pose estimation method based on ArUco markers is used to calculate the pose of the robot in the base coordinate system, and the virtual model of the robot is registered in three dimensions in the virtual environment. For the target object, a pose estimation method based on rendering comparison is adopted, combined with a residual neural network model, to estimate the pose of the target object; by rendering the 3D model of the target object from different perspectives, the rendered image and the actual scene image are input into the residual neural network model for comparison and learning, thereby realizing the pose estimation of the target object, and the virtual model of the target object is registered in the virtual environment.

5. The robot teleoperation method based on the fusion of virtual reality and mixed reality as described in claim 1, characterized in that, Step S3, the process of determining and visualizing the workspace boundary condition constraints of the robot, includes: The random pose of the end effector of the robot in the joint space is generated by Monte Carlo random sampling method, and then transformed into Cartesian space by forward kinematics to form a point cloud distribution. An envelope sphere model is constructed based on the point cloud distribution to represent the reachable workspace of the robot, and the envelope sphere model is rendered as a semi-transparent or wireframe form in the virtual environment.

6. The robot teleoperation method based on the fusion of virtual reality and mixed reality as described in claim 5, characterized in that, In step S3, the process of determining the spatial operation pose change constraint includes: Iterate through the rotation angle and height of the end effector of the robot relative to the target object; The desired pose of the robot end effector relative to the base is solved based on traversal parameters. Verify whether the desired pose is reachable by inverse kinematics solution; if the inverse kinematics solution exists, output the operation pose; otherwise, continue traversing until all traversal parameters have been verified.

7. The robot teleoperation method based on the fusion of virtual reality and mixed reality as described in claim 2, characterized in that, Step S4 specifically includes: A path tree is constructed by randomly sampling from the 3D point cloud data, and an optimized Informed-RRT is used. The algorithm performs path planning for the robot to obtain collision-free discrete path points; Cubic spline interpolation is used to allocate time to the discrete path points and render them as continuous motion trajectory lines; the expected trajectory and actual motion trajectory of the end effector of the work robot are displayed in real time in the virtual environment.

8. The robot teleoperation method based on the fusion of virtual reality and mixed reality as described in claim 7, characterized in that, Step S5 specifically includes: A simplified collision envelope model for the operation robot is constructed. Combined with the three-dimensional point cloud data of the remote environment, the operation robot and the environment are subjected to real-time collision detection. The collision risk of the original path is corrected, and a collision-free motion path of the operation robot is obtained. Based on the collision-free motion path, an offline control program for the work robot is generated to control the work robot to move along the planned trajectory; and the expected trajectory and actual motion trajectory of the work robot's end effector are displayed in real time through a virtual environment.

Images