A teleoperated robot online teaching system and method based on digital twin
By establishing a three-dimensional model and building a collision detection module in the online teaching of remote operation robots, the problem of lack of real-time posture planning and collision detection in the existing technology is solved, the safety and efficiency of teaching is improved, and the higher Reliability and security.
Patent Information
- Application Number
- CN202210106753.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-01-28
AI Technical Summary
The existing digital twin technology lacks real-time posture planning and collision detection in online teaching of remote operation robots, resulting in insecure and inefficient teaching.
Establish a three-dimensional model of the robot and its work scenario, and build a collision detection module to predict collisions between the robot and the work scenario in advance. Through the communication between digital twin units and physical units, plan teaching posture data and perform collision detection before movement. .
It improves the reliability, safety and efficiency of online teaching of remote operation robots, reduces collision risks, and makes teaching more reliable and efficient.
Smart Images

Figure CN114407015B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an online teaching system and method for a teleoperated robot based on digital twins, and relates to intelligent manufacturing and human-computer interaction technology. Background Art
[0002] Industrial robots are widely used in the processing and manufacturing industries, and the promotion and application rate of industrial robots is an important indicator of a country's industrial automation level. For teleoperated robots, since the instructor and the physical robot are in different physical spaces, the safety of teaching activities is reduced, which greatly limits the efficiency of online teaching programming of teleoperated robots.
[0003] Existing digital twin technology only monitors the working status of the physical robot in real time by building a digital twin model of the teleoperated robot, but there is little exploration of the application of real-time posture planning and collision detection of the robot. Real-time posture planning and collision detection can improve the human-machine interaction of the online teaching of the teleoperated robot, predict the collision interference between the robot and the working scene under the planned path and posture, and improve the programming efficiency of the online teaching of the teleoperated robot. Summary of the invention
[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a remote-controlled robot online teaching system and method based on digital twins, establishes a three-dimensional model of the robot and its working scene, and constructs a collision detection module. Before controlling the movement of the physical robot, the planned teaching posture data is first sent to the collision detection module to predict in advance whether there is a collision between the robot and the working scene, thereby reducing the risk of collision and improving the reliability, safety and teaching efficiency of the remote-controlled robot's online teaching.
[0005] The technical solution of the present invention is as follows:
[0006] In one aspect, the present invention provides a teleoperated robot online teaching system based on digital twin, comprising: a digital twin unit and a physical unit communicatively connected to the digital twin unit and a teleoperated robot teaching unit;
[0007] The physical unit is used to collect three-dimensional data of the assembly scene and position data of the physical robot; and the physical unit is also used to drive the physical robot to move according to instructions;
[0008] The teleoperated robot teaching unit is used to plan the motion path of the robot and output teaching posture data based on the motion path;
[0009] The digital twin unit includes a work scene reconstruction module, a robot digital twin modeling module, a collision detection module and an instruction output module; the work scene reconstruction module establishes a three-dimensional model of the work scene according to the three-dimensional data of the assembly scene; the robot digital twin modeling module establishes a three-dimensional model of the virtual robot according to the posture data of the physical robot, and creates a robot kinematic model corresponding to the physical robot; the collision detection module controls the three-dimensional model of the virtual robot to move according to the planned motion path based on the robot kinematic model and the teaching posture data, and performs collision detection between the three-dimensional model of the virtual robot and the three-dimensional model of the work scene during the movement; the instruction output module is used to generate instructions according to the teaching posture data, and output the instructions to the physical unit.
[0010] As a preferred embodiment, the physical unit includes an RGB-D camera, a physical robot, and a physical robot control cabinet;
[0011] The RGB-D camera is used to collect three-dimensional data of the assembly scene and the position and posture data of the physical robot;
[0012] The physical robot control cabinet is used to receive instructions and drive the physical robot to move according to the instructions.
[0013] As a preferred embodiment, the teleoperated robot teaching unit includes an augmented reality registration module and a tele-teaching posture planning module;
[0014] The augmented reality registration module establishes a three-dimensional model of the virtual robot according to the posture data of the physical robot;
[0015] The remote teaching posture planning module is used to plan the motion path of the virtual robot three-dimensional model and output teaching posture data based on the motion path.
[0016] As a preferred implementation, the physical unit collects an image of the physical robot and obtains position and posture data of the physical robot in the image;
[0017] The establishment of the virtual robot three-dimensional model according to the posture data of the physical robot is achieved by a hand-eye calibration module;
[0018] The hand-eye calibration module obtains an image of the physical robot and a base coordinate system of the physical robot, and calculates a transformation matrix from the image coordinate system to the base coordinate system of the physical robot through a hand-eye calibration algorithm;
[0019] The actual posture data of the physical robot is obtained according to the transformation matrix and the posture data of the physical robot in the image; and a virtual robot three-dimensional model consistent with the physical robot and the coordinate system is created according to the actual posture data of the physical robot.
[0020] As a preferred implementation, the three-dimensional data of the assembly scene collected by the physical unit is specifically a point cloud image of the assembly scene;
[0021] The work scene reconstruction module removes the point cloud image of the physical robot from the point cloud image of the assembly scene according to the transformation matrix from the image coordinate system to the physical robot base coordinate system, and obtains the point cloud image of the work scene as the three-dimensional model of the work scene.
[0022] On the other hand, the present invention also provides an online teaching method for a teleoperated robot based on digital twin, comprising the following steps:
[0023] Use image acquisition equipment to collect three-dimensional data of the assembly scene and the position and posture data of the physical robot;
[0024] The instructor plans the robot's motion path on the computer, and the computer outputs the teaching posture data based on the motion path;
[0025] Using a computer, a three-dimensional model of the working scene and a three-dimensional model of the virtual robot are respectively established according to the three-dimensional data of the assembly scene and the position and posture data of the physical robot;
[0026] Using a computer to control the virtual robot three-dimensional model to perform corresponding movements according to the teaching posture data, and performing collision detection between the virtual robot three-dimensional model and the work scene three-dimensional model during the movement;
[0027] If there is no collision, the computer outputs motion instructions to the physical robot control cabinet based on the teaching posture data, and the physical robot control cabinet controls the movement of the physical robot according to the instructions; if a collision occurs, it returns to re-plan the robot's motion path.
[0028] As a preferred implementation, the posture data of the physical robot is specifically the posture data of the physical robot in the coordinate system of the image acquisition device, and the steps of the computer establishing the three-dimensional model of the virtual robot according to the posture data of the physical robot are specifically as follows:
[0029] The transformation matrix from the image acquisition device coordinate system to the physical robot base coordinate system is calculated using the hand-eye calibration algorithm;
[0030] The actual posture data of the physical robot is obtained according to the transformation matrix and the posture data of the physical robot; and a virtual robot three-dimensional model consistent with the physical robot and the coordinate system is created according to the actual posture data of the physical robot.
[0031] As a preferred embodiment, the steps of establishing a three-dimensional model of the working scene based on the three-dimensional data of the assembly scene by a computer are specifically as follows:
[0032] The three-dimensional data of the assembly scene acquired by the image acquisition device is specifically a depth image of the assembly scene;
[0033] The computer converts the depth image of the assembly scene into a point cloud image of the assembly scene;
[0034] The computer removes the point cloud image of the physical robot from the point cloud image of the assembly scene according to the conversion matrix from the image acquisition device coordinate system to the physical robot base coordinate system, and obtains the point cloud image of the working scene;
[0035] An octree model of the working scene is established according to the point cloud image of the working scene.
[0036] As a preferred implementation manner, the computer converts the depth image of the assembly scene into a point cloud image of the assembly scene in the following specific steps:
[0037] Get the depth value Z of the image acquisition device;
[0038] The conversion relationship between the depth image coordinates and the point cloud coordinates is obtained according to the following formula:
[0039]
[0040] In the above formula, (X, Y, Z) is the three-dimensional coordinate of the point cloud, (x, y) is the depth image coordinate, and f x 、f y is the focal length in the x-axis and y-axis directions in the depth image, c x 、c y is the coordinate of the principal point in the depth image;
[0041] The mobile image acquisition device collects depth images of the assembly scene from different perspectives, adopts the point cloud registration algorithm between image frames, uses the newly captured depth image to fill the holes in the previous depth image, and fuses and splices them to obtain the point cloud image of the entire assembly scene;
[0042] After obtaining the point cloud image of the entire assembly scene, filtering processing is also performed, including:
[0043] Through-filter processing: Set a threshold range on each coordinate axis of the image acquisition device coordinate system, remove outliers outside the threshold range, and eliminate point clouds that are irrelevant to the robot's working scene;
[0044] Voxel filtering: The point cloud image after the through-filtering process is divided into several three-dimensional voxel grids. All points in the voxel grid are represented by a centroid point of the voxel grid to achieve downsampling of the point cloud image.
[0045] Radius filtering processing takes a point cloud in the point cloud image as the center of the sphere, sets the radius r and the point cloud number threshold. If the number of point clouds adjacent to the center of the sphere with a radius of r is less than the point cloud number threshold, the point cloud and all adjacent point clouds are removed; if it is greater than or equal to the point cloud number threshold, all point clouds are retained.
[0046] As a preferred implementation, the step of removing the point cloud image of the physical robot from the point cloud image of the assembly scene to obtain the point cloud image of the working scene is specifically as follows:
[0047] According to the conversion matrix from the image acquisition device coordinate system to the physical robot base coordinate system, the virtual robot 3D model coordinate system is made consistent with the robot coordinate system in the point cloud image of the assembly scene;
[0048] The virtual robot 3D model is overlaid on the point cloud image of the assembly scene, and the point cloud of the area covered by the virtual robot 3D model is deleted to obtain the point cloud image of the working scene.
[0049] The present invention has the following beneficial effects:
[0050] 1. The present invention discloses an online teaching system for a remote-controlled robot based on digital twins. The digital twin unit is used to establish a three-dimensional model of the robot and its working scene, and a collision detection module is constructed. Before controlling the movement of the physical robot, the planned teaching posture data is first sent to the collision detection module to predict in advance whether there is a collision between the robot and the working scene, thereby reducing the risk of collision and improving the reliability, safety and teaching efficiency of the online teaching of the remote-controlled robot.
[0051] 2. The present invention provides an online teaching method for a remote-controlled robot based on digital twins, which uses pass-through filtering, voxel filtering and radius filtering to process point cloud images, thereby narrowing the field of view of the point cloud images and making the edges of the point cloud images clearer.
[0052] 3. The present invention provides an online teaching method for a remote-controlled robot based on digital twins, which removes the point cloud image of the physical robot from the point cloud image of the assembly scene to avoid misjudging the physical robot as an obstacle for collision detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 This is a framework diagram of an online teaching system for a teleoperated robot according to a first embodiment of the present invention;
[0054] Figure 2 Schematic diagram of the hardware composition of the online teaching system for a teleoperated robot according to the first embodiment of the present invention;
[0055] Figure 3 A flowchart of an example of online teaching steps for a teleoperated robot provided in the second embodiment of the present invention;
[0056] Figure 4 A flowchart of obtaining a point cloud image of a working scene provided in the second embodiment of the present invention;
[0057] Figure 5 This is an example diagram of overlaying the virtual robot 3D model on the point cloud data of the assembly scene in the second embodiment of the present invention. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.
[0060] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0061] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0062] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.
[0063] Embodiment 1:
[0064] See also Figure 1 , this embodiment provides a teleoperated robot online teaching system based on digital twin, which is composed of a digital twin unit, a physical unit communicatively connected to the digital twin unit, and a teleoperated robot teaching unit;
[0065] The physical unit is used to collect three-dimensional data of the assembly scene and the position and posture data of the physical robot and send the above data to the digital twin unit; and the physical unit is also used to drive the physical robot to move according to the instructions;
[0066] The teleoperated robot teaching unit is used to plan the motion path of the robot, and output teaching posture data based on the motion path and send it to the digital twin unit;
[0067] The digital twin unit includes a work scene reconstruction module, a robot digital twin modeling module, a collision detection module and a command output module;
[0068] The work scene reconstruction module establishes a three-dimensional model of the work scene according to the three-dimensional data of the assembly scene;
[0069] The robot digital twin modeling module establishes a virtual robot three-dimensional model according to the posture data of the physical robot, and creates a robot kinematic model corresponding to the physical robot;
[0070] The collision detection module controls the virtual robot three-dimensional model to move according to the planned motion path based on the robot kinematic model and the teaching posture data, and performs collision detection between the virtual robot three-dimensional model and the work scene three-dimensional model during the motion process; if the collision detection module detects that a collision occurs, the remote-controlled robot teaching unit is notified to re-plan the robot motion path; if the collision detection module detects that there is no collision, an instruction is generated according to the teaching posture data outputted from the currently planned motion path, and the instruction is sent to the physical unit.
[0071] The instruction output module is used to generate instructions according to the teaching posture data and output the instructions to the physical unit.
[0072] See Figure 2 In one embodiment, the hardware components of the online teaching system described in the present invention include an RGB-D camera 10, a computer No. 11, a computer No. 2 12, a robot control cabinet 14, and a physical robot 15; and the system is operated by a teaching staff 13.
[0073] Among them, the RGB-D camera 10, the physical robot 15 and the physical robot control cabinet 14 constitute a physical unit;
[0074] The RGB-D camera 10 is used to collect the depth image of the assembly scene, obtain the RGB image of the physical robot, and obtain the posture data of the physical robot in the RGB image; the RGB-D camera 10 sends the collected image to the first computer 11 and the second computer 12 in real time;
[0075] The physical robot control cabinet 14 is used to receive instructions and drive the physical robot 15 to move according to the instructions.
[0076] Among them, the remote-controlled robot teaching unit is loaded in computer No. 11, and the digital twin unit is loaded in computer No. 2 12; computer No. 11 and computer No. 2 12 communicate in real time; the robot control cabinet 14 communicates with computer No. 2 12 and receives instructions issued by computer No. 2 12.
[0077] In one embodiment, the teleoperated robot teaching unit includes an augmented reality registration module and a tele-teaching posture planning module;
[0078] The augmented reality registration module establishes a three-dimensional model of the virtual robot according to the posture data of the physical robot;
[0079] The remote teaching posture planning module is used to plan the motion path of the virtual robot three-dimensional model, and outputs teaching posture data based on the motion path and sends it to the digital twin unit. The digital twin unit updates the movement of the corresponding joints in the virtual robot three-dimensional model in real time according to the teaching posture data.
[0080] In one embodiment, the augmented reality registration module and the robot digital twin modeling module both use a hand-eye calibration module to establish a virtual robot three-dimensional model based on the posture data of the physical robot;
[0081] The hand-eye calibration module uses a hand-eye calibration algorithm to calculate the transformation matrix between the RGB camera coordinate system in the RGB-D camera and the physical robot base coordinate system;
[0082] The actual posture data of the physical robot is obtained according to the transformation matrix and the posture data of the physical robot in the image; and a virtual robot three-dimensional model consistent with the physical robot and the coordinate system is created according to the actual posture data of the physical robot.
[0083] In one embodiment, the three-dimensional data of the assembly scene collected by the RGB-D camera is specifically a point cloud image of the assembly scene; first, a depth image of the assembly scene is collected by the RGB-D camera, and then the depth image is converted into a point cloud image;
[0084] The work scene reconstruction module removes the point cloud image of the physical robot from the point cloud image of the assembly scene according to the conversion matrix from the image coordinate system to the physical robot base coordinate system, so as to obtain the point cloud image of the work scene;
[0085] The work scene reconstruction module establishes an octree model of the work scene according to the point cloud image of the work scene; the collision detection module establishes an octree graph collision detection model according to the virtual robot three-dimensional model and the point cloud image of the work scene, and performs collision detection between the point cloud image of the work scene and the virtual robot three-dimensional model.
[0086] Embodiment 2:
[0087] This embodiment provides an online teaching method for a teleoperated robot based on digital twins, comprising the following steps:
[0088] Use RGB-D cameras to collect 3D data of the assembly scene and the pose data of the physical robot;
[0089] The instructor plans the robot's motion path on the computer, and the computer outputs the teaching posture data based on the motion path (posture data refers to position and posture data);
[0090] Using a computer, a three-dimensional model of the working scene and a three-dimensional model of the virtual robot are respectively established according to the three-dimensional data of the assembly scene and the position and posture data of the physical robot;
[0091] Using a computer to control the virtual robot three-dimensional model to perform corresponding movements according to the teaching posture data, and performing collision detection between the virtual robot three-dimensional model and the work scene three-dimensional model during the movement;
[0092] If there is no collision, the computer outputs motion instructions to the physical robot control cabinet based on the teaching posture data, and the physical robot control cabinet controls the movement of the physical robot according to the instructions; if a collision occurs, it returns to re-plan the robot's motion path.
[0093] In one embodiment, the posture data of the physical robot is specifically the posture data of the physical robot in the RGB camera coordinate system of the RGB-D camera, and the steps of the computer establishing the three-dimensional model of the virtual robot according to the posture data of the physical robot are specifically as follows:
[0094] The transformation matrix M from the RGB camera coordinate system in the RGB-D camera to the physical robot base coordinate system is calculated through the hand-eye calibration algorithm;
[0095] The actual posture data of the physical robot is obtained according to the transformation matrix M and the posture data of the physical robot in the RGB camera coordinate system; and a virtual robot three-dimensional model consistent with the physical robot and the coordinate system is created according to the actual posture data of the physical robot.
[0096] In one embodiment, the steps of establishing the three-dimensional model of the working scene according to the three-dimensional data of the assembly scene by the computer are specifically as follows:
[0097] The RGB-D camera acquires a depth image of the assembly scene through a depth camera;
[0098] The computer converts the depth image of the assembly scene into a point cloud image of the assembly scene;
[0099] The computer removes the point cloud image of the physical robot from the point cloud image of the assembly scene according to the conversion matrix from the image acquisition device coordinate system to the physical robot base coordinate system, and obtains the point cloud image of the working scene;
[0100] An octree model of the working scene is established according to the point cloud image of the working scene.
[0101] See also Figure 3 According to the above embodiment, the present invention provides a specific online teaching step of a teleoperated robot:
[0102] S10. The teaching staff uses a computer to establish a virtual robot three-dimensional model and a robot kinematic model according to the model, size, joint size and posture data of the physical robot.
[0103] S20, perform hand-eye calibration on the physical robot and the RGB-D camera to obtain a transformation matrix M from the RGB camera coordinate system to the physical robot base coordinate system;
[0104] S30. Generate a point cloud image of the assembly scene based on the depth image of the assembly scene captured by the RGB-D camera; remove the point cloud image of the physical robot from the point cloud image based on the transformation matrix M obtained in S20 to establish a point cloud image of the working scene; and establish an octree model of the working scene based on the point cloud image of the working scene.
[0105] S40, the teaching staff plans the motion path and posture of the robot using a computer, the computer outputs the teaching posture data in real time, and updates the motion of the corresponding joints of the robot in the virtual robot three-dimensional model according to the teaching posture data.
[0106] S50, based on the conversion matrix M from the RGB-D camera coordinate system to the physical robot base coordinate system, the virtual robot model is superimposed on the work scene octree model to establish a collision detection model of the virtual machine robot three-dimensional model.
[0107] S60. Use the octree model of the working scene and the three-dimensional model of the virtual robot to perform collision interference detection. If no collision occurs, generate a robot motion command based on the current teaching posture data and send it to the physical robot control cabinet to control the movement of the physical robot. If a collision occurs, send a pause command to the physical robot control cabinet and return to S40 to replan the path until the remote control is completed.
[0108] See also Figure 4 In one embodiment, in step S30, a point cloud image of the assembly scene is generated, and the point cloud image of the physical robot is removed from the point cloud image of the assembly scene. The method for establishing the point cloud image of the working scene is as follows:
[0109] Get the depth value Z of the image acquisition device;
[0110] S301, point cloud image generation: The RGB-D camera collects the depth image of the assembly scene, and uses the ROS robot control system to convert the depth image into a point cloud image to achieve three-dimensional reconstruction. The depth value Z of the RGB-D camera is known, and the conversion relationship between the depth image coordinates and the point cloud coordinates can be obtained according to the depth camera imaging model shown in the following formula:
[0111] The conversion relationship between the depth image coordinates and the point cloud coordinates is obtained according to the following formula:
[0112]
[0113] In the above formula, (X, Y, Z) is the three-dimensional coordinate of the point cloud, (x, y) is the depth image coordinate, and f x 、f y is the focal length in the x-axis and y-axis directions in the depth image, c x 、c y is the coordinate of the principal point in the depth image;
[0114] The mobile RGB-D camera collects depth images of the assembly scene from different perspectives. The point cloud registration algorithm between image frames is used to fill the holes in the previous depth images with the newly captured depth images, and the point cloud images of the entire assembly scene are obtained by fusion and splicing.
[0115] S302, after acquiring the point cloud image of the entire assembly scene, filtering is also performed. Since the field of view of the point cloud image obtained by the camera is too large, scene point clouds that are irrelevant to the robot workspace will be collected, which will cause the point cloud update rate to slow down and the edges of the point cloud image to be too rough, which is not conducive to subsequent collision detection. Therefore, this embodiment uses straight-through filtering, voxel filtering and radius filtering to process the point cloud image, narrow the field of view of the point cloud image, and make the edges of the point cloud image clearer. The specific steps are as follows:
[0116] Through-filter processing: a threshold range is set on each coordinate axis of the depth camera coordinate system of the RGB-D camera (set to (0-1.3 m) in this embodiment), outliers outside the threshold range are removed, point clouds irrelevant to the robot working scene are eliminated, and the field of view of the point cloud image is narrowed;
[0117] Voxel filtering: The point cloud image after the through-filtering process is divided into a number of three-dimensional voxel grids (the size of the three-dimensional voxel grid in this embodiment is 0.005m), and all points in the voxel grid are represented by a centroid point of the voxel grid to achieve downsampling of the point cloud image, so that the surface corresponding to the sampling point is more accurate and the edge of the point cloud is clearer;
[0118] Radius filtering processing: take a point cloud in the point cloud image as the center of the sphere, set the radius r (r = 0.002m in this embodiment) and the point cloud number threshold (the point cloud number threshold is set to 30 in this embodiment). If the number of point clouds adjacent to the center of the sphere with a radius of 0.002m is less than 30, then the point cloud and all adjacent point clouds are removed; if it is greater than or equal to 30, then all point clouds are retained to further eliminate outliers that are not related to the robot working scene.
[0119] S303: In order to avoid misjudging the physical robot as an obstacle for collision detection, the point cloud image of the physical robot needs to be removed from the point cloud image of the assembly scene. The specific steps are as follows:
[0120] According to the transformation matrix M from the RGB-D camera coordinate system to the physical robot base coordinate system, the virtual robot 3D model coordinate system is made consistent with the robot coordinate system in the point cloud image of the assembly scene;
[0121] S304, please refer to Figure 5 , the virtual robot 3D model is overlaid on the point cloud image of the assembly scene, and the point cloud of the area covered by the virtual robot 3D model is deleted to obtain the point cloud image of the working scene.
[0122] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. An online teaching system for teleoperated robots based on digital twins, It is characterized in that include: A digital twin unit, a physical unit communicatively connected to the digital twin unit, and a teleoperated robot teaching unit; The physical unit is used to collect three-dimensional data of the assembly scene and position data of the physical robot; and the physical unit is also used to drive the physical robot to move according to instructions; The teleoperated robot teaching unit is used to plan the motion path of the robot and output teaching posture data based on the motion path; The digital twin unit includes a work scene reconstruction module, a robot digital twin modeling module, a collision detection module and an instruction output module; the work scene reconstruction module establishes a three-dimensional model of the work scene according to the three-dimensional data of the assembly scene; the robot digital twin modeling module establishes a three-dimensional model of the virtual robot according to the posture data of the physical robot, and creates a robot kinematic model corresponding to the physical robot; the collision detection module controls the three-dimensional model of the virtual robot to move according to the planned motion path based on the robot kinematic model and the teaching posture data, and performs collision detection between the three-dimensional model of the virtual robot and the three-dimensional model of the work scene during the motion; the instruction output module is used to generate instructions according to the teaching posture data, and output the instructions to the physical unit; The three-dimensional data of the assembly scene collected by the physical unit is specifically a point cloud image of the assembly scene; The work scene reconstruction module removes the point cloud image of the physical robot from the point cloud image of the assembly scene according to the conversion matrix from the image coordinate system to the physical robot base coordinate system, and obtains the point cloud image of the work scene as the three-dimensional model of the work scene; The physical unit collects an image of the physical robot and obtains position and posture data of the physical robot in the image; The establishment of the virtual robot three-dimensional model according to the posture data of the physical robot is achieved by a hand-eye calibration module; The hand-eye calibration module obtains an image of the physical robot and a base coordinate system of the physical robot, and calculates a transformation matrix from the image coordinate system to the base coordinate system of the physical robot through a hand-eye calibration algorithm; The actual posture data of the physical robot is obtained according to the transformation matrix and the posture data of the physical robot in the image; and a virtual robot three-dimensional model consistent with the physical robot and the coordinate system is created according to the actual posture data of the physical robot.
2. According to claim 1, a teleoperated robot online teaching system based on digital twin, Features: The physical unit includes an RGB-D camera, a physical robot, and a physical robot control cabinet; The RGB-D camera is used to collect three-dimensional data of the assembly scene and the position and posture data of the physical robot; The physical robot control cabinet is used to receive instructions and drive the physical robot to move according to the instructions.
3. According to claim 1, a teleoperated robot online teaching system based on digital twin, Features: The remote-operated robot teaching unit includes an augmented reality registration module and a remote teaching posture planning module; The augmented reality registration module establishes a three-dimensional model of the virtual robot according to the posture data of the physical robot; The remote teaching posture planning module is used to plan the motion path of the virtual robot three-dimensional model and output teaching posture data based on the motion path.
4. An online teaching method for teleoperated robots based on digital twins, It is characterized in that The following steps are involved: Use image acquisition equipment to collect three-dimensional data of the assembly scene and the position and posture data of the physical robot; The instructor plans the robot's motion path on the computer, and the computer outputs the teaching posture data based on the motion path; Using a computer, a three-dimensional model of the working scene and a three-dimensional model of the virtual robot are respectively established according to the three-dimensional data of the assembly scene and the position and posture data of the physical robot; Using a computer to control the virtual robot three-dimensional model to perform corresponding movements according to the teaching posture data, and performing collision detection between the virtual robot three-dimensional model and the work scene three-dimensional model during the movement; If there is no collision, the computer outputs motion instructions to the physical robot control cabinet based on the teaching posture data, and the physical robot control cabinet controls the movement of the physical robot according to the instructions; if a collision occurs, it returns to re-plan the robot's motion path; The posture data of the physical robot is specifically the posture data of the physical robot in the coordinate system of the image acquisition device. The steps of the computer establishing the three-dimensional model of the virtual robot according to the posture data of the physical robot are specifically as follows: The transformation matrix from the image acquisition device coordinate system to the physical robot base coordinate system is calculated using the hand-eye calibration algorithm; Acquire actual posture data of the physical robot according to the transformation matrix and the posture data of the physical robot; create a virtual robot three-dimensional model consistent with the physical robot and the coordinate system according to the actual posture data of the physical robot; The specific steps of the computer establishing a three-dimensional model of the working scene based on the three-dimensional data of the assembly scene are as follows: The three-dimensional data of the assembly scene acquired by the image acquisition device is specifically a depth image of the assembly scene; The computer converts the depth image of the assembly scene into a point cloud image of the assembly scene; The computer removes the point cloud image of the physical robot from the point cloud image of the assembly scene according to the conversion matrix from the image acquisition device coordinate system to the physical robot base coordinate system, and obtains the point cloud image of the working scene; Establish an octree model of the working scene according to the point cloud image of the working scene; The computer converts the depth image of the assembly scene into a point cloud image of the assembly scene in the following specific steps: Get the depth value Z of the image acquisition device; The conversion relationship between the depth image coordinates and the point cloud coordinates is obtained according to the following formula: In the above formula, (X, Y, Z) is the three-dimensional coordinate of the point cloud, (x, y) is the depth image coordinate, and f x 、f y is the focal length in the x-axis and y-axis directions in the depth image, c x 、c y is the coordinate of the principal point in the depth image; The mobile image acquisition device collects depth images of the assembly scene from different perspectives, adopts the point cloud registration algorithm between image frames, uses the newly captured depth image to fill the holes in the previous depth image, and fuses and splices them to obtain the point cloud image of the entire assembly scene; After obtaining the point cloud image of the entire assembly scene, filtering processing is also performed, including: Through-filter processing: Set a threshold range on each coordinate axis of the image acquisition device coordinate system, remove outliers outside the threshold range, and eliminate point clouds that are irrelevant to the robot's working scene; Voxel filtering: The point cloud image after the through-filtering process is divided into several three-dimensional voxel grids. All points in the voxel grid are represented by a centroid point of the voxel grid to achieve downsampling of the point cloud image. Radius filtering processing, taking a certain point cloud in the point cloud image as the sphere center, setting the radius r and the point cloud number threshold. If the number of point clouds adjacent to the sphere center within the sphere with radius r is less than the point cloud number threshold, then the point cloud and all adjacent point clouds are removed; if it is greater than or equal to the point cloud number threshold, then all point clouds are retained; The step of removing the point cloud image of the physical robot from the point cloud image of the assembly scene to obtain the point cloud image of the working scene is specifically as follows: According to the conversion matrix from the image acquisition device coordinate system to the physical robot base coordinate system, the virtual robot 3D model coordinate system is made consistent with the robot coordinate system in the point cloud image of the assembly scene; The virtual robot 3D model is overlaid on the point cloud image of the assembly scene, and the point cloud of the area covered by the virtual robot 3D model is deleted to obtain the point cloud image of the working scene.
