A digital twin modeling method and system for assembly robot teleoperation environment

By identifying and calculating the parts and robot position information in the remote operating environment of the assembly robot, the three-dimensional model reconstruction accuracy and consistency problems in traditional methods are solved, and high-precision digital twin modeling is achieved.

CN114417616BActive Publication Date: 2024-11-05QINGDAO UNIV OF TECH +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210067998.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-20
Publication Date
2024-11-05
Estimated Expiration
2042-01-20

AI Technical Summary

Technical Problem

The traditional digital twin modeling method of the remote operation environment of assembled robots is based on visual reconstruction, making it difficult to achieve accurate three-dimensional model reconstruction, especially the occlusion problem, lack of semantics, and it is difficult to ensure the consistency of the model.

Method used

The image acquisition device collects remote operating environment images, recognizes the components being assembled, generates three-dimensional models using predefined assembly sequence, calculates the position information of the assembly and robot, and establishes a digital twin model for the remote operating environment of the assembly robot to avoid based on Dense reconstruction of vision improves the accuracy and visualization of three-dimensional models.

Benefits of technology

It realizes high-precision three-dimensional model reconstruction of assembly and robots, obtains its relative positional relationship, and establishes an accurate remote operating environment digital twin model, improving the consistency and visualization of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114417616B_ABST
    Figure CN114417616B_ABST
Patent Text Reader

Abstract

The present invention relates to a digital twin modeling method for an assembly robot teleoperation environment, comprising the following steps: acquiring an image of the teleoperation environment; identifying parts being assembled; querying an assembly sequence of an assembly, and obtaining a list of assembled parts according to the parts being assembled; generating a three-dimensional model of the current assembly according to the list, and calculating the position and posture information of the current assembly in a coordinate system of an image acquisition device; loading the three-dimensional model of the robot, and determining a coordinate transformation relationship between the robot coordinate system and the image acquisition device coordinate system; determining the position and posture information of the robot in the coordinate system of the image acquisition device according to the coordinate transformation relationship; determining the relative position relationship between the current assembly and the robot according to the position and posture information of the current assembly and the robot in the coordinate system of the image acquisition device; and establishing a digital twin model of the teleoperation environment based on the three-dimensional model of the current assembly, the three-dimensional model of the robot, and the relative position relationship between the current assembly and the robot.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a digital twin modeling method and system for an assembly robot teleoperation environment, and belongs to the technical field of intelligent manufacturing and digital twin modeling. Background Art

[0002] In the teleoperation of assembly robots, the local operator needs to control the remote robot to complete the assembly task. The teleoperation environment of the remote assembly robot mainly includes elements such as the robot and the assembly. In order to accurately reconstruct the geometric models of the robot and the assembly, their position and posture need to be accurately measured. Therefore, the remote environment is a dynamically changing environment.

[0003] In order to build a digital twin model of a remote teleoperation environment on a low-end computer, it is necessary to dynamically build a 3D model of the assembly and the robot according to the assembly process. Consistent geometric modeling is the key to ensure that the twin model is consistent with the teleoperation environment.

[0004] Traditional modeling methods are usually vision-based reconstruction methods, specifically computer technology that uses two-dimensional projection to restore three-dimensional information. The reconstruction results are usually point clouds, meshes, voxels and other models, which lack semantics, cannot achieve component segmentation, and have occlusion problems, making it difficult to achieve global reconstruction. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a digital twin modeling method for an assembly robot teleoperation environment. By positioning the assembly and the robot and determining the positional relationship between the assembly and the robot, dense reconstruction based on vision is avoided, and the accuracy and visualization effect of the three-dimensional model are improved.

[0006] The technical solution of the present invention is as follows:

[0007] In one aspect, the present invention provides a digital twin modeling method for an assembly robot teleoperation environment, comprising the following steps:

[0008] Collect images of the teleoperation environment through an image acquisition device;

[0009] Identify the parts currently being assembled in the image; query the assembly order of the assembly, and obtain the list of assembled parts based on the parts currently being assembled;

[0010] Generate a three-dimensional model of the current assembly according to the list of assembled parts, and calculate the pose information of the current assembly in the coordinate system of the image acquisition device;

[0011] Load the robot 3D model and determine the coordinate transformation relationship between the robot coordinate system and the image acquisition device coordinate system;

[0012] The robot in the image is calibrated according to the coordinate transformation relationship to determine the position and posture information of the robot in the coordinate system of the image acquisition device;

[0013] Determine the relative position relationship between the current assembly and the robot according to the position and posture information of the current assembly in the image acquisition device coordinate system and the position and posture information of the robot in the image acquisition device coordinate system;

[0014] A digital twin model of the teleoperation environment is established based on the three-dimensional model of the current assembly, the three-dimensional model of the robot, and the relative position relationship between the current assembly and the robot.

[0015] As a preferred implementation, the image captured by the image acquisition device includes a depth image, and the step of identifying the component currently being assembled in the image is specifically as follows:

[0016] The regions of different parts of the assembly in the depth image are marked with different color labels to generate instance segmentation images;

[0017] The color labels of the instance segmentation images are counted to identify the parts currently being assembled.

[0018] As a preferred implementation, the step of generating a three-dimensional model of the current assembly according to the assembled parts list is specifically as follows:

[0019] Load the 3D models of each component in the assembled component list;

[0020] Determine the position of each component in the list on the assembly according to the predefined constraint relationship between the components of the assembly;

[0021] Adjust the position of the 3D model of each component according to the position of each component in the list on the assembly to generate a 3D model of the current assembly.

[0022] As a preferred implementation, the step of calculating the pose information of the current assembly in the image acquisition device coordinate system is specifically as follows:

[0023] Preprocess the depth image, remove the background, and retain the depth image of the current assembly;

[0024] The depth image of the current assembly is converted into an assembly point cloud using the internal parameters and imaging model of the image acquisition device;

[0025] Inputting the assembly point cloud into the point cloud feature extraction network to extract the point cloud features of the assembly;

[0026] The point cloud features of the assembly are input into the pre-trained pose estimation network, and the pose information of the assembly in the coordinate system of the image acquisition device is output.

[0027] As a preferred implementation, the pre-training process of the pose estimation network is:

[0028] Determine the initial information; extract and record the label pose information of the assembly 3D model point cloud at the initial perspective. The label pose information includes the rotation matrix R of each point in the 3D model point cloud. i and the offset matrix T i , i is the index of each point in the 3D model point cloud;

[0029] Point cloud conversion: extracting an assembly depth image of the assembly three-dimensional model at another perspective different from the initial perspective, and converting the assembly depth image into an assembly point cloud using the image acquisition device internal parameters and the imaging model;

[0030] Pose prediction: Input the assembly point cloud into the point cloud feature extraction network, output the point cloud point-by-point features, input the point cloud point-by-point features into the pose estimation network, and predict the pose prediction information of each point, including the predicted rotation matrix R′ i and the prediction offset matrix T′ i ;

[0031] Calculate the Euclidean distance between the pose prediction information and the label pose information of each point, and generate confidence based on the Euclidean distance; if the confidence is less than the set threshold, execute the image update step; if the confidence is greater than the set threshold or the number of training times reaches the set value, output the optimal pose prediction information of the assembly at the current perspective; determine whether the training is completed, if not, return to the point cloud conversion step, replace the assembly depth image at the next perspective to continue training, and execute the image update step if completed;

[0032] Image update; predict the rotation matrix R′ i and the prediction offset matrix T′ i As input, the assembly point cloud is subjected to displacement and rotation transformation, the three-dimensional coordinates of the assembly point cloud are updated, and the updated assembly point cloud is input into the point cloud feature extraction network for further training.

[0033] As a preferred embodiment, the image acquisition device is an RGB-D camera, and the acquired image also includes an RGB image; the step of determining the coordinate transformation relationship between the robot coordinate system and the image acquisition device coordinate system is specifically:

[0034] Setting positioning markers at the joints of the robot;

[0035] Control the robot to do intermittent motion at each point. At each intermittent motion point, the robot controller reads the coordinates P of the positioning marker in the robot coordinate system. i (x wi ,y wi ,z wi), and simultaneously identify the pixel coordinates Z of the feature points of the positioning marker in the RGB image i (u i ,v i );

[0036] According to the pixel coordinate Z of the feature point of the positioning marker in the RGB image i (u i ,v i ) and the imaging model of the RGB-D camera, and using the positional relationship between the depth camera and the color camera in the RGB-D camera, the three-dimensional coordinates P of the feature points of the positioning marker in the camera coordinate system are obtained. i '(x ci ,y ci ,z ci );

[0037] Obtain the coordinates P of the positioning markers of more than four intermittent motion points in the robot coordinate system i (x wi ,y wi ,z wi ) and the three-dimensional coordinates P of the feature points of the positioning marker in the camera coordinate system i '(x ci ,y ci ,z ci ), calculate the transformation matrix between the robot coordinate system and the camera coordinate system.

[0038] As a preferred implementation, in the step of controlling the robot to perform point-to-point intermittent motion, the trajectory of the point-to-point intermittent motion is set to be a plurality of squares.

[0039] In another aspect, the present invention provides a digital twin modeling system for an assembly robot teleoperation environment, comprising:

[0040] Image acquisition equipment, used to acquire images of the teleoperation environment;

[0041] The assembly digitization module is preloaded with the 3D models of each component in the assembly, the constraint relationships of each component, and the assembly sequence of the assembly;

[0042] The assembly reconstruction module is used to identify the parts currently being assembled in the image and query the assembly order of the assembly, obtain the list of assembled parts according to the parts currently being assembled, and generate a three-dimensional model of the current assembly according to the list of assembled parts;

[0043] Assembly positioning module, used to calculate the position and posture information of the current assembly in the coordinate system of the image acquisition device;

[0044] Robot digital module, pre-loaded with robot 3D model;

[0045] The robot positioning module is used to determine the coordinate conversion relationship between the robot coordinate system and the image acquisition device coordinate system; and to calibrate the robot in the image according to the coordinate conversion relationship to determine the position information of the robot in the image acquisition device coordinate system;

[0046] The digital twin model reconstruction module is used to determine the relative position relationship between the current assembly and the robot based on the posture information of the current assembly in the image acquisition device coordinate system and the posture information of the robot in the image acquisition device coordinate system; and to establish a digital twin model of the teleoperation environment based on the three-dimensional model of the current assembly, the three-dimensional model of the robot and the relative position relationship between the current assembly and the robot.

[0047] As a preferred implementation, the image acquisition device is an RGB-D camera.

[0048] On the other hand, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the digital twin modeling method of the remote operation environment of an assembly robot as described in any embodiment of the present invention is implemented.

[0049] The present invention has the following beneficial effects:

[0050] The present invention provides a digital twin modeling method for an assembly robot teleoperation environment, which identifies the parts being assembled, obtains a list of assembled parts using a pre-input assembly sequence, and reconstructs a high-precision three-dimensional model of the current assembly based on the list of assembled parts. The position information of the assembly and the robot in the coordinate system of the image acquisition device is calculated to obtain the relative position relationship between the robot and the assembly, and an accurate digital twin model of the teleoperation environment is established based on this relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of an embodiment of the present invention;

[0052] Figure 2 It is a schematic diagram of controlling a robot to perform intermittent point motion in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0055] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0056] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0057] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0058] Embodiment 1:

[0059] See also Figure 1 , this embodiment provides a digital twin modeling method for an assembly robot teleoperation environment, comprising the following steps:

[0060] Collect images of the teleoperation environment through an image acquisition device;

[0061] Identify the parts currently being assembled in the image; query the assembly order of the assembly, and obtain the list of assembled parts based on the parts currently being assembled;

[0062] Generate a three-dimensional model of the current assembly according to the list of assembled parts, and calculate the pose information of the current assembly in the coordinate system of the image acquisition device;

[0063] Load the robot 3D model and determine the coordinate transformation relationship between the robot coordinate system and the image acquisition device coordinate system;

[0064] The robot in the image is calibrated according to the coordinate transformation relationship to determine the position and posture information of the robot in the coordinate system of the image acquisition device;

[0065] Determine the relative position relationship between the current assembly and the robot according to the position and posture information of the current assembly in the image acquisition device coordinate system and the position and posture information of the robot in the image acquisition device coordinate system;

[0066] A digital twin model of the teleoperation environment is established based on the three-dimensional model of the current assembly, the three-dimensional model of the robot, and the relative position relationship between the current assembly and the robot.

[0067] This embodiment identifies the parts being assembled, obtains the list of assembled parts using the pre-input assembly order, and reconstructs a high-precision three-dimensional model of the current assembly based on the list of assembled parts. The position information of the assembly and the robot in the coordinate system of the image acquisition device is calculated to obtain the relative position relationship between the robot and the assembly, and based on this relationship, an accurate digital twin model of the teleoperation environment is established.

[0068] As a preferred implementation of this embodiment, the image captured by the image acquisition device includes a depth image, and the step of identifying the component currently being assembled in the image is specifically:

[0069] The regions of different parts of the assembly in the depth image are marked with different color labels to generate instance segmentation images;

[0070] The color labels of the instance segmentation images are counted to identify the parts currently being assembled.

[0071] As a preferred implementation of this embodiment, the step of generating a three-dimensional model of the current assembly according to the assembled parts list is specifically as follows:

[0072] Load the 3D models of each component in the assembled component list;

[0073] Determine the position of each component in the list on the assembly according to the predefined constraint relationship between the components of the assembly;

[0074] Adjust the position of the 3D model of each component according to the position of each component in the list on the assembly to generate a 3D model of the current assembly.

[0075] As a preferred implementation of this embodiment, the step of calculating the pose information of the current assembly in the image acquisition device coordinate system is specifically as follows:

[0076] Preprocess the depth image, remove the background, retain only the current assembly, and obtain the depth image of the current assembly;

[0077] The depth image of the current assembly is converted into a depth image point cloud using the internal parameters and imaging model of the image acquisition device, hereinafter referred to as the assembly point cloud;

[0078] Input the assembly point cloud into the PointNet++ point cloud feature extraction network to extract the point cloud features of the assembly;

[0079] The point cloud features of the assembly are input into the pre-trained Dense Fusion pose estimation network, and the pose information of the assembly in the coordinate system of the image acquisition device is output.

[0080] This embodiment matches the three-dimensional model of the assembly with the depth image, uses the PointNet++ network to extract point cloud features, and uses the pose estimation network Dense Fusion to estimate the pose information of the assembly to achieve the positioning of the assembly.

[0081] As a preferred implementation of this embodiment, the pre-training process of the pose estimation network is:

[0082] Determine the initial information; extract and record the label pose information of the assembly 3D model point cloud at the initial perspective. The label pose information includes the rotation matrix R of each point in the 3D model point cloud. i and the offset matrix T i , i is the index of each point in the 3D model point cloud;

[0083] Point cloud conversion: extracting an assembly depth image of the assembly three-dimensional model at another perspective different from the initial perspective, and converting the assembly depth image into an assembly point cloud using the image acquisition device internal parameters and the imaging model;

[0084] Pose prediction: Input the assembly point cloud into the PointNet++ point cloud feature extraction network, extract the geometric features of the assembly point cloud, generate dense point cloud point features, input the point cloud point features into the Dense Fusion pose estimation network, and predict the pose prediction information of each point. The pose prediction information includes the predicted rotation matrix R′ i and the prediction offset matrix T′ i ;

[0085] Calculate the Euclidean distance between the pose prediction information and the label pose information of each point, and generate confidence based on the Euclidean distance; if the confidence is less than the set threshold, execute the image update step; if the confidence is greater than the set threshold or the number of training times reaches the set value, output the optimal pose prediction information of the assembly at the current perspective; determine whether the training is completed, if not, return to the point cloud conversion step, replace the assembly depth image at the next perspective to continue training, and execute the image update step if completed;

[0086] Image update; predict the rotation matrix R′ i and the prediction offset matrix T′ i As input, the assembly point cloud is displaced and rotated to update the three-dimensional coordinates of the assembly point cloud, and the updated assembly point cloud is input into the PointNet++ point cloud feature extraction network for further training.

[0087] See also Figure 2 As a preferred implementation of this embodiment, the image acquisition device is an RGB-D camera, and the acquired image also includes an RGB image; the step of determining the coordinate transformation relationship between the robot coordinate system and the image acquisition device coordinate system is specifically as follows:

[0088] Set positioning markers (such as small balls of special shapes and colors) at the joints of the robot;

[0089] Control the robot to do intermittent motion at each intermittent motion point. The robot controller reads the positioning marker in the robot coordinate system (O w -X w Y w Z w ) under the coordinate P i (x wi ,y wi ,z wi ), and simultaneously identify the pixel coordinates Z of the feature points of the positioning marker in the RGB image i (u i ,v i );

[0090] According to the pixel coordinate Z of the feature point of the positioning marker in the RGB image i (u i ,v i ) and the imaging model of the RGB-D camera, and using the positional relationship between the depth camera and the color camera in the RGB-D camera, the feature points of the positioning marker are calculated in the camera coordinate system (O c -X c Y c Z c ) under the three-dimensional coordinates P i '(x ci ,y ci ,z ci );

[0091] Obtain the coordinates P of the positioning markers of more than four intermittent motion points in the robot coordinate system i (x wi ,y wi ,z wi ) and the three-dimensional coordinates P of the feature points of the positioning marker in the camera coordinate system i '(x ci ,y ci ,z ci ), calculate the transformation matrix between the robot coordinate system and the camera coordinate system, which can be expressed as:

[0092]

[0093] Among them, R is a 3*3 rotation matrix, and T is a 3*1 translation vector, which represents the direction and position of the robot coordinate system in the camera coordinate system.

[0094] As a preferred implementation of this embodiment, in the step of controlling the robot to perform intermittent point motion, in order to improve the positioning accuracy, the trajectory of the intermittent point motion can be set to multiple squares, and M can be solved by constructing and solving the least squares problem. wc , as shown below:

[0095]

[0096] Find M wc Then, the color camera coordinate system O is obtained. c -X c Y c Z c With the robot base coordinate system O w -X w Y w Z w The conversion relationship between them is established to complete the robot orientation calibration and obtain the position and posture information of the robot's three-dimensional model.

[0097] In another aspect, the present invention provides a digital twin modeling system for an assembly robot teleoperation environment, comprising:

[0098] Image acquisition equipment, used to acquire images of the teleoperation environment;

[0099] The assembly digitization module is preloaded with the 3D models of each component in the assembly, the constraint relationships of each component, and the assembly sequence of the assembly;

[0100] The assembly reconstruction module is used to identify the parts currently being assembled in the image and query the assembly order of the assembly, obtain the list of assembled parts according to the parts currently being assembled, and generate a three-dimensional model of the current assembly according to the list of assembled parts;

[0101] Assembly positioning module, used to calculate the position and posture information of the current assembly in the coordinate system of the image acquisition device;

[0102] Robot digital module, pre-loaded with robot 3D model;

[0103] The robot positioning module is used to determine the coordinate conversion relationship between the robot coordinate system and the image acquisition device coordinate system; and to calibrate the robot in the image according to the coordinate conversion relationship to determine the position information of the robot in the image acquisition device coordinate system;

[0104] The digital twin model reconstruction module is used to determine the relative position relationship between the current assembly and the robot based on the posture information of the current assembly in the image acquisition device coordinate system and the posture information of the robot in the image acquisition device coordinate system; and to establish a digital twin model of the teleoperation environment based on the three-dimensional model of the current assembly, the three-dimensional model of the robot and the relative position relationship between the current assembly and the robot.

[0105] As a preferred implementation, the image acquisition device is an RGB-D camera.

[0106] On the other hand, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the digital twin modeling method of the remote operation environment of an assembly robot as described in any embodiment of the present invention is implemented.

[0107] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A digital twin modeling method for an assembly robot teleoperation environment, characterized in that: The following steps are involved: Collect images of the teleoperation environment through an image acquisition device; Identify the component currently being assembled in the image; Query the assembly order of the assembly and obtain the list of assembled parts based on the parts currently being assembled; Generate a three-dimensional model of the current assembly according to the list of assembled parts, and calculate the pose information of the current assembly in the coordinate system of the image acquisition device; Load the robot 3D model and determine the coordinate transformation relationship between the robot coordinate system and the image acquisition device coordinate system; The robot in the image is calibrated according to the coordinate transformation relationship to determine the position and posture information of the robot in the coordinate system of the image acquisition device; Determine the relative position relationship between the current assembly and the robot according to the position and posture information of the current assembly in the image acquisition device coordinate system and the position and posture information of the robot in the image acquisition device coordinate system; A digital twin model of the teleoperation environment is established based on the 3D model of the current assembly, the 3D model of the robot, and the relative position relationship between the current assembly and the robot; The step of calculating the position and posture information of the current assembly in the image acquisition device coordinate system is specifically as follows: Preprocess the depth image, remove the background, and retain the depth image of the current assembly; The depth image of the current assembly is converted into an assembly point cloud using the internal parameters and imaging model of the image acquisition device; Inputting the assembly point cloud into the point cloud feature extraction network to extract the point cloud features of the assembly; Input the point cloud features of the assembly into the pre-trained pose estimation network, and output the pose information of the assembly in the coordinate system of the image acquisition device; Among them, the pre-training process of the pose estimation network is: Determine the initial information; extract and record the label pose information of the assembly 3D model point cloud at the initial perspective. The label pose information includes the rotation matrix R of each point in the 3D model point cloud. i and the offset matrix T i , i is the index of each point in the 3D model point cloud; Point cloud conversion: extracting an assembly depth image of the assembly three-dimensional model at another perspective different from the initial perspective, and converting the assembly depth image into an assembly point cloud using the image acquisition device internal parameters and the imaging model; Pose prediction: Input the assembly point cloud into the point cloud feature extraction network, output the point cloud point-by-point features, input the point cloud point-by-point features into the pose estimation network, and predict the pose prediction information of each point, including the predicted rotation matrix R′ i and the prediction offset matrix T′ i ; Calculate the Euclidean distance between the pose prediction information and the label pose information of each point, and generate confidence based on the Euclidean distance; if the confidence is less than the set threshold, execute the image update step; if the confidence is greater than the set threshold or the number of training times reaches the set value, output the optimal pose prediction information of the assembly at the current perspective; determine whether the training is completed, if not, return to the point cloud conversion step, replace the assembly depth image at the next perspective to continue training, and execute the image update step if completed; Image update; predict the rotation matrix R′ i and the prediction offset matrix T′ i As input, the assembly point cloud is subjected to displacement and rotation transformation, the three-dimensional coordinates of the assembly point cloud are updated, and the updated assembly point cloud is input into the point cloud feature extraction network for further training.

2. A digital twin modeling method for an assembly robot teleoperation environment according to claim 1, characterized in that: The image captured by the image acquisition device includes a depth image, and the step of identifying the component currently being assembled in the image is specifically: The regions of different parts of the assembly in the depth image are marked with different color labels to generate instance segmentation images; The color labels of the instance segmentation images are counted to identify the parts currently being assembled.

3. The digital twin modeling method of the assembly robot teleoperation environment according to claim 2 is characterized in that: The steps of generating a three-dimensional model of the current assembly according to the assembled parts list are specifically as follows: Load the 3D models of each component in the assembled component list; Determine the position of each component in the list on the assembly according to the predefined constraint relationship between the components of the assembly; Adjust the position of the 3D model of each component according to the position of each component in the list on the assembly to generate a 3D model of the current assembly.

4. The digital twin modeling method of the assembly robot teleoperation environment according to claim 2 is characterized in that: The image acquisition device is an RGB-D camera, and the acquired image also includes an RGB image; the step of determining the coordinate conversion relationship between the robot coordinate system and the image acquisition device coordinate system is specifically as follows: Setting positioning markers at the joints of the robot; Control the robot to do intermittent motion at each point. At each intermittent motion point, the robot controller reads the coordinates P of the positioning marker in the robot coordinate system. i (x wi ,y wi ,z wi ), and simultaneously identify the pixel coordinates Z of the feature points of the positioning marker in the RGB image i (u i ,v i ); According to the pixel coordinate Z of the feature point of the positioning marker in the RGB image i (u i ,v i ) and the imaging model of the RGB-D camera, and using the positional relationship between the depth camera and the color camera in the RGB-D camera, the three-dimensional coordinates P of the feature points of the positioning marker in the camera coordinate system are obtained. i '(x ci ,y ci ,z ci ); Obtain the coordinates P of the positioning markers of more than four intermittent motion points in the robot coordinate system i (x wi ,y wi ,z wi ) and the three-dimensional coordinates P of the feature points of the positioning marker in the camera coordinate system i '(x ci ,y ci ,z ci ), calculate the transformation matrix between the robot coordinate system and the camera coordinate system.

5. The digital twin modeling method of the assembly robot teleoperation environment according to claim 4 is characterized in that: In the step of controlling the robot to perform point-to-point intermittent motion, the trajectory of the point-to-point intermittent motion is set to be a plurality of squares.

6. A digital twin modeling system for an assembly robot teleoperation environment, characterized in that: include: Image acquisition equipment, used to acquire images of the teleoperation environment; The assembly digitization module is preloaded with the 3D models of each component in the assembly, the constraint relationships of each component, and the assembly sequence of the assembly; The assembly reconstruction module is used to identify the parts currently being assembled in the image and query the assembly order of the assembly, and obtain the list of assembled parts based on the parts currently being assembled; And generate a 3D model of the current assembly based on the list of assembled parts; Assembly positioning module, used to calculate the position and posture information of the current assembly in the coordinate system of the image acquisition device; Robot digital module, pre-loaded with robot 3D model; The robot positioning module is used to determine the coordinate conversion relationship between the robot coordinate system and the image acquisition device coordinate system; and to calibrate the robot in the image according to the coordinate conversion relationship to determine the position information of the robot in the image acquisition device coordinate system; A digital twin model reconstruction module is used to determine the relative position relationship between the current assembly and the robot based on the position and posture information of the current assembly in the image acquisition device coordinate system and the position and posture information of the robot in the image acquisition device coordinate system; A digital twin model of the teleoperation environment is established based on the 3D model of the current assembly, the 3D model of the robot, and the relative position relationship between the current assembly and the robot; The step of calculating the position and posture information of the current assembly in the image acquisition device coordinate system is specifically as follows: Preprocess the depth image, remove the background, and retain the depth image of the current assembly; The depth image of the current assembly is converted into an assembly point cloud using the internal parameters and imaging model of the image acquisition device; Inputting the assembly point cloud into the point cloud feature extraction network to extract the point cloud features of the assembly; Input the point cloud features of the assembly into the pre-trained pose estimation network, and output the pose information of the assembly in the coordinate system of the image acquisition device; Among them, the pre-training process of the pose estimation network is: Determine the initial information; extract and record the label pose information of the assembly 3D model point cloud at the initial perspective. The label pose information includes the rotation matrix R of each point in the 3D model point cloud. i and the offset matrix T i , i is the index of each point in the 3D model point cloud; Point cloud conversion: extracting an assembly depth image of the assembly three-dimensional model at another perspective different from the initial perspective, and converting the assembly depth image into an assembly point cloud using the image acquisition device internal parameters and the imaging model; Pose prediction: Input the assembly point cloud into the point cloud feature extraction network, output the point cloud point-by-point features, input the point cloud point-by-point features into the pose estimation network, and predict the pose prediction information of each point, including the predicted rotation matrix R′ i and the prediction offset matrix T′ i ; Calculate the Euclidean distance between the pose prediction information and the label pose information of each point, and generate confidence based on the Euclidean distance; if the confidence is less than the set threshold, execute the image update step; if the confidence is greater than the set threshold or the number of training times reaches the set value, output the optimal pose prediction information of the assembly at the current perspective; determine whether the training is completed, if not, return to the point cloud conversion step, replace the assembly depth image at the next perspective to continue training, and execute the image update step if completed; Image update; predict the rotation matrix R′ i and the prediction offset matrix T′ i As input, the assembly point cloud is subjected to displacement and rotation transformation, the three-dimensional coordinates of the assembly point cloud are updated, and the updated assembly point cloud is input into the point cloud feature extraction network for further training.

7. A digital twin modeling system for assembly robot teleoperation environment according to claim 6, characterized in that: The image acquisition device is an RGB-D camera.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the digital twin modeling method of the assembly robot teleoperation environment is implemented as described in any one of claims 1 to 5.