Masonry robot virtual-real cooperative control method and system based on digital twinning

Abstract

The invention discloses a masonry robot virtual-real cooperative control method and system based on digital twinning, and aims to realize high-precision and high-safety automatic masonry of a coke oven by a robot. The method comprises the following steps: firstly, constructing a digital twin system of a masonry robot, and setting a masonry robot entity industrial control process: firstly, conveying refractory bricks to a specified photographing position by a brick conveying platform, and then photographing and identifying by a truss clamping jaw; the truss clamping jaw moves to a refractory brick grabbing position for grabbing, and meanwhile, the mechanical arm with the slurry pipe is used for spraying slurry; and after the movable truss moves to the refractory brick placement position, accurate placement is identified. In the virtual-real cooperative control process, the virtual entity model is updated in real time through state information acquired by a sensor of a real system; making and optimizing a control strategy in the virtual environment; and real-time decision making is carried out by utilizing the virtual entity model, so that the operation of the physical entity is adjusted.

Description

Technical Field

[0001] The present invention relates to the technical field of masonry robots, and more specifically, to a virtual-reality collaborative control method and system for a masonry robot based on digital twins. Background Art

[0002] The production life of a coke oven is about 20 years, and the construction, replacement and repair of a coke oven are sustainable. The coke oven masonry process is complicated, the labor efficiency is low, the masonry workload is large and the precision requirements are high. The robot intelligent masonry of a large coke oven body can not only save a lot of labor, the operation process is energy-saving and environmentally friendly, but also greatly shorten the construction period, put into production as soon as possible, and generate huge economic benefits.

[0003] The Chinese invention patents with application numbers CN202310035929.7 and CN202311194555.X disclose the technical solutions of intelligent masonry equipment or methods for coke ovens, and the overall solution of masonry robots, focusing on improving the masonry efficiency and quality of the machines, but the specific control of the masonry robots is not described in detail. The masonry process of coke ovens is complicated, and there are more than 700 types of refractory bricks for a single coke oven, requiring high precision. Once the control accuracy of the masonry robot is difficult to meet the masonry process requirements, the quality and safety of the masonry will be difficult to guarantee.

[0004] Application No. 201911120960.0 discloses a reconfigurable system and method for industrial robot manufacturing system based on digital twin, which includes: data acquisition unit, five-dimensional digital twin model unit, twin dependency tree, reconstruction strategy unit and virtual visualization simulation platform. This technical solution realizes reconstruction design, verification and optimization without interfering with the normal production tasks of the manufacturing system. However, the system only involves system reconstruction strategy and lacks joint control method with physical industrial control process.

[0005] In summary, there is currently no method and system for virtual-reality collaborative control of coke oven masonry robots based on digital twins that can improve the control accuracy of coke oven masonry robots. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention proposes a virtual-reality collaborative control method and system for a masonry robot based on digital twins, and optimizes the control process of the coke oven masonry robot to enable the robot to automatically masonry the coke oven with high precision and high safety.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a virtual-reality collaborative control method for a masonry robot based on digital twins, comprising the following steps:

[0008] Step S1, constructing a digital twin system of a masonry robot; the digital twin system of the masonry robot includes a physical entity module, a virtual entity module, a connection module, a twin data module and a service module; the physical entity module reflects the on-site coke oven automatic masonry robot and its surrounding environment, including the geometric size information, movement mode and corresponding material properties of the masonry robot; the virtual entity module establishes a virtual geometric model based on information such as the physical entity structure shape, material properties and assembly relationship, performs three-dimensional simplification and format conversion and imports the model into Unity3D; the connection module is responsible for data interaction between various parts; the twin data module is used to store, extract, convert and process twin data; the twin data includes relevant data of physical entities, virtual entities and service systems, domain knowledge and its fusion data, and is continuously updated and optimized as real-time data is generated; the service module includes a collection of modules and data at all levels, and integrates various information systems such as evaluation, control and optimization, and provides intelligent operation, precise control and reliable operation and maintenance services based on physical entity and virtual entity models;

[0009] Step S2, setting the industrial control process of the masonry robot entity: the masonry robot entity is mainly divided into three functional modules: brick delivery platform, truss gripper recognition and grabbing bricks, and mechanical arm mud application; first, the brick delivery platform transports the refractory bricks to the designated photo position, and then the truss gripper with an industrial camera takes photos and recognizes them; then the truss gripper moves to the refractory brick grabbing position to grab them, and the mechanical arm with a mud pipe sprays the bottom mud; after the mobile truss moves to the top of the refractory brick placement position, the side mud is sprayed and the industrial camera recognizes and accurately places it; the above actions are repeated repeatedly to complete the masonry task;

[0010] Step S3, implement virtual-reality collaborative control of the masonry robot: obtain sensor data from the real system in real time, and fuse these data with the virtual model; use the status information obtained by the sensor to update the virtual entity model; formulate and optimize the control strategy in the virtual environment; use the virtual entity model to make real-time decisions to adjust the operation of the physical entity; first set the data including the masonry task, the initial position and speed of the robot, start the masonry robot and the digital twin system of the masonry robot, obtain the current data information of the masonry robot, map it to the digital twin in Unity3D, and start masonry at the same time; the digital twin system of the masonry robot determines whether the masonry robot is running stably and whether the control accuracy is accurate; if it is stable, continue the masonry task; if it is unstable, perform simulation adjustment in the digital twin system of the masonry robot, simulate and calculate the appropriate posture, speed and running trajectory and feed back to the physical masonry robot to adjust its running state until the masonry task is completed.

[0011] By adopting the above technical scheme, it is possible to realize online real-time monitoring of the operating status of the masonry robot and predict possible system failures, so that corrections and compensations can be made in time, reducing the risks brought by errors or failures in actual operations, and providing guarantees for the safe and reliable operation of the system to ensure the quality and safety of masonry; according to the structural strength of easily consumable parts, the parts that may be damaged can be predicted in advance, and they can be replaced in time in advance to avoid the robot working in an unstable operating state, so as to prevent robot failures, which is of great significance to the intelligent operation and maintenance of masonry robots.

[0012] Furthermore, the data interaction between the various parts in the connection module is completed using methods including but not limited to PLC communication, serial port communication and Socket communication.

[0013] Furthermore, the masonry robot includes a truss robot, a brick delivery platform, a masonry platform, and a robot control cabinet; the truss robot includes a truss body and a ground rail; the truss body is divided into a gripper truss and a robotic arm truss; the gripper truss includes a Y-axis, the Y-axis includes a slide, and the Y-axis slides along the X-axis; the slide includes a lifting rod, and the slide slides along the Y-axis; the lifting rod slides along the Z-axis and includes a gripper; the gripper rotates around the Z-axis, and an industrial depth camera is mounted on it; the robotic arm truss includes a truss slide and a robotic arm.

[0014] By adopting the above technical solution, the architecture of the masonry robot is clear, which can better meet the smooth progress of the virtual-real collaborative control process of the masonry process.

[0015] Furthermore, in step S2, the truss clamp with an industrial camera takes pictures to identify and obtain the brick model, gripping position and placement position; the machine vision three-dimensional point cloud method is used to find the center of the refractory brick to determine the gripping position and accuracy.

[0016] Furthermore, in step S3, the digital twin combines the Unity3D built-in collider with the AABB hierarchical bounding box to perform collision detection; first, the built-in Box Collider detector is used to perform preliminary detection on the target, and for the detected target, an AABB hierarchical bounding box tree is constructed to perform detection level by level; the tree traversal method is used to determine whether there is any intersection between the two targets; if there is no intersection, the two objects do not collide, and if there is an intersection, a fine detection is performed to complete the intersection detection of the basic graphics elements on the leaf nodes.

[0017] By adopting the above technical solution, the collision detection method in the digital twin makes up for the shortcomings of the built-in collider of Unity3D, making the bounding box of the object more accurate, thereby improving the accuracy of collision detection.

[0018] In a second aspect, the present invention provides a masonry robot virtual-real collaborative control system based on digital twins, including a masonry robot digital twin system and a masonry robot physical industrial control system;

[0019] The digital twin system of the masonry robot includes a physical entity module, a virtual entity module, a connection module, a twin data module and a service module; the physical entity module reflects the on-site coke oven automatic masonry robot and its surrounding environment, including the geometric size information, movement mode and corresponding material properties of the masonry robot; the virtual entity module establishes a virtual geometric model based on information such as the physical entity structure, shape, material properties and assembly relationship, performs three-dimensional simplification and format conversion, and imports the model into Unity3D; the connection module is responsible for data interaction between various parts; the twin data module is used to store, extract, convert and process twin data; the twin data includes relevant data of physical entities, virtual entities, and service systems, domain knowledge and its fusion data, and is continuously updated and optimized as real-time data is generated; the service module includes the collection application of modules and data at all levels, and integrates various information systems such as evaluation, control, and optimization, providing intelligent operation, precise control and reliable operation and maintenance services based on physical entity and virtual entity models;

[0020] The masonry robot entity industrial control system performs module arrangement based on step S2 as described above;

[0021] The masonry robot digital twin system and the masonry robot physical industrial control system perform virtual-reality collaborative control through step S3 as described above.

[0022] Compared with the prior art, the present invention has the following technical effects:

[0023] (1) The present invention can realize online real-time monitoring of the operating status of the masonry robot and predict possible system failures, so that corrections and compensations can be made in a timely manner, reducing the risks brought by errors or failures in actual operations, and providing guarantees for the safe and reliable operation of the system to ensure the quality and safety of masonry. It can predict in advance the parts that may be damaged based on the structural strength of the easily damaged parts, and can replace them in time in advance to avoid the robot working in an unstable operating state and prevent robot failures. This has a great effect on the intelligent operation and maintenance of the masonry robot.

[0024] (2) The present invention changes the original single control mode of the masonry robot and realizes the virtual-real collaborative control of the masonry robot. The virtual-real collaborative control can better realize human-machine interaction, enabling the operator to more intuitively understand and control the robot's behavior, thereby improving the collaboration efficiency;

[0025] (3) The virtual environment of the present invention can provide the robot with richer simulation information, enabling the robot to make better real-time decisions in complex environments. Through virtual simulation, the control algorithm can be tested and optimized in different scenarios, thereby improving the efficiency and reliability of the robot in actual operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention and its features and advantages will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0027] Figure 1 This is a framework diagram of a digital twin system of a masonry robot in one embodiment of the present invention;

[0028] Figure 2 A diagram showing the main components of a masonry robot in one embodiment of the present invention;

[0029] Figure 3 It is a flow chart of the physical industrial control of a masonry robot in one embodiment of the present invention;

[0030] Figure 4 It is a flow chart of virtual-real collaborative control of a masonry robot in one embodiment of the present invention;

[0031] Figure 5 A flow chart of mixed-level bounding box collision detection in one embodiment of the present invention;

[0032] Figure 6 This is a comparison diagram of the effects of Unity3D built-in collider and hybrid hierarchical bounding box collision detection in one embodiment of the present invention;

[0033] Among them, 1. Ground rail; 2. Gripper truss; 3. Robot arm truss; 4. Y-axis; 5. Slide; 6. Z-axis; 7. X-axis. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0035] In the following detailed description, many specific details are described to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the known methods (such as modeling using Unity3D) are not shown in detail to avoid blurring the main purpose of the present invention; and the undetailed technologies involved in the following effect embodiments are prior arts that can be retrieved.

[0036] In addition, the order of execution of actions, steps, etc. in the devices and methods shown in the claims, specifications and drawings can be implemented in any order as long as there is no special explicit limitation on the order and the output of the previous processing is not used in the subsequent processing.

[0037] Example

[0038] This embodiment provides a virtual-reality collaborative control method and system for a masonry robot based on digital twins, and the control process includes the following steps:

[0039] Step S1: construct a masonry robot digital twin system; see Figure 1 The digital twin system of the masonry robot includes a physical entity module, a virtual entity module, a connection module, a twin data module and a service module; the physical entity module reflects the on-site coke oven automatic masonry robot and its surrounding environment, including the geometric size information, movement mode and corresponding material properties of the masonry robot; the virtual entity module is a virtual geometric model established based on the physical entity structure, shape, material properties and assembly relationship information, and performs three-dimensional simplification and format conversion and imports the model into Unity3D; the connection module is responsible for the data interaction between the various parts. The data interaction between the various parts is preferably completed by PLC communication, serial communication and Socket communication. The twin data module is used to store, extract, convert and process the twin data; the twin data includes the relevant data of the physical entity, virtual entity and service system, domain knowledge and its fusion data, and is continuously updated and optimized as the real-time data is generated; the service module includes the collection application of modules and data at all levels, integrated evaluation, control, optimization and other information systems, and provides intelligent operation, precise control and reliable operation and maintenance services based on physical entity and virtual entity models.

[0040] Step S2, set up the industrial control process of the masonry robot entity: the masonry robot entity is mainly divided into three functional modules: brick delivery platform, truss gripper recognition and brick grabbing, and mechanical arm for mud smearing. Figure 2 , the masonry robot exemplarily includes a truss robot, a brick delivery platform, a masonry platform, and a robot control cabinet; the truss robot includes a truss body and a ground rail 1; the truss body is divided into a gripper truss 2 and a robotic arm truss 3; the gripper truss includes a Y-axis 4, the Y-axis includes a slide 5, and the Y-axis 4 slides along the X-axis 7; the slide 5 includes a lifting rod, and the slide 5 slides along the Y-axis 4; the lifting rod slides along the Z-axis 6 and includes a gripper; the gripper rotates around the Z-axis 6, and an industrial depth camera is mounted on it; the robotic arm truss includes a truss slide and a robotic arm. The robotic arm example is a UR10 robotic arm. The UR10 robotic arm is a 6-axis robotic arm, with a base, a large arm, a small arm, a first wrist, a second wrist, and a third wrist (the 6-axis parent-child levels decrease in sequence, and the third wrist is connected to the spray gun).

[0041] See also Figure 3First, the brick delivery platform transports the refractory bricks to the designated photo position, and then the truss gripper with an industrial camera takes photos for identification; more specifically, the identification mainly identifies the brick model, the grabbing position and the placement position; the machine vision three-dimensional point cloud method is used to find the center of the refractory brick to determine the position and accuracy of the gripping. The truss gripper then moves to the refractory brick grabbing position to grab the brick, and the bottom surface mud is sprayed by the mechanical arm with a mud pipe; after the mobile truss moves to the top of the refractory brick placement position, the side mud is sprayed and the industrial camera identifies and accurately places it; the above actions are repeated repeatedly to complete the masonry task.

[0042] Step S3, implement virtual-real collaborative control of the masonry robot: obtain sensor data from the real system in real time, and fuse these data with the virtual model; use the status information obtained by the sensor to update the virtual entity model to improve its accuracy; formulate and optimize the control strategy in the virtual environment; use the virtual entity model to make real-time decisions to adjust the operation of the physical entity. While physically controlling the masonry robot, the digital twin technology is used to map the masonry robot in real time in the digital twin to monitor the real-time motion status of the masonry robot. If an abnormality occurs, it can be discovered in time and the operating status of the physical robot can be adjusted through twin simulation. Through predictions in the virtual environment, possible risks and problems can be avoided in advance, improving the safety and stability of the system.

[0043] See also Figure 4 First, set the data including the masonry task, the robot's initial position and speed, start the masonry robot and the masonry robot digital twin system, obtain the current data information of the masonry robot, map it to the digital twin in Unity3D, and start masonry at the same time; the masonry robot digital twin system determines whether the masonry robot is running stably and whether the control accuracy is accurate; if it is stable, continue the masonry task; if it is unstable, perform simulation adjustments in the masonry robot digital twin system, calculate the appropriate posture, speed and running trajectory through simulation and feed back to the physical masonry robot to adjust its running status until the masonry task is completed.

[0044] Unity3D provides a collision detection capability based on bounding boxes, which can be divided into two-dimensional colliders and three-dimensional colliders. Two-dimensional colliders are suitable for detecting objects in a plane, and three-dimensional colliders are suitable for detecting objects in space. You only need to add a corresponding collider component to the object to perform collision detection. The colliders in Unity3D can be divided into Collider, Capsule Collider, Sphere Collider, MeshCollider chain, etc. according to the shape of the bounding box. However, the built-in colliders of Unity3D still have the following limitations: the built-in colliders are all implemented with a single bounding box, which is a simple form of collision detection. This detection is not accurate, does not have a hierarchical tree structure, and cannot perform more precise detection of objects.

[0045] This embodiment uses a new hybrid hierarchical bounding box technology for collision detection: Hybrid hierarchical bounding box collision detection combines the built-in collider of Unity3D with the hierarchical AABB bounding box. The detection process is as follows: Figure 5 As shown in the figure, a hybrid hierarchical bounding box collision detection based on Unity3D built-in collider and AABB hierarchical bounding box is implemented. First, the built-in BoxCollider detector is used to perform preliminary detection on the target. For the detected target, an AABB hierarchical bounding box tree is constructed and detected level by level. The tree traversal method is used to determine whether there is an intersection between the two targets. If there is no intersection, the two objects do not collide. If there is an intersection, a fine detection is performed to complete the intersection detection of basic primitives on the leaf node. The proposed detection algorithm can make up for the shortcomings of Unity3D built-in collider, improve the accuracy of the algorithm, and realize accurate detection of collisions.

[0046] The principle of hierarchical bounding box technology is to construct a bounding box at the outermost edge of the object and use it as the root node, then divide it into several smaller units and construct a bounding box with it as the child node. The above steps are repeated until it cannot be divided, and finally a bounding box tree is formed. The hierarchical bounding box tree method is used to determine the collision of the model, that is, first determine whether the two targets intersect. If the two targets do not intersect, it is determined that the two targets will not collide. If the two targets intersect, then check whether the child nodes intersect until the two leaf nodes of the two trees are traversed. The use of hierarchical bounding box technology can effectively improve the accuracy of conflict detection and effectively reduce the complexity of detection.

[0047] The collision detection code is as follows:

[0048] Each collider has a bounding box, and we can get the AABB from the collider. (AABB is the smallest rectangle that encloses an object (a cube in 3D), with each face parallel to the coordinate axis) For the built-in collision box, you can get it as follows:

[0049] void GetAABB(Collider collider, out Vector3 min, out Vector3 max)

[0050] {

[0051] Bounds bounds = collider.bounds;

[0052] min = bounds.min;

[0053] max = bounds.max;

[0054] }

[0055] The judgment logic of AABB intersection is as follows:

[0056] bool IsAABBColliding(Vector3 minA, Vector3 maxA, Vector3 minB,Vector3 maxB)

[0057] {

[0058] return (minA.x <= maxB.x && maxA.x >= minB.x) &&

[0059] (minA.y <= maxB.y && maxA.y >= minB.y) &&

[0060] (minA.z <= maxB.z && maxA.z >= minB.z);

[0061] }

[0062] See also Figure 6 The first and second left pictures are the results of collision detection using Unity3D built-in collider and hybrid hierarchical bounding box for the same position. From the collision detection results, we can see that Unity3D built-in collider detects collision at a position where no collision occurs, which has a large detection error, while the result of hybrid hierarchical bounding box collision detection is more accurate.

[0063] Taking the masonry construction of the combustion chamber area as an example, the specific operation of the virtual-reality collaborative control system of the masonry robot based on digital twins is explained:

[0064] 1) Prepare the preparatory work and start the masonry robot and its twin system;

[0065] 2) After setting the masonry tasks according to the masonry plan, the masonry robot starts masonry, and the twin system performs real-time monitoring and prediction;

[0066] 3) If the masonry is stable and normal, the twin system will not interfere and only perform real-time monitoring. However, if abnormal masonry is found or the masonry accuracy does not meet the masonry process requirements, timely intervention will be carried out;

[0067] 4) The twin system will first use simulation to discover possible errors in advance, such as robot collisions, large masonry errors, etc.; the operating status of the masonry robot can be modified on the twin system. The appropriate posture, speed and other data are first calculated through simulation, and then fed back to the physical masonry robot to modify the operating status.

[0068] 5) The process of constructing a hierarchical bounding box tree: The first step is to find the AABB bounding box of the detected object as the root node of the tree; the second step is to find the longest axis of the bounding box, which is the splitting axis. The midpoint of this axis is used as the splitting point, and a plane perpendicular to the splitting axis is made through the splitting point, which is the splitting plane. The triangular facets of the model are divided into two subsets according to the splitting plane. For the triangles located on the splitting plane, they are divided into a set with fewer triangles to make the structure of the binary tree as balanced as possible; the third step is to construct bounding boxes for the divided sets respectively as the left and right child nodes of the parent node, and repeat the process of steps 2 and 3; the fourth step is to end the construction process until all leaf nodes are basic triangle primitives.

[0069] 6) The twin system, as a virtual control method, plays a role in detection and compensation for physical control, ensuring that the masonry task can be completed smoothly.

[0070] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0071] In summary, the present invention provides a virtual-reality collaborative control method and system for a masonry robot based on digital twins, so as to realize the automatic masonry of coke ovens with high precision and high safety by the robot. First, a digital twin system of a masonry robot is constructed, and a physical industrial control process of a masonry robot is set: the brick delivery platform first transports the refractory bricks to the designated photo position, and then the truss gripper takes photos for identification; then the truss gripper moves to the refractory brick grabbing position for grabbing, and at the same time, a mechanical arm with a mud pipe sprays mud; after the truss moves to the refractory brick placement position, it identifies and accurately places the bricks. During the virtual-reality collaborative control process, the virtual entity model is updated in real time through the status information obtained by the sensors of the real system; the control strategy is formulated and optimized in the virtual environment; and the virtual entity model is used to make real-time decisions to adjust the operation of the physical entity.

[0072] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0073] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the systems and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A virtual-reality collaborative control method for masonry robots based on digital twins, characterized in that: The following steps are involved: Step S1, constructing a digital twin system of a masonry robot; the digital twin system of the masonry robot includes a physical entity module, a virtual entity module, a connection module, a twin data module and a service module; the physical entity module reflects the on-site coke oven automatic masonry robot and its surrounding environment, including the geometric size information, movement mode and corresponding material properties of the masonry robot; the virtual entity module establishes a virtual geometric model based on information such as the physical entity structure shape, material properties and assembly relationship, performs three-dimensional simplification and format conversion and imports the model into Unity3D; the connection module is responsible for data interaction between various parts; the twin data module is used to store, extract, convert and process twin data; the twin data includes relevant data of physical entities, virtual entities and service systems, domain knowledge and its fusion data, and is continuously updated and optimized as real-time data is generated; the service module includes a collection of modules and data at all levels, and integrates various information systems such as evaluation, control and optimization, and provides intelligent operation, precise control and reliable operation and maintenance services based on physical entity and virtual entity models; Step S2, setting the industrial control process of the masonry robot entity: the masonry robot entity is mainly divided into three functional modules: brick delivery platform, truss gripper recognition and grabbing bricks, and mechanical arm mud application; first, the brick delivery platform transports the refractory bricks to the designated photo position, and then the truss gripper with an industrial camera takes photos and recognizes them; then the truss gripper moves to the refractory brick grabbing position to grab them, and the mechanical arm with a mud pipe sprays the bottom mud; after the mobile truss moves to the top of the refractory brick placement position, the side mud is sprayed and the industrial camera recognizes and accurately places it; the above actions are repeated repeatedly to complete the masonry task; Step S3, implement virtual-reality collaborative control of the masonry robot: obtain sensor data from the real system in real time, and fuse these data with the virtual model; use the status information obtained by the sensor to update the virtual entity model; formulate and optimize the control strategy in the virtual environment; use the virtual entity model to make real-time decisions to adjust the operation of the physical entity; first set the data including the masonry task, the initial position and speed of the robot, start the masonry robot and the digital twin system of the masonry robot, obtain the current data information of the masonry robot, map it to the digital twin in Unity3D, and start masonry at the same time; the digital twin system of the masonry robot determines whether the masonry robot is running stably and whether the control accuracy is accurate; if it is stable, continue the masonry task; if it is unstable, perform simulation adjustment in the digital twin system of the masonry robot, simulate and calculate the appropriate posture, speed and running trajectory and feed back to the physical masonry robot to adjust its running state until the masonry task is completed.

2. According to claim 1, a virtual-reality collaborative control method of a masonry robot based on digital twins is characterized in that: The data interaction between the various parts in the connection module is completed by using, including but not limited to, PLC communication, serial port communication and Socket communication.

3. According to the method of virtual-reality collaborative control of a masonry robot based on digital twins according to claim 1, it is characterized in that: The masonry robot includes a truss robot, a brick delivery platform, a masonry platform, and a robot control cabinet; the truss robot includes a truss body and a ground rail; the truss body is divided into a gripper truss and a robotic arm truss; the gripper truss includes a Y-axis, the Y-axis includes a slide, and the Y-axis slides along the X-axis; the slide includes a lifting rod, and the slide slides along the Y-axis; the lifting rod slides along the Z-axis and includes a gripper; the gripper rotates around the Z-axis, and an industrial depth camera is mounted on it; the robotic arm truss includes a truss slide and a robotic arm.

4. A virtual-reality collaborative control method for a masonry robot based on digital twins according to claim 1 or 3, characterized in that: In step S2, the truss gripper with an industrial camera takes pictures to identify the brick model, the gripping position and the placement position; the machine vision three-dimensional point cloud method is used to find the center of the refractory brick to determine the gripping position and accuracy.

5. The virtual-reality collaborative control method of a masonry robot based on digital twin according to claim 1 is characterized in that: In step S3, the digital twin combines the Unity3D built-in collider with the AABB hierarchical bounding box to perform collision detection; first, the built-in Box Collider detector is used to perform preliminary detection on the target, and for the detected target, an AABB hierarchical bounding box tree is constructed to perform detection level by level; a tree traversal method is used to determine whether there is an intersection between two targets; If they do not intersect, the two objects do not collide. If they intersect, they enter a detailed detection process and complete the intersection detection of basic graphics elements on the leaf nodes.

6. A virtual-reality collaborative control system for masonry robots based on digital twins according to claim 1, characterized in that: Including masonry robot digital twin system and masonry robot physical industrial control system; The digital twin system of the masonry robot includes a physical entity module, a virtual entity module, a connection module, a twin data module and a service module; the physical entity module reflects the on-site coke oven automatic masonry robot and its surrounding environment, including the geometric size information, movement mode and corresponding material properties of the masonry robot; the virtual entity module establishes a virtual geometric model based on information such as the physical entity structure, shape, material properties and assembly relationship, performs three-dimensional simplification and format conversion, and imports the model into Unity3D; the connection module is responsible for data interaction between various parts; the twin data module is used to store, extract, convert and process twin data; the twin data includes relevant data of physical entities, virtual entities, and service systems, domain knowledge and its fusion data, and is continuously updated and optimized as real-time data is generated; the service module includes the collection application of modules and data at all levels, and integrates various information systems such as evaluation, control, and optimization, providing intelligent operation, precise control and reliable operation and maintenance services based on physical entity and virtual entity models; The physical industrial control system of the masonry robot performs module arrangement based on step S2 as described in any one of claims 1 to 5; The digital twin system of the masonry robot and the physical industrial control system of the masonry robot perform virtual-reality collaborative control through step S3 as described in any one of claims 1 to 5.

Images