An experimental platform for mining robots based on digital twin technology

Abstract

The present invention discloses an experimental platform for a mining robot based on digital twin, which includes a server, a virtual scene generation module, a mining robot twin, a path and trajectory planning module, a multi-degree-of-freedom mining robot, a controller, and an experimental platform. The server is used to perform the calculations required by the virtual scene generation module, the mining robot twin, the path and trajectory planning module, and the controller. The virtual scene generation module generates in real time the virtual working environment required by the mining robot twin, and realizes the motion control of the mining robot twin through the path and trajectory planning module. The controller controls the motion of the multi-degree-of-freedom mining robot and collects the information fed back by the multi-degree-of-freedom mining robot. It communicates with the virtual scene generation module through a network to exchange virtual and real information in real time. The experimental platform includes a sand table and a lidar installed on the top. The sand table is used to simulate the working environment of the construction site, and the multi-degree-of-freedom mining robot operates on the sand table.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and particularly to an experimental platform for an excavation robot based on digital twin technology. Background Art

[0002] Construction sites are one of the most dangerous working environments in the world, with a high casualty rate. According to statistics, on average, one-fifth of fatal occupational injuries each year are attributed to the construction industry. The operation of heavy construction equipment such as excavators is a key part of most construction projects. Due to space limitations and time constraints, heavy construction equipment, vehicles, and workers usually need to work simultaneously in a limited working space, which usually results in poor performance in terms of safety and productivity. Blind spots in equipment and work areas are one of the main causes of collision accidents. Workers are highly concentrated and exposed to a noisy environment for a long time, making them often in a state of fatigue, which will greatly affect the ability of workers and operators of heavy equipment to identify hazards, thus increasing the probability of safety accidents.

[0003] Therefore, the teleoperation of heavy construction equipment such as excavators becomes very important. Scholars at home and abroad have carried out many studies on teleoperated engineering robots. Researchers reconstructed a complete three-dimensional scene using information from multiple perspectives in the working environment, providing multiple perspectives for remote operators, so that operators can complete tasks more simply and safely. However, the amount of data transmitted by pictures is huge and the latency is large. Then some researchers used the binocular stereo vision cue method, pasted positioning marks on the key parts of heavy construction equipment, and used the points on the stickers as feature points for matching, but the actual effect is affected by light and the robustness is not strong. Summary of the Invention

[0004] The present invention provides an experimental platform for an excavation robot based on digital twin technology. Based on digital twin, this platform can generate a human-computer interaction interface based on augmented reality, getting rid of the vision limitation of operators and comprehensively grasping the change information of the entire construction site.

[0005] To achieve the above object, the specific technical solutions adopted by the present invention are as follows:

[0006] The present invention mainly includes a server, a virtual scene generation module, a twin body of an excavation robot, a path and trajectory planning module, a multi-degree-of-freedom excavation robot, a controller, and an experimental platform.

[0007] The server is used to perform the calculations required by the virtual scene generation module, the twin body of the excavation robot, the path and trajectory planning module, and the controller. The controller is installed on the multi-degree-of-freedom excavation robot.

[0008] The virtual scene generation module is generated by 3D virtual reality software simulation. By reading the laser point cloud data in the database and the established 3D scene model, it generates in real time the virtual operation environment required for the excavation robot twin. Through an external control joystick and a somatosensory controller, it realizes the motion control of the excavation robot through the path and trajectory planning module. The external control joystick and somatosensory controller collect the bending changes of the operator's left hand finger joints through the path and trajectory planning module, and plan a reasonable motion path and trajectory according to the terrain of the virtual operation environment, which is used to control the manipulator posture of the excavation robot twin and further control the manipulator posture of the multi-degree-of-freedom excavation robot.

[0009] The multi-degree-of-freedom excavation robot includes a bucket, a dipper arm, a boom, a slewing platform, a traveling device, a Hall encoder, and an angle sensor. The angle sensors are installed on the bucket, the dipper arm, the boom, and the slewing platform to obtain the working state of the excavation robot in real time. The Hall encoder is installed on the motor of the traveling device to accurately feedback the traveling speed and distance of the excavation robot. The multi-degree-of-freedom excavation robot and the excavation robot twin in the virtual scene correspond one by one in terms of appearance and function. The excavation robot twin can achieve consistent motion with the multi-degree-of-freedom excavation robot through the received sensor data of the multi-degree-of-freedom excavation robot; the motion trajectory and motion path of the excavation robot twin can also be planned in the virtual environment and used to control the motion of the multi-degree-of-freedom excavation robot.

[0010] The controller includes a Raspberry Pi 4B and a motor driver board. The Raspberry Pi 4B is connected to the motor driver board through the GPIO port, outputs control signals to control the motion of the multi-degree-of-freedom excavation robot, and collects the information feedback by the angle sensor and the Hall sensor; communicates with the virtual scene generation module through the network to exchange virtual and real information in real time. At the same time, through the driver board, it controls the multi-degree-of-freedom excavation robot to perform various actions according to the generated trajectory to complete various tasks; the experimental platform includes a sand table, columns, an electric slide rail, and a lidar. The sand table is used to simulate the operation environment of the construction site. There are four columns standing around, and the columns are connected by an electric slide rail. The lidar is installed on the electric slide rail at the top of the sand table to collect the position and posture information of the multi-degree-of-freedom excavation robot and the changes of the sand pile in the sand table in real time, and synchronize the change data of the operation environment to the virtual scene generation module to update the virtual operation scene.

[0011] Preferably, the virtual scene generation module can generate a virtual operation scene of the excavation robot consistent with the experimental platform through three-dimensional grid dynamic visualization technology based on the 3D laser point cloud collected in real time by the lidar above the sand table of the experimental platform and the three-dimensional model of the virtual scene established in the database, and can update the virtual operation scene in real time; through the combined control of an external control joystick and a somatosensory controller, combined with the path and trajectory planning module, control the motion state of the multi-degree-of-freedom excavation robot; communicate with the controller of the multi-degree-of-freedom excavation robot through the network, synchronize the control commands of the joystick and the somatosensory controller to the multi-degree-of-freedom excavation robot, and can control the multi-degree-of-freedom excavation robot to operate within the experimental platform.

[0012] Preferably, the sand table of the experimental platform can simulate the operation environment of the construction site, and the feedback laser point cloud data is transmitted to the database in the server for synchronous positioning and real-time mapping of the multi-degree-of-freedom excavation robot in the virtual scene generation module, and at the same time, the virtual operation scene is updated in real time.

[0013] The present invention also provides an experimental method for an excavation robot based on digital twin, and the functions are specifically realized through the following steps:

[0014] Step 1: The operator connects the lidar in the experimental platform to the virtual scene generation module, then runs the virtual scene generation module and the excavation robot twin body, and at the same time, according to the sand table data feedback by the lidar, performs positioning and mapping synchronously to initialize the virtual operation scene.

[0015] Step 2: Turn on the power of the multi-degree-of-freedom excavation robot and turn on the power switch to make the multi-degree-of-freedom excavation robot in a listening state. At this time, the multi-degree-of-freedom excavation robot will establish communication with the virtual scene generation module, synchronize the joint angle information to the virtual scene generation module, and initialize the posture of the excavation robot twin body.

[0016] Step 3: The operator connects the control joystick and the somatosensory controller to the virtual scene generation module. After the operator confirms that the virtual scene and the twin body of the multi-degree-of-freedom excavation robot correspond correctly to the sand table scene and the position and posture of the multi-degree-of-freedom excavation robot, the experiment can be started.

[0017] Step 4: The operator selects the joystick button: If the manual mode is selected, the excavation robot twin body can be controlled to reach the set target position through the joystick according to the virtual operation scene displayed on the human-computer interaction interface of the virtual scene generation module; if the automatic mode is selected, the excavation trajectory is automatically generated according to the set task by using the path planning and trajectory optimization method. In either mode, the multi-degree-of-freedom excavation robot and the excavation robot twin body will maintain the consistency of motion through the transmitted trajectory control.

[0018] Step 5: The operator places the left hand above the somatosensory controller and operates the control joystick with the right hand. At this time, based on the generated trajectory by the bending changes of the left hand fingers, the excavation robot twin can be controlled, and further the multi-degree-of-freedom excavation robot can be controlled to perform various set excavation tasks in the operation scenario. Meanwhile, the lidar located at the top of the sand table will feedback the sand table change data to the virtual scene generation module in real time to update the virtual operation scenario in real time.

[0019] Step 6: The operator selects the automatic mode through the joystick button. In this mode, various pre-set excavation tasks can be selected for automatic operation. The virtual scene generation module will rely on the real-time data feedback from the sensors of the multi-degree-of-freedom excavation robot and the lidar on the electric slide rail of the sand table to plan the path in real time and operate autonomously according to the selected excavation task.

[0020] Due to the adoption of the above technical solutions, the technical effects achieved by the present invention are as follows:

[0021] An experimental platform for an excavation robot based on digital twin of the present application is based on digital twin technology. By reading the joint data of the multi-degree-of-freedom excavation robot and remotely synchronizing it into the excavation robot twin, through an external control joystick and somatosensory controller, and cooperating with the human-computer interaction interface based on augmented reality technology, the operator can observe the entire construction site in all directions, eliminate the visual blind area in the traditional operation scenario, and avoid the occurrence of safety accidents.

[0022] Compared with other methods, the present invention can generate the virtual operation environment required for the excavation robot twin in real time by reading the laser point cloud data in the database and the established 3D scene model. The update of the twin pose only needs to synchronize the joint point data of the excavation robot, without the need to transmit the complete image data, greatly reducing the time delay. The external control joystick and somatosensory controller make the operator's experience more real.

[0023] The method of the present application solves the problems that traditional construction machinery has high requirements for the operator to get started, cannot observe the entire construction site in all directions, and has visual blind areas, thus improving the operation efficiency of the entire project. Description of the Drawings

[0024] Figure 1 It is the overall structural block diagram of the experimental platform;

[0025] Figure 2 It is the overall schematic diagram of the experimental platform;

[0026] Figure 3 It is the sand table simulation excavation scene diagram;

[0027] Figure 4 It is the structural diagram of the multi-degree-of-freedom excavation robot;

[0028] Figure 5 It is the wiring diagram of the controller and the motor drive board;

[0029] Figure 6 It is the schematic diagram of the motor drive board;

[0030] Among them, 1. boom, 2. bucket, 3. arm, 4. slewing mechanism, 5. travel device, 6. lidar. Specific embodiments

[0031] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] As Figure 1 and Figure 2 shown, an experimental platform for an excavation robot based on digital twin technology includes a server, a virtual scene generation module, a digital twin of the excavation robot, a path and trajectory planning module, a multi-degree-of-freedom excavation robot, a controller, and an experimental platform; the server is used to perform the calculations required by the virtual scene generation module, the digital twin of the excavation robot, the path and trajectory planning module, and the controller; the virtual scene generation module generates in real time the virtual working environment required by the digital twin of the excavation robot, and realizes the motion control of the digital twin of the excavation robot through the path and trajectory planning module; the multi-degree-of-freedom excavation robot corresponds one-to-one with the digital twin of the excavation robot in the virtual scene in terms of appearance and function, and the digital twin of the excavation robot can achieve consistent motion with the multi-degree-of-freedom excavation robot through the sensor data of the multi-degree-of-freedom excavation robot received; the controller controls the motion of the multi-degree-of-freedom excavation robot, and collects the information fed back by the multi-degree-of-freedom excavation robot; communicates with the virtual scene generation module through the network to exchange virtual and real information in real time; the experimental platform includes a sand table and a lidar 6 arranged on the top, the sand table is used to simulate the working environment of the construction site, the multi-degree-of-freedom excavation robot works on the sand table, and the lidar 6 is used to collect the change data in the sand table.

[0033] The virtual scene generation module is generated by 3D virtual reality software simulation, and generates in real time the virtual working environment required by the digital twin of the excavation robot by reading the laser point cloud data in the database and the established three-dimensional model of the scene. The virtual scene generation module is externally connected with a control joystick and a somatosensory controller, and the server controls the calculation of the somatosensory controller.

[0034] The externally connected somatosensory controller selects the relatively mature Leap Motion module on the market. This module is equipped with a high-frame-rate binocular camera system, which extracts the three-dimensional position of the hand through the principle of binocular stereo vision imaging. It can track ten fingers of a person simultaneously and output the change data of each joint, with an accuracy of up to 1 / 100 mm. And through algorithm optimization, the acquisition speed can reach 200 frames per second, and any change of the finger can be collected in time. This application uses it to collect the bending change of the operator's index finger joint, so as to control the posture of the manipulator of the excavation robot twin, combine the data of the control joystick, and plan a reasonable trajectory according to the terrain of the virtual operation environment by using the path and trajectory planning module, and further control the posture of the multi-degree-of-freedom excavation robot.

[0035] As shown in Figure 3, the sand table can simulate various excavation environments, and various obstacles can be artificially added to make the experimental scene more flexible and changeable, with higher requirements for the control accuracy, timeliness, and stability of the multi-degree-of-freedom excavation robot. The lidar 6 on the top of the sand table can move freely through the electric slide rail to collect the change information inside the sand table in real time, and the virtual scene generation module can update the virtual operation scene according to this information.

[0036] As shown in Figure 4, the multi-degree-of-freedom excavation robot has a structure similar to that of a real excavator, and also has a bucket 2, a boom 1, an arm 3, a slewing mechanism 4, and a traveling device 5. The traveling device 5 also adopts a crawler structure, which can cope with various complex environments. Hall encoders and angle sensors are installed on the multi-degree-of-freedom excavation robot to collect the posture and position information of the multi-degree-of-freedom excavation robot during operation.

[0037] As Figure 5 shown, the controller includes a Raspberry Pi 4B and a motor driver board. The Raspberry Pi 4B is connected to the motor driver board through the GPIO port, collects the information fed back by the angle sensor and the Hall sensor, and then obtains the angle information and speed information of the multi-degree-of-freedom excavation robot, and further controls the movement of the multi-degree-of-freedom excavation robot.

[0038] As Figure 6 shown, the motor driver board includes a power supply circuit and a motor drive circuit. The power supply circuit reduces the input voltage to 3.3V and 5V to meet the voltage required by the motor drive circuit. The motor drive circuit can amplify the drive current of the GPIO port, so that the GPIO port with very weak load capacity has the ability to control the movement of the multi-degree-of-freedom excavation robot.

[0039] When conducting the experiment, first set up the excavation environment in the sand table, including the excavation target and surrounding obstacles. Place the multi-degree-of-freedom excavation robot in the sand table and connect the lidar to the virtual scene generation module. Then start the virtual scene generation module and observe the human-computer interaction interface until the virtual scene generation module initializes the position of the excavation robot twin and the working environment according to the lidar point cloud data. Connect the external control joystick and somatosensory controller to the virtual scene generation module, and turn on the power of the multi-degree-of-freedom excavation robot. After waiting for the excavation robot twin to have the same posture as the multi-degree-of-freedom excavation robot, it is possible to cooperate with the human-computer interaction interface and control the excavation robot to work in the sand table through the control joystick and somatosensory controller.

[0040] The experimental platform of the excavation robot based on digital twin can be applied to the actual situation. The multi-degree-of-freedom excavation robot corresponds to the actual excavator, the experimental platform corresponds to the actual construction site, and the lidar can be installed separately on other engineering construction machinery to adapt to the complex and changeable construction environment.

[0041] It is also possible to replace the multi-degree-of-freedom excavation robot with other construction machinery robots, such as bulldozer robots, loading robots, dump truck robots, etc. Only need to generate the corresponding robot twin in the virtual environment generation module in advance, and then install the angle sensor and Hall encoder in the corresponding positions. Establish a connection with the twin through the network, and the consistent movement of the twin and the real body can be achieved.

[0042] In this specification, on the premise of the control scheme of the present invention, detailed implementation manners and specific operation processes are given, but the present invention is not limited to the above implementation manners, so the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An experimental platform for a mining robot based on digital twin technology, characterized in that: It includes a server, a virtual scene generation module, a mining robot twin, a path and trajectory planning module, a multi-degree-of-freedom mining robot, a controller, and an experimental platform; The server is used to perform the calculations required by the virtual scene generation module, the mining robot twin, the path and trajectory planning module, and the controller; The virtual scene generation module generates the virtual working environment required by the mining robot twin in real time, and realizes the motion control of the mining robot twin through the path and trajectory planning module; The multi-degree-of-freedom mining robot corresponds one-to-one with the mining robot twin in the virtual scene in terms of appearance and function. The mining robot twin can achieve consistent motion with the multi-degree-of-freedom mining robot through the received sensor data of the multi-degree-of-freedom mining robot; The controller controls the motion of the multi-degree-of-freedom mining robot and collects the information fed back by the multi-degree-of-freedom mining robot; communicates with the virtual scene generation module through the network to exchange virtual and real information in real time; The experimental platform includes a sand table and a lidar (6) installed on the top. The sand table is used to simulate the working environment of the construction site. The multi-degree-of-freedom mining robot operates on the sand table. The lidar (6) is used to collect the change data in the sand table; the virtual scene generation module is simulated and generated by 3D virtual reality software, and generates the virtual working environment required by the mining robot twin in real time by reading the lidar point cloud data and the established 3D scene model in the database. The virtual scene generation module is externally connected with a control joystick and a somatosensory controller; the externally connected somatosensory controller is used to control the arm posture of the mining robot twin by collecting the bending changes of the operator's index finger joints, combines the data of the control joystick, and plans a reasonable trajectory according to the terrain of the virtual working environment by using the path and trajectory planning module, and further controls the posture of the multi-degree-of-freedom mining robot; The lidar point cloud data fed back is transmitted to the database in the server, which is used to collect the changes of the sand pile in the sand table in real time, perform synchronous localization and real-time mapping of the multi-degree-of-freedom mining robot in the virtual scene generation module, and synchronize the change data of the working environment to the virtual scene generation module to update the virtual working scene in real time.

2. The experimental platform of an excavation robot based on digital twin technology according to claim 1, wherein: The multi-degree-of-freedom mining robot includes an aluminum alloy bucket, a boom, an arm, a rotary table, and a traveling device. A Hall encoder, an angle sensor, and a controller are installed on the multi-degree-of-freedom mining robot. The Hall encoder and the angle sensor collect the posture and position information of the multi-degree-of-freedom mining robot during operation.

3. The experimental platform for a mining robot based on digital twin technology according to claim 2, characterized in that: The controller includes a Raspberry Pi 4B and a motor driver board. The Raspberry Pi 4B is connected to the motor driver board through the GPIO port to collect the information fed back by the angle sensor and the Hall sensor.

4. The experimental platform of an excavation robot based on digital twin technology according to claim 2, characterized in that: The angle sensor is installed on the bucket, boom, arm, and rotary table of the multi-degree-of-freedom mining robot to obtain the posture of the multi-degree-of-freedom mining robot in real time. The Hall encoder is installed on the motor of the traveling device to accurately feedback the traveling speed and distance of the multi-degree-of-freedom mining robot.

5. The experimental platform of an excavation robot based on digital twin technology according to claim 1, characterized in that: The experimental platform further includes columns and an electric slide rail. Four columns are arranged around the sand table, and the columns are connected by the electric slide rail. The lidar (6) is installed on the electric slide rail at the top of the sand table.

6. The experimental platform of an excavation robot based on digital twin technology according to claim 1, characterized in that: The virtual scene generation module generates a virtual operation scene of the excavation robot consistent with the experimental platform through three-dimensional grid dynamic visualization technology according to the lidar point cloud collected in real time by the lidar (6) above the sand table of the experimental platform and the virtual scene three-dimensional model established in the database, and can update the virtual operation scene in real time; through the combined control of the external control joystick and the somatosensory controller, combined with the path and trajectory planning module, the working state of the multi-degree-of-freedom excavation robot is controlled; through communication with the controller of the multi-degree-of-freedom excavation robot via the network, the control commands of the control joystick and the somatosensory controller are synchronized to the multi-degree-of-freedom excavation robot, and the multi-degree-of-freedom excavation robot can be controlled to operate within the experimental platform.

7. A mining robot experimental platform based on digital twin technology according to claim 1, characterized in that: The experimental process includes the following steps: Step 1, the operator connects the lidar (6) in the experimental platform to the virtual scene generation module, then runs the virtual scene generation module, and at the same time, according to the lidar data of the sand table feedback by the lidar (6), performs positioning and mapping synchronously to initialize the virtual operation scene. Step 2, turn on the power of the multi-degree-of-freedom excavation robot and turn on the power switch to make the multi-degree-of-freedom excavation robot in a listening state. At this time, the multi-degree-of-freedom excavation robot will establish communication with the virtual scene generation module and synchronize the joint angle information to the virtual scene generation module to initialize the posture of the excavation robot twin. Step 3, the operator connects the control joystick and the somatosensory controller to the virtual scene generation module. After the operator confirms that the virtual scene and the excavation robot twin correspond to the position and posture of the sand table scene and the multi-degree-of-freedom excavation robot correctly, the experiment can be started. Step 4, the operator selects the joystick button: if the manual mode is selected, the excavation robot twin can be controlled to reach the set target position through the joystick according to the virtual operation scene displayed on the interface of the virtual scene generation module; if the automatic mode is selected, the excavation trajectory is automatically generated using the path planning and trajectory planning method according to the set task. Step 5, the operator places the left hand above the somatosensory controller and operates the control joystick with the right hand. At this time, based on the generated trajectory, the posture of the mechanical arm of the excavation robot twin can be controlled by the bending change of the left hand fingers, and further control the multi-degree-of-freedom excavation robot to perform various set excavation tasks in the operation scene. At the same time, the lidar (6) located at the top of the sand table will feedback the sand table change data to the virtual scene generation module in real time to update the virtual operation scene in real time. Step 6: The operator selects the automatic mode through the joystick button. In this mode, various pre-set excavation tasks can be selected for automatic operation. The virtual scene generation module will rely on the sensors of the multi-degree-of-freedom excavation robot and the real-time data feedback by the lidar on the electric slide rail of the sand table to plan the path in real time and perform autonomous operation according to the selected excavation task.

Images