Man-machine collaborative remote operation system for laser additive repair

Abstract

The invention discloses a man-machine cooperation teleoperation system for laser additive repair. The man-machine cooperation teleoperation system comprises a mixed reality interaction master end system, a mobile additive repair slave end system and a communication system. The mixed reality interaction main end system comprises a mixed reality module, an additive repair control module, a mapping and positioning module and a man-machine interaction module; the mobile additive repair slave end system comprises an information acquisition module, an additive repair execution module and a navigation obstacle avoidance module; the communication system realizes data sharing of the master system and the slave system through various communication links. The system effectively integrates a remote control technology and robot automation, combines human experience judgment and the accurate execution capability of a machine to adjust a repair strategy in real time, can effectively deal with uncertainty and complexity in an operation scene, enables operators to be far away from harmful gas, high temperature, radiation and other severe environments, and improves the safety of the operators under the condition of protecting the safety of the operators. The operation flexibility is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot teleoperation, and in particular relates to a human-machine collaborative teleoperation system for laser additive repair. Background Art

[0002] At present, the global manufacturing industry is facing the dual challenges of environmental pressure and economic transformation. As an efficient and green remanufacturing technology, laser additive repair technology provides a new solution for the sustainable development of the manufacturing industry. The future development focus of the manufacturing industry is to promote innovation-driven, quality-oriented and green development. Laser additive repair technology is designed around these principles. It uses high-energy lasers to directly act on metal materials to achieve precise repair and functional restoration of waste parts, greatly improving the utilization efficiency of materials and reducing energy consumption and waste in the production process.

[0003] In the laser additive repair process, the fine powder or wire material used is melted under the action of the laser beam and accurately deposited on the part to be repaired to form a strong metallurgical bond. This technology has a high degree of automation and precise control capabilities, which can ensure the geometric dimensions and performance of the repaired parts, and even exceed the performance of the original parts in some cases. This technology is not only suitable for conventional metal parts repair, but also shows wide application potential in aerospace, shipbuilding, automobiles, heavy industry and medical equipment.

[0004] However, despite the many advantages of laser additive repair technology in terms of accuracy and reliability, there are still some challenges in actual operation. For example, repair operations often need to be carried out in specific and complex industrial environments, which may contain hazardous substances, extreme temperatures or other conditions that are harmful to operators. In addition, the complexity of the operation and high skill requirements also limit the widespread application of this technology. Summary of the invention

[0005] To solve the above problems, the present invention discloses a laser additive repair human-machine collaborative remote operation system, which effectively combines remote operation technology and robot automation, integrates the high adaptability of operators and the high precision of robots, and enhances operational flexibility while protecting the safety of operators.

[0006] To achieve the above object, the technical solution of the present invention is as follows: A laser additive repair human-machine collaborative remote control system comprises a mixed reality interactive master system, a mobile additive repair slave system and a communication system.

[0007] The mixed reality interactive master end system is developed based on Unity3D, supports cross-platform and multi-system operations, and is mainly responsible for remote control input, virtual simulation and human-computer interaction, processing feedback information from the slave end, and building a human-in-the-loop control link. The mixed reality interactive master end system includes a mixed reality module, an additive repair control module, a mapping and positioning module, and a human-computer interaction module.

[0008] Furthermore, the mixed reality module develops a highly realistic robot arm simulator and mirrors the real robot arm in real time; it also dynamically reconstructs the 3D model of the unknown object; it also integrates the real-time camera image stream and embeds the real-world data into the simulation scene to achieve a highly interactive mixed reality environment. The main steps of reconstructing the 3D model of the unknown object are: using statistical filtering SOR to remove outliers, using moving least squares MLS to smooth the point cloud, and finally using the ICP algorithm to register the pre-processed point cloud.

[0009] Furthermore, the additive repair control module includes robot arm control and laser control. The robot arm control is used to set the robot arm's motion parameters, control mode, and provide real-time feedback and display the robot arm's true position and posture; the laser control is used to set laser parameters, select laser repair paths, and display laser status.

[0010] Furthermore, the mapping and positioning module provides mapping and positioning functions and mobile platform control options. The gmapping algorithm is used for positioning and map construction; the direction and speed of the mobile platform are controlled by the keyboard and joystick, and displayed in real time on the interactive interface.

[0011] Furthermore, the human-computer interaction module uses multiple channels to interact with the host system, including hand controller, gesture recognition, VR device and voice input. The hand controller provides precise control input for changing the robot arm posture and selecting the repair target; gesture recognition and VR device provide more natural human-computer interaction for switching user perspectives and adjusting control parameters, and voice input provides additional effective control when the user needs to keep his hands free or operate at a distance.

[0012] The mobile additive repair slave system is developed based on ROS, and is mainly responsible for executing the control instructions sent by the master, completing the movement and repair tasks, and feeding back the information collected by the slave sensors to the master. The mobile additive repair slave system includes an information collection module, an additive repair execution module, and a navigation and obstacle avoidance module.

[0013] Furthermore, the information acquisition module includes two binocular cameras and a 16-line laser radar. One of the cameras is installed at the end of the robotic arm, close to the laser welding gun, to provide a close-up view of the repair operation; the other camera is located at the base of the robotic arm to capture a wider angle of the working scene. The laser radar is installed at the front end of the mobile platform to collect high-precision 3D point clouds for positioning and obstacle detection in complex environments.

[0014] Furthermore, the additive repair execution module calculates the target path of the robot and controls the PLC of the laser according to the repair path selected by the host system and the set laser parameters. The RRT-Connect algorithm is used to plan the robot path, and the Cartesian space trajectory is smoothed using the cubic spline interpolation method, while the joint space trajectory is interpolated using a quintic polynomial.

[0015] Furthermore, the navigation and obstacle avoidance module combines the base camera depth data with the lidar data to identify obstacles, and uses the TEB algorithm to dynamically plan the robot's movement path.

[0016] The communication system uses a variety of communication links to achieve data sharing between the master and slave ends, including wired cables, wireless radio stations and 5G networks. In close proximity and relatively fixed and safe environments, the communication system uses wired cables to provide stable and efficient data transmission; in wider and more complex operating environments, the communication system uses wireless radio stations to provide flexible mobile support; in cross-regional and long-distance operating tasks, the communication system uses 5G networks to provide low-latency and efficient data transmission to support a wider range of operations.

[0017] The working principle of the present invention is: The operator selects the laser repair target and path on the master system through the interactive device based on the reconstructed work scene information. After confirming that the plan is correct, the control command is sent to the slave system through the communication link to control the slave device to complete the repair operation. At the same time, the slave system feeds back the information collected by the sensor to the master in real time, so that the operator can monitor and adjust the repair process.

[0018] The beneficial effects of the present invention are: 1) The system integrates remote control technology and robotic automation, allowing operators to remotely control the machine through advanced control interfaces and real-time feedback to perform complex repair tasks. At the same time, the intelligent algorithm in the system can help optimize the operation path and parameters, while ensuring the quality of the repair, so that operators can stay away from harmful gases, high temperatures, radiation and other harsh environments; 2) In current industrial production, many tasks are difficult to automate due to their non-standardization and variability. This system adjusts the repair strategy in real time based on actual conditions, combining human experience and judgment with the precise execution capabilities of the machine, which can effectively cope with the uncertainty and complexity in production. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 4 is a system structure diagram of an embodiment of the present invention.

[0020] Figure 2 This is the host-side human-computer interaction interface of an embodiment of the present invention.

[0021] Figure 3 This is a diagram of the composition of the slave system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0023] The embodiment of the present invention provides a laser additive repair human-machine collaborative remote control system, and its overall system block diagram is as follows: Figure 1 As shown, the system includes a communication system, a mixed reality interaction master system, and a mobile additive repair slave system.

[0024] The communication system is mainly used for the transmission of control instructions, color images, 3D point clouds and equipment status information, including the host computer system installed on the mixed reality interactive master end and the industrial computer system installed on the mobile additive repair slave end. The two systems are connected through multiple communication links and use the Socket communication protocol. The host computer system on the master end transmits equipment control instructions and laser repair path information such as robot arm movement, mobile platform movement, laser switch, etc. to the industrial computer system on the slave end, and the industrial computer system on the slave end transmits the collected color images, camera 3D point clouds, laser 3D point clouds, ambient temperature, humidity and other sensor information and equipment status information such as robot arm position, laser status, and mobile platform position to the host computer system on the master end. In short-distance (10-15 meters) operations, the communication link uses wired cables to provide stable and efficient data transmission; in medium-distance operations, the communication link uses radio stations to provide flexible mobile support; in long-distance operations, the communication link uses 5G networks to provide low-latency and efficient data transmission; the delay of all communication links is controlled within 200ms.

[0025] The mixed reality interactive host system is mainly responsible for remote control input, virtual simulation and human-computer interaction, processing slave feedback information, and building a human-in-the-loop control link, including a mixed reality module, an additive repair control module, a mapping and positioning module, and a human-computer interaction module. The system is developed based on Unity3D and installed in the Windows system of a portable control box. It is also connected to peripheral devices such as 3DSystem Touch hand controller, camera, microphone, Oculus Quest 2 VR glasses for interaction with users. The hand controller is mainly used to provide precise control input to the robot arm posture and select laser repair targets; the camera is used to capture the user's hand movements, thereby identifying the corresponding gesture intentions to control device parameters or switch the user's perspective; the microphone is used to collect user voice and identify the corresponding semantic information to make corresponding adjustments; the VR device provides a rich and realistic visual experience on the one hand, and provides rough motion input on the other hand, which is used to select repair targets and adjust the user's perspective.

[0026] Figure 2 A schematic diagram of the human-computer interaction interface of the host system is provided, which mainly shows some functions of the mixed reality module, additive repair module and mapping and positioning module.

[0027] Among them, the mixed reality module is mainly responsible for image calibration, image display, hand-eye calibration, unknown object modeling and other functions. In this module, the system develops a highly realistic robot arm simulator by importing the URDF file of the real robot arm, and mirrors and synchronizes the real robot arm in real time; in the laser repair operation, the module can dynamically reconstruct the 3D model of the unknown object and integrate it into the Unity virtual environment; at the same time, the module further integrates the real-time camera image stream, embeds the real-world data into the virtual scene, and realizes a highly interactive mixed reality environment.

[0028] In particular, in order to obtain accurate visual information and ensure the consistency of the coordinate system of the three-dimensional data and the robot arm, the system performs detailed camera calibration and hand-eye calibration. Camera calibration is performed using a standardized checkerboard pattern, and internal and external parameters are calculated by capturing image sets at different positions and angles. The Tsai-Lenz algorithm is then used to calculate the fixed relationship between the camera and the end of the robot arm to ensure the accuracy and consistency of the virtual welding operation in the Unity3D environment. In addition, in terms of three-dimensional reconstruction, the system achieves three-dimensional model reconstruction of unknown objects by accurately aligning and fusing point cloud data sets acquired from different angles. The main steps are: using statistical filtering SOR to remove outlier noise, using moving least squares MLS to smooth the point cloud, and finally using the ICP algorithm to align the preprocessed point cloud.

[0029] The additive repair control module includes robot arm control and laser control. The robot arm control is used to set the robot arm's motion parameters (joint angle, end speed, etc.), control mode (based on ROS control and local API control), and real-time feedback and display of the robot arm's true posture, etc. The laser control is used to set laser parameters (laser power, powder feeder speed, processing technology, etc.), select laser repair path, and display laser status (ready signal, error signal, handshake signal, light output status, etc.).

[0030] The mapping and positioning module provides mapping and positioning functions and mobile platform control options. The system uses the gmapping algorithm for positioning and map construction, and displays the current position in real time on the map interface; at the same time, the keyboard and joystick are used to control the direction and speed of the mobile platform, and the current speed of the mobile platform is displayed on the interactive interface.

[0031] The mobile additive repair slave system is mainly responsible for executing the control instructions sent by the master, completing the movement and repair tasks, and feeding back the information collected by the slave sensors to the master. It includes an information acquisition module, an additive repair execution module, and a navigation and obstacle avoidance module. The system is developed based on ROS, installed in the industrial computer of the mobile platform, and connected to peripheral devices such as robotic arms, lasers, cameras, radars, etc. Figure 3 The composition framework of the slave system is shown.

[0032] Among them, the depth camera and laser radar belong to the information acquisition module, which is mainly used to collect environmental information of the working scene around the mobile platform, which is used to reconstruct the workpiece to be repaired and establish a grid map. The binocular camera uses Real SenseD435 to provide color images and depth images. One camera is installed at the end of the robotic arm, close to the laser welding gun, to provide a close-up view of the repair operation; the other camera is located at the base of the robotic arm to capture a wider angle of the working scene. The laser radar uses Helios16, which is installed at the front end of the mobile platform to collect high-precision 3D point clouds for positioning and obstacle detection in complex environments.

[0033] Among them, the robot arm and the laser belong to the additive repair execution module. This module calculates the target path of the robot arm and sets the state of the laser according to the repair path selected by the host system and the set laser parameters. Finally, it controls the robot arm to execute the corresponding trajectory and enables the laser to complete the repair of the damaged workpiece. In the path planning of the robot arm, the module uses the RRT-Connect algorithm to plan the path of the robot arm, and uses the cubic spline interpolation method to smooth the Cartesian space trajectory, and uses the quintic polynomial to interpolate the joint space trajectory.

[0034] Among them, the mobile platform belongs to the navigation and obstacle avoidance module, which combines the base camera depth data with the lidar data to identify obstacles, and uses the TEB algorithm to dynamically plan the robot's movement path, and finally controls the mobile platform to avoid obstacles and reach the target work location.

[0035] The laser additive repair human-machine collaborative remote operation system provided by the embodiment of the present invention includes a mixed reality interactive master system, a mobile additive repair slave system and a communication system. Among them, the mixed reality interactive master system includes a mixed reality module, an additive repair control module, a mapping and positioning module and a human-machine interaction module; the mobile additive repair slave system includes an information acquisition module, an additive repair execution module and a navigation obstacle avoidance module. This system effectively integrates remote control technology and robotic automation, allowing operators to remotely control machines to perform complex repair tasks through advanced control interfaces and real-time feedback. At the same time, the intelligent algorithms in the system can help optimize operation paths and parameters, while ensuring the quality of repair, allowing operators to stay away from harmful gases, high temperatures, radiation and other harsh environments; at the same time, this system adjusts the repair strategy in real time according to actual conditions, combined with human experience and judgment and the precise execution capabilities of the machine, which can effectively deal with the uncertainty and complexity in production.

[0036] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A laser additive repair human-machine collaborative remote control system, characterized in that: The system includes a mixed reality interactive master system, a mobile additive repair slave system, and a communication system; The mixed reality interaction host system is developed based on Unity3D, supports cross-platform and multi-system operations, is responsible for remote control input, virtual simulation and human-computer interaction, processes slave feedback information, and builds a human-in-the-loop control link; the mixed reality interaction host system includes a mixed reality module, an additive repair control module, a mapping and positioning module, and a human-computer interaction module; The mobile additive repair slave system is developed based on ROS, and is mainly responsible for executing the control instructions sent by the master, completing the movement and repair tasks, and feeding back the information collected by the slave sensor to the master; the mobile additive repair slave system includes an information collection module, an additive repair execution module, and a navigation and obstacle avoidance module; The communication system uses a variety of communication links to achieve master-slave data sharing, including wired cables, radio stations and 5G networks; in close proximity, fixed and safe environments, the communication system uses wired cables to provide stable and efficient data transmission; in broader and more complex operating environments, the communication system uses radio stations to provide flexible mobility support; in cross-regional and long-distance operation tasks, the communication system uses 5G networks to provide low-latency and efficient data transmission to support a wider range of operations.

2. According to claim 1, a laser additive repair human-machine collaborative teleoperation system is characterized in that: The mixed reality module develops a highly realistic robot arm simulator and mirrors the real robot arm in real time; dynamically reconstructs the 3D model of unknown objects; and integrates real-time camera image streams to embed real-world data into the simulation scene to achieve a highly interactive mixed reality environment; The steps of reconstructing the 3D model of the unknown object are: using statistical filtering SOR to remove outlier noise points, using moving least squares MLS to smooth the point cloud, and finally using the ICP algorithm to align the preprocessed point cloud.

3. According to claim 1, a laser additive repair human-machine collaborative teleoperation system is characterized in that: The additive repair control module includes robot arm control and laser control; the robot arm control is used to set the motion parameters and control mode of the robot arm, and to provide real-time feedback and display the actual position of the robot arm; the laser control is used to set laser parameters, select laser repair path and display laser status.

4. The laser additive repair human-machine collaborative teleoperation system according to claim 1, characterized in that: The mapping and positioning module provides mapping and positioning functions and mobile platform control options; the gmapping algorithm is used for positioning and map construction; the direction and speed of the mobile platform are controlled by the keyboard and joystick, and displayed in real time on the interactive interface.

5. The laser additive repair human-machine collaborative teleoperation system according to claim 1, characterized in that: The human-computer interaction module uses multiple channels to interact with the host system, including hand controller, gesture recognition, VR equipment and voice input; The hand controller provides precise control input for changing the robot arm's posture and selecting repair targets; gesture recognition and VR devices provide more natural human-computer interaction for switching user perspectives and adjusting control parameters; voice input provides additional effective control when the user needs to keep his hands free or operate at a long distance.

6. The laser additive repair human-machine collaborative teleoperation system according to claim 1, characterized in that: The information acquisition module includes two binocular cameras and a 16-line laser radar. One camera is installed at the end of the robotic arm, close to the laser welding gun, to provide a close-up view of the repair operation; the other camera is located at the base of the robotic arm to capture a wider angle of the working scene; the laser radar is installed at the front end of the mobile platform to collect high-precision 3D point clouds for positioning and obstacle detection in complex environments.

7. The laser additive repair human-machine collaborative teleoperation system according to claim 1, characterized in that: The additive repair execution module calculates the target path of the robot and controls the PLC of the laser according to the repair path selected by the host system and the set laser parameters; the RRT-Connect algorithm is used to plan the robot path, and the Cartesian space trajectory is smoothed using the cubic spline interpolation method, while the joint space trajectory is interpolated using a quintic polynomial.

8. The laser additive repair human-machine collaborative teleoperation system according to claim 1, characterized in that: The navigation and obstacle avoidance module combines the base camera depth data with the lidar data to identify obstacles, and uses the TEB algorithm to dynamically plan the robot's movement path.

9. The laser additive repair human-machine collaborative teleoperation system according to claim 1, characterized in that: Its working principle is: The operator selects the laser repair target and path on the master system through the interactive device based on the reconstructed work scene information. After confirming that the plan is correct, the control command is sent to the slave system through the communication link to control the slave device to complete the repair operation. At the same time, the slave system feeds back the information collected by the sensor to the master in real time, so that the operator can monitor and adjust the repair process.

Images