Production line robot visual digitization system based on digital twinning technology

Abstract

The invention discloses a production line robot visual digitization system based on a digital twinning technology. The production line robot visual digitization system comprises a digital twinning module, a data fusion processing module, an interaction control module and a physical unit. The physical unit comprises an industrial robot and a multi-source sensor array. The multi-source sensor array comprises a laser radar and a CCD camera which are installed on the industrial robot. The physical unit is in communication connection with the data fusion processing module, and the data fusion processing module carries out space-time alignment fusion and feature extraction on obtained robot pose data, laser radar data and visual data; and the digital twin module maps the operation state of the physical production line in real time based on virtual-real synchronization. Compared with the prior art, the production line robot visual digitization system based on the digital twin technology has the advantages that multi-module cooperation is achieved, and production line intelligence is improved based on production line robot visual application.

Description

TECHNICAL FIELD

[0001] The application relates to the field of industrial automation and digital twinning technology, and particularly relates to a production line robot vision digital system based on digital twinning technology. BACKGROUND

[0002] In modern industrial production, production line robots are increasingly widely applied and undertake key tasks such as material handling, assembly and welding. However, traditional production line robot systems have some limitations.

[0003] On the one hand, robots mainly rely on preset programs to run and are difficult to adapt to complex and changeable production environments in real time, lacking dynamic perception and flexible adjustment capabilities for actual production conditions.

[0004] On the other hand, there is a lack of effective visual monitoring and prediction means in the production process, and it is difficult to discover potential problems in advance and intervene in time, resulting in limited production efficiency and causing great losses when a fault occurs. SUMMARY

[0005] The technical problem to be solved by the application is to overcome the above technical defects and provide a production line robot vision digital system based on digital twinning technology, which improves the intelligence of the production line based on multi-module cooperation and production line robot vision application.

[0006] To solve the above technical problems, the technical scheme provided by the application is as follows: a production line robot vision digital system based on digital twinning technology, comprising a digital twinning module, a data fusion processing module, an interactive control module and a physical unit. The physical unit comprises an industrial robot and a multi-source sensor array, and the multi-source sensor array comprises a laser radar and a CCD camera installed on the industrial robot. The physical unit is in communication connection with the data fusion processing module, the data fusion processing module performs spatio-temporal alignment fusion and feature extraction on the acquired robot pose data, laser radar data and vision data. The digital twinning module is used for real-time mapping of the running state of the physical production line based on virtual-real synchronization, the interactive control module is used for displaying the mapping data of the digital twinning module, and an optimized control instruction is generated and fed back to the physical unit.

[0007] Preferably, the digital twinning module comprises a modeling unit, a simulation unit and a rendering unit. The modeling unit constructs a three-dimensional model of the production line space. The simulation unit acquires output information of the data fusion processing module to simulate kinematic behavior of the robot. The rendering unit performs real-time visualization processing on the three-dimensional model and simulation data.

[0008] Preferably, the interaction control module comprises a simulation verification unit, a delay compensator and a safety judge. The simulation verification unit performs pre-rehearsal verification on the optimization control instruction and outputs the instruction result. The delay compensator outputs the physical unit execution delay and generates a compensation instruction. The safety judge comprises a threshold value set for judging the virtual-real deviation in the digital twin module, and performs emergency stop protection when the deviation exceeds the threshold value.

[0009] Preferably, the data fusion processing module comprises: The laser radar point cloud data is voxel filtered and ground segmented to extract the three-dimensional contour features of the workpiece. The image collected by the CCD camera is subjected to distortion correction and feature matching to identify the surface texture mark of the workpiece. The robot pose data and sensor data are fused to output the six-degree-of-freedom pose of the workpiece in space-time alignment.

[0010] Preferably, the same group of industrial robots also has a sensor synchronization controller for time synchronization of the laser radar, CCD camera and robot pose information.

[0011] Preferably, the modeling unit comprises a device static model obtained from a CAD drawing and a workpiece dynamic model obtained from point cloud data to construct a space three-dimensional model.

[0012] Preferably, the interaction control module further comprises a multi-level permission management unit, which sets three levels of permissions including operators, process engineers and system administrators.

[0013] Further, the specific steps of the delay compensator outputting the physical unit execution delay and generating a compensation instruction are as follows: S1. Establish a delay prediction model to estimate the total delay time Δt of the completion of the current action of the physical unit; S2. Use the total delay time Δt to calculate the predicted future state of the physical unit after the current time plus the total delay time Δt; S3. Based on the predicted future state in step S2, generate a compensation instruction and output it to the physical unit for execution. The total delay time Δt is the sum of the predicted visual processing delay, simulation calculation delay, network transmission delay, processing and motion delay.

[0014] Preferably, the total delay time Δt is calculated by the timestamp method, and the specific steps are as follows: S11. All physical units in the entire system use the precise time protocol, and the following key nodes are marked with timestamps.

[0015] a. The physical time when the CCD camera exposure ends is marked with a timestamp T1.

[0016] b. The digital twin module timestamps the moment before sending the optimized control instruction to the simulation verification unit as T2.

[0017] c. The simulation verification unit timestamps the moment when the simulation verification is completed and the final instruction result is output to the delay compensator as T3.

[0018] d. The industrial robot timestamps the moment when the final motion instruction is received as T4.

[0019] e. The industrial robot timestamps the moment when the task is completely executed and the "task completed" signal is fed back as T5.

[0020] S12. Delay decomposition calculation, calculate the delay of each stage: visual processing delay Δt1=T2-T1, simulation calculation delay Δt2=T3-T2, network transmission delay Δt3=T4-T3, processing and motion delay Δt4=T5-T4, S13. Calculate the historical measured delay of each stage of a task cycle, the system sets a fixed length history delay queue to store the measured delay of the last N tasks, when a new task prediction total delay is needed, the phased prediction method is used to predict the delay of each sub-stage respectively, to get the predicted visual processing delay Δt1', the predicted simulation calculation Δt2', the predicted network transmission delay Δt3' and the predicted processing and motion delay Δt4', and then sum to get the total delay time Δt: Δt=Δt1'+Δt2'+Δt3'+Δt4'.

[0021] Compared with the prior art, the advantages of the present application are that the virtual-real synchronous mapping and optimization control are realized through digital twin technology, the problems in the production process can be predicted in advance and the robot operation can be adjusted in time, the production downtime is reduced, and the overall operation efficiency of the production line is improved. Based on multi-source data deep fusion and accurate feature extraction, the system can more accurately identify the position and state of the workpiece, ensure the efficiency and safety of the production line, and be easy to use. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 It is a structure schematic diagram of a production line robot vision digital system based on digital twin technology. DETAILED DESCRIPTION

[0023] The present application will be further described in detail below with reference to the accompanying drawings.

[0024] The present application will be further described in detail below with reference to the accompanying drawings. Figure 1As shown, a production line robot vision digitalization system based on digital twin technology includes a digital twin module, a data fusion processing module, an interactive control module and a physical unit; the physical unit includes an industrial robot and a multi-source sensor array, the multi-source sensor array includes a laser radar and a CCD camera installed on the industrial robot; the physical unit is in communication connection with the data fusion processing module, the data fusion processing module performs space-time alignment fusion and feature extraction on the acquired robot pose data, laser radar data and vision data; the digital twin module is based on virtual-real synchronization to map the running state of the physical production line in real time, and the interactive control module is used to display the mapping data of the digital twin module and generate optimized control instructions feedback to the physical unit.

[0025] In one embodiment, the digital twin module includes a modeling unit, a simulation unit and a rendering unit; the modeling unit constructs a three-dimensional model of the work production line space; the simulation unit obtains the output information of the data fusion processing module to simulate the kinematic behavior of the robot; and the rendering unit performs real-time visualization processing on the three-dimensional model and simulation data.

[0026] In one embodiment: The interactive control module includes a simulation verification unit, a delay compensator and a safety judge; the simulation verification unit performs pre-verification on the optimized control instructions and outputs instruction results; the delay compensator outputs the execution delay of the physical unit and generates compensation instructions; and the safety judge includes a threshold for judging the virtual-real deviation in the digital twin module, and performs emergency stop protection when the deviation exceeds the threshold.

[0027] The data fusion processing module includes: voxel filtering and ground segmentation of laser radar point cloud data, extraction of workpiece three-dimensional contour features; distortion correction and feature matching based on images collected by the CCD camera, identification of workpiece surface texture marks; fusion of robot pose data and sensor data, output of space-time aligned workpiece six-degree-of-freedom pose.

[0028] In one embodiment: The same group of industrial robots also has a sensor synchronization controller for time synchronization of laser radar, CCD camera and robot pose information, the modeling unit includes a device static model and a workpiece dynamic model for constructing a three-dimensional model of the space, and the interactive control module also includes a multi-level permission management unit, which sets three levels of permissions including operators, process engineers and system administrators.

[0029] In the specific implementation of the present application: The industrial robot is fixedly installed, a high-precision and high-resolution laser radar sensor is selected, and the sensor is installed at the end effector of the industrial robot to ensure scanning of the production line environment and accurate acquisition of point cloud data; A CCD camera with high pixels and high frame rate is selected, and the installation angle and position of the camera are adjusted according to actual shooting requirements to ensure that the surface image of the workpiece can be clearly collected; A sensor synchronization controller is arranged in the same group of industrial robots, and the laser radar, the CCD camera and the robot pose information are time-synchronized to ensure the consistency of the multi-source data in the time dimension.

[0030] During operation, the CAD drawing of the equipment is obtained and imported into the digital twin development platform to construct a static model of the equipment, point cloud data of the production line environment is collected by sensors such as laser radars, a dynamic model of the workpiece is generated by using a point cloud processing algorithm, the static model and the dynamic model are fused to construct a complete spatial three-dimensional model; Based on the output information of the data fusion processing module, a robot kinematics model is established to simulate the motion behavior of the robot, and the motion process of the robot is accurately simulated by setting simulation parameters such as motion speed and acceleration; the three-dimensional model and the simulation data are real-time visualized by a rendering unit, and effects such as lighting and material are added to improve the realism of the virtual scene; The simulation verification unit simulates the instruction execution process in the digital twin model, analyzes the motion trajectory and pose change of the robot, evaluates the feasibility and effectiveness of the instruction, and outputs the instruction result; According to the actual execution delay of the industrial robot in the physical unit, the difference between the instruction issuing time and the actual execution time of the robot is monitored in real time to generate a compensation instruction. The safety judge sets a threshold for judging the virtual-real deviation in the digital twin module, and compares the state of the robot in the digital twin model with the actual state of the robot in the physical unit in real time. When the deviation exceeds the threshold, the emergency stop protection mechanism is triggered immediately to stop the robot running and ensure production safety.

[0031] When the three-level permissions of the operator, the process engineer and the system administrator in the application are used, the operator can only perform basic production operation monitoring, the process engineer can adjust the process parameters, and the system administrator has the highest permission and can perform system configuration and management operations.

[0032] The specific steps of the delay compensator outputting the physical unit execution delay to generate a compensation instruction are: S1. Establish a delay prediction model to estimate the total delay time Δt of the completion of the current action of the physical unit, wherein the total delay time Δt is the sum of the visual processing delay, the simulation calculation delay, the network transmission delay, the processing and motion delay.

[0033] The total delay time Δt is calculated by using the timestamp method, and the specific steps are as follows: S11. All physical units in the system use a precise time protocol, such as Network Time Protocol (NTP) or Precision Time Protocol (PTP / IEEE 1588), to ensure that all components in the system have timestamps based on the same clock source, and the time error between them should be within sub-millisecond. The following key nodes are time-stamped: a. The physical moment when the CCD camera exposure ends, time-stamped T1; this timestamp is usually written directly into the image metadata by the camera hardware or acquisition card when the image data is generated, and is the most accurate.

[0034] b. The moment before the digital twin module sends the optimized control instructions to the simulation verification unit, time-stamped T2; this timestamp is generated by software.

[0035] c. The moment when the simulation verification unit completes the rehearsal verification and outputs the final instruction result to the delay compensator, time-stamped T3.

[0036] d. The moment when the industrial robot receives the final motion instruction, time-stamped T4.

[0037] e. The moment when the industrial robot completely executes the instruction and feeds back the "task completion" signal, time-stamped T5.

[0038] S12. Delay decomposition calculation Calculate the delay of each stage: Visual processing delay, Δt1 = T2 - T1, which is the time spent from taking a photo to generating instructions.

[0039] Simulation calculation delay, Δt2 = T3 - T2, which is the time spent by the simulation unit in rehearsal.

[0040] Network transmission delay, Δt3 = T4 - T3, which is the network transmission time from the simulation unit to the robot controller.

[0041] Processing and motion delay, Δt4 = T5 - T4, which is the time spent by the controller in parsing instructions, planning paths, and driving the robot to execute until completion. This is the most variable part of the delay, depending on the distance of the target point and the motion speed of the robot.

[0042] S13. According to S12, the historical measured delay of each stage of a task cycle is calculated, and a fixed-length historical delay queue is set to store the measured delay of each stage of the last N tasks. N can be a suitable number that can make more accurate prediction, such as N = 100 times. When a new task needs to be predicted, the delay of each sub-stage is predicted respectively by using the stage-by-stage prediction method to obtain the predicted visual processing delay Δt1', the predicted simulation calculation delay Δt2', the predicted network transmission delay Δt3', and the predicted processing and motion delay Δt4'. There are many methods for calculating the delay of each sub-stage, such as simple average method, exponential weighted moving average method, sliding window average method, etc. Then the total delay time Δt is obtained by summation: Δt = Δt1' + Δt2' + Δt3' + Δt4'. By predicting the delay of each stage respectively and then summing, the overall prediction accuracy can be significantly improved. Through this detailed timestamp recording and calculation method, the system can obtain an estimated relatively reliable total delay time, which lays a solid foundation for accurate compensation in the following steps.

[0043] S2. Using the total delay time Δt, the predicted future state of the physical unit after the current time plus the total delay time Δt is calculated.

[0044] S3. Based on the predicted future state in step S2, a compensation instruction is generated and output to the physical unit for execution.

[0045] Based on the present application, the robot grasps a workpiece on a moving conveyor belt on a production line as an example: The scene is set as a straight-line motion conveyor belt moving at a constant speed V = 0.2 m / s, and the robot must accurately grasp a workpiece from the conveyor belt, equipped with a visual digital system with a delay compensator of the present application.

[0046] 1. Initial instruction generation The CCD camera takes a picture at t0, and identifies the target workpiece.

[0047] Through image processing, the accurate position coordinates P_current = (x0, y0, z0) of the workpiece at the current time are calculated. It is assumed that x0 = 1.0 meters (assuming the conveyor belt moves along the X-axis).

[0048] Based on this position, a trajectory of the robot motion is planned, and an initial control instruction is generated: "move the end effector (gripper) of the robot to position P_current (1.0, y0, z0) and perform grasping".

[0049] 2. Simulation verification and delay prediction The simulation verification unit receives this initial instruction and pre-rehearses in a virtual environment. It finds that the trajectory is collision-free and reachable, and the verification passes. It outputs "instruction feasible, ideal result is to successfully grab at P_current point."

[0050] Meanwhile, the delay prediction model starts working.

[0051] It estimates from historical data that the total delay time of the entire system Δt = 0.5 seconds from the visual shooting to the robot finally executing the grabbing instruction.

[0052] This delay includes image processing time, simulation calculation time, instruction transmission time, and the time required for the robot itself to move to the target point.

[0053] 3. State prediction The digital twin module is based on two key inputs: the current state of the workpiece: position P_current = 1.0 meters, speed V = 0.2 meters / second (uniform speed). Total delay Δt = 0.5 seconds.

[0054] Perform prediction calculation: Since the conveyor belt moves at a constant speed, a linear prediction model is used.

[0055] The moving distance D of the workpiece within Δt time is V * Δt = 0.2 m / s * 0.5 s = 0.1 meters.

[0056] Therefore, at t0 + Δt time (i.e. the time when the robot actually executes the grabbing action), the position of the workpiece will no longer be 1.0 meters, but P_future = P_current + (V * Δt) = 1.0 + 0.1 = 1.1 meters.

[0057] 4. Generate compensation instructions "In order to let the robot grab the workpiece at future t0 + Δt time, perform correction calculation, the original instruction received is: "go to P_current (1.0, y0, z0)".

[0058] And the prediction information is: "the workpiece will be in P_future (1.1, y0, z0) in the future".

[0059] The modifier generates compensation instructions: "modify the target point to P_future", that is, "move the robot's end effector to position (1.1, y0, z0) and execute the grab".

[0060] 5. Execution The compensation command is sent to the robot controller and the robot starts to move. Since the robot's own movement time is calculated in, it exactly moves to the target point (1.1, y0, z0) at about t0 + 0.5 seconds. This makes the robot's gripper and the workpiece meet at the same time, at the same spatial point, and the grab is successful.

[0061] The contents not described in detail in the specification belong to the prior art known to those skilled in the art.

[0062] In the present application, unless otherwise explicitly specified and limited, "on" or "under" of a first feature to a second feature can include that the first and second features are in direct contact, or can include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, "on", "above" and "over" of a first feature to a second feature include that the first feature is directly above and obliquely above the second feature, or only means that the first feature is higher than the second feature in horizontal height. "Under", "below" and "underneath" of a first feature to a second feature include that the first feature is directly below and obliquely below the second feature, or only means that the first feature is lower than the second feature in horizontal height The above describes the present application and its embodiments, which are not limited, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired by it, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solution can be designed, which should belong to the protection scope of the present application.

Claims

1. A production line robot vision digitalization system based on digital twin technology, characterized in that: The physical unit comprises an industrial robot and a multi-source sensor array, the multi-source sensor array comprising a laser radar and a CCD camera mounted on the industrial robot. The physical unit is in communication connection with the data fusion processing module, the data fusion processing module performing space-time alignment fusion and feature extraction on acquired robot pose data, laser radar data and vision data. The digital twin module is used for real-time mapping of the running state of the physical production line based on virtual-real synchronization, the interactive control module is used for displaying mapping data of the digital twin module, and an optimized control instruction is generated and fed back to the physical unit. The digital twin module comprises a modeling unit, a simulation unit and a rendering unit.

2. The line robot vision digitalization system based on digital twin technology according to claim 1, characterized in that: The modeling unit constructs a three-dimensional model of the work production line space. The simulation unit acquires output information of the data fusion processing module to simulate kinematic behavior of the robot. The rendering unit performs real-time visual processing on the three-dimensional model and simulation data. The interactive control module comprises a simulation verification unit, a delay compensator and a safety judge.

3. The line robot vision digitalization system based on digital twin technology according to claim 1, characterized in that: The simulation verification unit performs pre-rehearsal verification on the optimized control instruction, and outputs an instruction result. The delay compensator outputs an execution delay of the physical unit, and generates a compensation instruction. The safety judge comprises a threshold value for judging virtual-real deviation in the digital twin module, and performs emergency stop protection when the deviation exceeds the threshold value. The data fusion processing module comprises:

4. The line robot vision digitalization system based on digital twin technology according to claim 1, characterized in that: The laser radar point cloud data is voxel filtered and ground segmented to extract three-dimensional contour features of the workpiece. Distortion correction and feature matching are performed on images collected by the CCD camera to identify surface texture markers of the workpiece. Robot pose data and sensor data are fused to output a six-degree-of-freedom pose of the workpiece in space-time alignment. The same group of the industrial robot further comprises a sensor synchronization controller for time synchronization of the laser radar, the CCD camera and the robot pose information.

5. The line robot vision digitalization system based on digital twin technology according to claim 4, characterized in that: The modeling unit comprises a device static model obtained from CAD drawings and a workpiece dynamic model obtained from point cloud data to construct a space three-dimensional model.

6. The line robot vision digitalization system based on digital twin technology according to claim 2, characterized in that: The interactive control module further comprises a multi-level permission management unit, which sets three levels of permissions including operators, process engineers and system administrators.

7. The line robot vision digitalization system based on digital twin technology according to claim 1, characterized in that: The specific steps of the delay compensator outputting an execution delay of the physical unit to generate a compensation instruction are as follows:

8. The line robot vision digitalization system based on digital twin technology according to claim 3, characterized in that: S1. Establish a delay prediction model to estimate the total delay time Δt of the completion of the current action of the physical unit; S2. Use the total delay time Δt to calculate the predicted future state of the physical unit after the current time plus the total delay time Δt; S3. Based on the predicted future state in step S2, generate a compensation instruction and output it to the physical unit for execution. The total delay time Δt is the sum of the predicted vision processing delay, simulation calculation delay, network transmission delay, processing and motion delay.

9. The line robot vision digitalization system based on digital twin technology according to claim 8, characterized in that:

10. The production line robot vision digitalization system based on digital twin technology according to claim 8, wherein: The total delay time Δt is calculated by the timestamp method, and the specific steps are as follows: S11. Use the precise time protocol in all physical units in the entire system, and stamp the following key nodes with timestamps: ​ a. The physical time when the CCD camera exposure ends is time-stamped T1; b. The time when the digital twin module sends the optimized control instruction to the simulation verification unit is time-stamped T2; c. The time when the simulation verification unit completes the rehearsal verification and outputs the final instruction result to the delay compensator is time-stamped T3; d. The time when the industrial robot receives the final motion instruction is time-stamped T4; e. The time when the industrial robot completely executes the instruction and feeds back the "task completion" signal is time-stamped T5; S12. Delay decomposition calculation Calculate the delay of each stage: Visual processing delay Δt1 = T2 - T1, Simulation calculation delay Δt2 = T3 - T2, Network transmission delay Δt3 = T4 - T3, Processing and motion delay Δt4 = T5 - T4, S13. Calculate the historical measured delays of each stage for a task cycle. The system sets a fixed-length historical delay queue to store the measured delays of the last N tasks. When a new task needs to be predicted for the total delay, use the stage-by-stage prediction method to predict the delay of each sub-stage to obtain the predicted visual processing delay Δt1', the predicted simulation calculation delay Δt2', the predicted network transmission delay Δt3', and the predicted processing and motion delay Δt4'. Then sum them up to get the total delay time Δt: Δt = Δt1' + Δt2' + Δt3' + Δt4'.

Images