Industrial pipeline control method and apparatus

Abstract

This invention discloses an industrial assembly line control method and apparatus. The method includes: when material is detected arriving on the industrial assembly line, acquiring three-dimensional point cloud data of the material and generating a conveyor belt operation model based on the instantaneous speed of the conveyor belt; extracting features from the three-dimensional point cloud data to obtain a classification label for the material; using the conveyor belt operation model to perform position compensation on the material to obtain compensated position information; generating motion control commands based on the classification label, the compensated position information, and the current operating state of the robot corresponding to the industrial assembly line; and controlling the robot to perform a grasping action on the material according to the motion control commands to execute the unloading task. This invention solves the technical problems in related technologies where low accuracy of dynamic material perception and poor coordination between components in industrial assembly line control systems lead to inaccurate positioning and low reliability.

Description

Technical Field

[0001] This invention relates to the field of automated production technology, and more specifically, to an industrial production line control method and apparatus. Background Technology

[0002] With the rapid development of intelligent manufacturing and flexible production systems, industrial robot production lines are widely used in automation devices and intelligent sorting in fields such as home appliances, electronics, and automobiles. However, current mainstream systems face multiple technical bottlenecks in dynamic material sensing, multi-source collaborative positioning, real-time decision control, and production line simulation optimization, making it difficult to meet the demands of modern production that requires high precision, high efficiency, diverse varieties, and small batches.

[0003] Existing industrial production lines generally rely on vision cameras to identify and locate materials. However, vision systems are significantly affected by ambient light, material reflection, shadows, and dust interference, making them prone to false detections and missed detections in complex industrial environments, resulting in a high failure rate in material handling.

[0004] Furthermore, existing systems generally neglect the dynamic motion characteristics of conveyor belts. The material position data collected by lidar or sensors are instantaneous static points, without spatiotemporal compensation based on the real-time speed of the conveyor belt. When the robot performs a grasping action based on these static coordinates, the displacement of the material during the transmission process causes a spatial deviation between the end effector and the target material. Especially in circular or high-speed conveyor belt systems, this deviation can reach several centimeters, causing grasping failure, material falling, or even equipment collisions, significantly reducing system reliability.

[0005] Furthermore, in terms of material classification, most systems rely on preset thresholds, rule bases, or simple template matching to achieve classification, lacking the ability to adaptively identify irregularly shaped, non-standard, and multi-sized materials. Adding new material categories requires manual reconfiguration of rules, resulting in poor system scalability and an inability to meet the intelligent requirements of flexible manufacturing for "one-click production line switching."

[0006] At the system debugging and optimization level, the parameter adjustment and path planning of existing production lines rely heavily on repeated trial and error with physical prototypes, resulting in long debugging cycles, high costs, and difficulty in reproducing abnormal operating conditions. More importantly, there are serious data silos between the various subsystems (sensing, control, scheduling, and simulation), and a lack of a unified data acquisition, storage, analysis, and feedback mechanism.

[0007] There is currently no effective solution to the above problems. Summary of the Invention

[0008] This invention provides an industrial assembly line control method and apparatus to at least solve the technical problems in related technologies where the low accuracy of dynamic material sensing and poor coordination between components in industrial assembly line control systems lead to inaccurate positioning and low reliability.

[0009] According to one aspect of the present invention, an industrial production line control method is provided, comprising: when material is detected arriving on the industrial production line, acquiring three-dimensional point cloud data of the material, and generating a conveyor belt operation model based on the instantaneous speed of the conveyor belt of the industrial production line; extracting features from the three-dimensional point cloud data to obtain a classification label for the material; using the conveyor belt operation model to perform position compensation on the material to obtain compensated position information of the material; generating motion control instructions based on the classification label, the compensated position information, and the current working state of the robot corresponding to the industrial production line; and controlling the robot to perform a grasping action on the material according to the motion control instructions to perform a material unloading task.

[0010] Optionally, the industrial assembly line control method further includes: after the industrial assembly line is started, calibrating the clocks of each component on the industrial assembly line using the clock of the robot controller on the industrial assembly line as the master clock, wherein the component includes at least one of the following: a data acquisition component, a sensing component, and a control component.

[0011] Optionally, when material is detected arriving on the industrial production line, the three-dimensional point cloud data of the material is collected, including: detecting the position of the material using a through-beam sensor on the industrial production line; when the through-beam sensor detects that the material has entered the scanning area of ​​the lidar on the industrial production line, the lidar is triggered to start, so as to perform a data acquisition operation on the material and obtain the three-dimensional point cloud data.

[0012] Optionally, feature extraction is performed on the three-dimensional point cloud data to obtain the classification label of the material, including: uploading the three-dimensional point cloud data to a point cloud data processing module in the cloud, using the point cloud data processing module to extract features from the three-dimensional point cloud data to obtain feature data of the material; obtaining the material type of the material based on the feature data; and obtaining the classification label according to the material type.

[0013] Optionally, generating a conveyor belt operation model based on the instantaneous speed of the conveyor belt in the industrial production line includes: acquiring the rotation pulse signal of the drive motor of the conveyor belt through an encoder on the industrial production line; calculating the instantaneous speed of the conveyor belt based on the rotation pulse signal; and generating the conveyor belt operation model according to the characteristic information of the conveyor belt and the instantaneous speed of the conveyor belt.

[0014] Optionally, the material is positionally compensated using the conveyor belt operation model to obtain the compensated position information of the material, including: acquiring the initial position of the material in the lidar coordinate system and the point cloud detection timestamp of the three-dimensional point cloud data; processing the conveyor belt operation model within a time interval to obtain the total displacement and displacement components of the material in the direction of movement of the conveyor belt; and superimposing the displacement components onto the initial position to obtain the compensated position information.

[0015] Optionally, the industrial production line control method further includes: based on the compensated position information and the calibration rigid body transformation matrix between the lidar and the robot, expanding the three-dimensional point cloud data coordinates in the lidar coordinate system to a homogeneous coordinate form to obtain the spatial position information of the material in the robot base coordinate system.

[0016] Optionally, generating motion control instructions based on the classification label, the compensated position information, and the current operating state of the robot corresponding to the industrial production line includes: determining a material grasping strategy by matching a preset grasping strategy library with the spatial position information of the classification label and the compensated position information in the robot base coordinate system through a task scheduling algorithm; selecting a target robot based on the current operating state; and generating the motion control instructions based on the material grasping strategy and the feature information of the target robot.

[0017] Optionally, the industrial production line control method further includes: after determining that the robot's feeding has failed, dynamically correcting the robot's movement path to obtain a corrected path; controlling the robot to move according to the corrected path to grab the material, and performing the feeding task on the material.

[0018] Optionally, the industrial assembly line control method further includes: using a simulation platform and based on the assembly line characteristic information of the industrial assembly line and the robot characteristic information of the robot, performing simulation processing on the industrial assembly line and the robot to obtain a material unloading simulation environment for the robot; controlling the robot model corresponding to the robot to perform an efficiency task on the material in the material unloading simulation environment to obtain a material unloading simulation result; and generating unloading prompt information based on the material unloading simulation result.

[0019] According to another aspect of the present invention, an industrial production line control device is also provided, comprising: a data acquisition unit, configured to acquire three-dimensional point cloud data of the material when the arrival of the material on the industrial production line is detected, and generate a conveyor belt operation model based on the instantaneous speed of the conveyor belt of the industrial production line; a feature extraction unit, configured to extract features from the three-dimensional point cloud data to obtain a classification label for the material; a compensation unit, configured to perform position compensation on the material using the conveyor belt operation model to obtain compensated position information of the material; an instruction generation unit, configured to generate motion control instructions based on the classification label, the compensated position information, and the current working state of the robot corresponding to the industrial production line; and a first control unit, configured to control the robot to perform a grasping action on the material according to the motion control instructions to perform a material unloading task.

[0020] Optionally, the industrial production line control device further includes: a calibration unit, used to calibrate the clocks of various components on the industrial production line by using the clock of the robot controller on the industrial production line as the main clock after the industrial production line is started, wherein the components include at least one of the following: a data acquisition component, a sensing component, and a control component.

[0021] Optionally, the acquisition unit includes a detection module, used to detect the position of the material using a through-beam sensor on the industrial production line. When the through-beam sensor detects that the material has entered the scanning area of ​​the lidar on the industrial production line, it triggers the lidar to start, so as to perform data acquisition operations on the material and obtain the three-dimensional point cloud data.

[0022] Optionally, the feature extraction unit includes: an upload module for uploading the three-dimensional point cloud data to a point cloud data processing module in the cloud, so as to use the point cloud data processing module to extract features from the three-dimensional point cloud data to obtain feature data of the material; a first acquisition module for obtaining the material type of the material based on the feature data; and a second acquisition module for obtaining the classification label according to the material type.

[0023] Optionally, the acquisition unit includes: an acquisition module for acquiring rotational pulse signals of the drive motor of the conveyor belt through an encoder on the industrial production line; a calculation module for calculating the instantaneous speed of the conveyor belt based on the rotational pulse signals; and a first generation module for generating the conveyor belt operation model based on the characteristic information of the conveyor belt and the instantaneous speed of the conveyor belt.

[0024] Optionally, the compensation unit includes: a third acquisition module, used to acquire the initial position of the material in the lidar coordinate system and the point cloud detection timestamp of the three-dimensional point cloud data; a processing module, used to process the conveyor belt running model within a time interval to obtain the total displacement and displacement components of the material in the direction of movement of the conveyor belt; and a superposition module, used to superimpose the displacement components onto the initial position to obtain the compensated position information.

[0025] Optionally, the industrial production line control device further includes: a calibration unit, used to calculate the spatial position information of the material in the robot base coordinate system based on the compensated position information and the calibration rigid body transformation matrix between the lidar and the robot; and an expansion unit, used to expand the coordinates of the three-dimensional point cloud data in the lidar coordinate system into homogeneous coordinates to obtain the spatial position information of the material in the robot base coordinate system.

[0026] Optionally, the instruction generation unit includes: a matching module, used to determine a material grasping strategy by matching a preset grasping strategy library with the spatial position information of the classification label and the compensated position information in the robot base coordinate system through a task scheduling algorithm; a selection module, used to select a target robot according to the current operation status; and a second generation module, used to generate the motion control instruction according to the material grasping strategy and the feature information of the target robot.

[0027] Optionally, the industrial production line control device further includes: a correction unit, used to dynamically correct the movement path of the robot after determining that the robot's unloading has failed, so as to obtain a corrected path; and a second control unit, used to control the robot to move according to the corrected path to grab the material and perform the unloading task on the material.

[0028] Optionally, the industrial assembly line control device further includes: a simulation unit, used to perform simulation processing on the industrial assembly line and the robot through a simulation platform, based on the assembly line feature information of the industrial assembly line and the robot feature information of the robot, to obtain the material unloading simulation environment of the robot; a third control unit, used to control the robot model corresponding to the robot to perform efficiency tasks on the material in the material unloading simulation environment, so as to obtain the material unloading simulation result; and a third generation module, used to generate unloading prompt information based on the material unloading simulation result.

[0029] According to one aspect of the present invention, an industrial assembly line control system is provided, wherein the industrial assembly line control system uses the industrial assembly line control method described in any one of the above embodiments.

[0030] According to one aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the industrial production line control method described in any one of the foregoing embodiments.

[0031] According to one aspect of the present invention, a processor is provided for running a program, wherein the program executes the industrial production line control method described in any one of the foregoing embodiments.

[0032] According to one aspect of the present invention, a computer program product is provided, including computer instructions that, when executed by a processor, perform the industrial production line control method described in any one of the above embodiments.

[0033] By applying the technical solution of this application, the above-mentioned industrial assembly line control method, when material is detected arriving on the industrial assembly line, collects the three-dimensional point cloud data of the material and generates a conveyor belt operation model based on the instantaneous speed of the conveyor belt of the industrial assembly line; extracts features from the three-dimensional point cloud data to obtain the material's classification label; uses the conveyor belt operation model to perform position compensation on the material to obtain the compensated position information of the material; generates motion control commands based on the classification label, the compensated position information, and the current working state of the robot corresponding to the industrial assembly line; controls the robot to perform a grasping action on the material according to the motion control commands to perform the unloading task. This achieves multi-modal collaborative perception through lidar and encoder, cross-coordinate system rigid body transformation positioning, cloud-based intelligent classification, and "perception-decision-execution-feedback" full-link digital twin closed-loop control, solving the technical problems of low dynamic material perception accuracy and poor coordination between components in industrial assembly line control systems, which lead to inaccurate positioning and low reliability. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0035] Figure 1 This is a hardware structure block diagram of a mobile terminal for an industrial production line control method according to an embodiment of the present invention.

[0036] Figure 2 This is a flowchart of an industrial production line control method according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of a digital twin intelligent control system according to an embodiment of the present invention;

[0038] Figure 4This is a diagram of the architecture of a digital twin intelligent control system according to an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of a timestamp synchronization structure according to an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the pipeline scene and coordinate system mapping according to an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of an intelligent classification algorithm model according to an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram of the cloud data storage table structure according to an embodiment of the present invention;

[0043] Figure 9 This is a schematic diagram of the motion compensation algorithm according to an embodiment of the present invention;

[0044] Figure 10 This is a schematic diagram of the simulation platform structure according to an embodiment of the present invention;

[0045] Figure 11 This is an optional digital twin intelligent control system operation flowchart according to an embodiment of the present invention;

[0046] Figure 12 This is a schematic diagram of an industrial production line control device according to an embodiment of the present invention.

[0047] The above figures include the following reference numerals:

[0048] 102. Processor; 104. Memory; 106. Transmission Equipment; 108. Input / Output Device; 301. Production Line; 302. Conveyor Belt; 303. Conveyor Belt Controller; 304. LiDAR; 305. Through-beam Sensor; 306. Encoder; 307. Robot Controller; 308. Material Rack; 309. Robot Support; 310. Timestamp Synchronization Module; 311. Data Processing and Simulation Module; 312. Cloud Module; 313. Simulation Platform; 314. Data Storage Module; 315. Record Backtracking Module; 316. Task Scheduling Algorithm; 317. Intelligent Classification Algorithm; 318. Point Cloud Processing Algorithm; 319. Coordinate Transformation Algorithm; 320. Motion Compensation Algorithm; 321. Motion Correction Algorithm; 322. Velocity Modeling Algorithm; 323. Robot Motion Simulation Module; 324. Conveyor Belt Motion Simulation Module. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0051] As described in the background section, related technologies in industrial production line control systems suffer from low accuracy in dynamic material sensing and poor coordination between components, leading to inaccurate positioning and low reliability. Embodiments of this invention provide an industrial production line control method and apparatus, an industrial production line, a computer-readable storage medium, a processor, and a computer program product.

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0053] The methods and embodiments provided in this invention can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for an industrial production line control method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0054] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the industrial production line control method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0055] Example 1

[0056] According to an embodiment of the present invention, a method embodiment of an industrial production line control method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0057] Figure 2 This is a flowchart of an industrial production line control method according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0058] Step S202: When material is detected arriving on the industrial production line, collect the three-dimensional point cloud data of the material and generate a conveyor belt operation model based on the instantaneous speed of the conveyor belt on the industrial production line.

[0059] In this embodiment, "collecting three-dimensional point cloud data of materials" refers to using lidar to perform non-contact scanning of materials moving on the conveyor belt, obtaining a set of surface points in three-dimensional space to fully characterize the material's shape and spatial posture; "generating a conveyor belt operation model based on the instantaneous speed of the conveyor belt" refers to using the pulse signals fed back by the encoder in real time to calculate the current operating speed of the conveyor belt, and combining it with its motion trajectory to establish a dynamic displacement prediction model, which is used to quantify the spatial displacement of the material caused by the movement of the conveyor belt, thereby providing a physical basis for subsequent position correction.

[0060] In this method, when the through-beam sensor detects that the material has entered the scanning area of ​​the lidar, the lidar is triggered to perform a three-dimensional point cloud scan of the material to obtain its spatial contour data in the lidar coordinate system. At the same time, the encoder collects the pulse signal of the conveyor belt in real time, and calculates the instantaneous running speed of the conveyor belt at each moment by combining the calibrated pulse-displacement coefficient. Based on this speed information, a dynamic displacement prediction model that changes with time is constructed to characterize the spatial displacement trend of the material during the movement of the conveyor belt.

[0061] By implementing this control method, non-contact 3D shape capture of dynamic materials and accurate modeling of conveyor belt motion were achieved. This overcomes the problem of traditional vision systems failing under varying lighting conditions or low contrast environments, providing a highly reliable raw data foundation and physical basis for subsequent motion compensation and precise positioning.

[0062] Step S204: Extract features from the 3D point cloud data to obtain the material classification labels.

[0063] In this embodiment, "feature extraction from 3D point cloud data" refers to automatically identifying and quantifying the geometric attributes of materials from point clouds collected by LiDAR, including structured features such as length, width, height, outline shape, center of gravity position, and attitude angle; the classification label of the material refers to inputting the extracted features into an intelligent classification model deployed in the cloud, comparing it with a historical material sample library through a pre-trained deep learning algorithm, and outputting the category identifier to which the material belongs, rather than relying on manually preset rules.

[0064] In this method, the 3D point cloud data collected by LiDAR is input into an intelligent classification model deployed in the cloud. The model extracts the geometric features of the point cloud, such as the outer contour area, length, width and height dimensions, center of gravity offset, surface curvature distribution and attitude angle of the material, and other structured parameters. Combined with a pre-trained deep learning network, these features are matched with category templates in a historical material sample library, and finally outputs a classification label with confidence, rather than relying on manually preset thresholds or rules.

[0065] By implementing this control method, the system can automatically and intelligently classify materials with irregular shapes, varying sizes, and diverse materials. This eliminates the need for traditional sorting methods based on fixed templates or manually set rules, significantly improves the system's adaptability to a wide variety of small-batch materials in flexible production, reduces the cost of manual intervention, and provides reliable data input for subsequent adaptive task allocation.

[0066] Step S206: Use the conveyor belt running model to perform position compensation on the material to obtain the compensated position information of the material.

[0067] In this embodiment, "using the conveyor belt running model to compensate for the position of the material" means that based on the instantaneous speed model of the conveyor belt established in step S202, combined with the timestamp of the point cloud collected by the lidar and the delay time of the robot's grasping, the displacement vector generated by the material during the movement of the conveyor belt is calculated by numerical integration, and tangent correction is performed according to the direction of the conveyor belt movement. Finally, after the original point cloud coordinates are transformed from the lidar coordinate system to the robot coordinate system, the motion compensation amount is superimposed to obtain the true spatial position of the material at the moment of robot grasping.

[0068] This method is based on the instantaneous speed model of the conveyor belt established in step S202. It combines the timestamp of the point cloud collected by the lidar with the time delay of the robot's actual grasping action. The displacement of the material along the tangential direction during the movement of the conveyor belt is calculated by numerical integration. The displacement vector is corrected according to the trajectory of the conveyor belt. The original point cloud coordinates are then mapped from the lidar coordinate system to the robot coordinate system through rigid body transformation. This motion compensation is then superimposed to obtain the true spatial position of the material at the moment of robot grasping.

[0069] By implementing this control method, the positioning deviation caused by the "detection-execution" time difference due to the continuous movement of the conveyor belt is effectively eliminated, ensuring that the robot's gripping point is always aligned with the actual endpoint of the material's movement. This significantly improves the gripping accuracy in dynamic scenarios, controls the positioning error to the millimeter level, and greatly reduces the risks of material leakage, mis-grabbing, and collisions.

[0070] Step S208: Generate motion control commands based on the classification labels, compensated position information, and the current operating status of the robot corresponding to the industrial production line.

[0071] In this embodiment, "generating motion control instructions based on classification labels, compensated position information, and the current operating status of the robot corresponding to the industrial production line" refers to jointly analyzing the material category (such as material ID and type label) output in step S204, the corrected three-dimensional coordinates output in step S206, and the robot's current arm position, task queue, load status, and other operating information. The task scheduling algorithm then dynamically plans the optimal grasping path, grasping posture, and execution priority to generate a complete control sequence containing target position, movement speed, and end-effector action instructions.

[0072] In this method, the material classification labels (such as material type and priority) output in step S204 and the compensated three-dimensional position information output in step S206, along with real-time running data such as the robot's current joint posture, task queue, load status, and path occupancy, are input into the task scheduling algorithm. The algorithm comprehensively judges the optimal grasping order, motion path, end-effector posture, and action timing to generate a complete motion control sequence that includes target coordinates, motion speed, acceleration curve, and gripper switch commands.

[0073] By implementing this control method, intelligent collaborative decision-making based on material properties, spatial location, and robot capabilities is achieved. This enables the system to dynamically adapt to complex production line scenarios such as multi-task concurrency, priority changes, and robot resource conflicts, avoiding instruction conflicts and idle waiting, improving overall scheduling efficiency and system flexibility, and supporting efficient automated operation under multi-variety, small-batch production modes.

[0074] In step S210, the robot is controlled to perform a gripping action on the material according to the motion control command in order to perform the unloading task.

[0075] In this embodiment, "controlling the robot to perform a gripping action on the material according to the motion control command to perform the unloading task" means that the motion control command generated in step S208 drives the execution mechanism in real time through the robot controller, so that the robot end effector moves along the planned path to the compensated position, completes the precise gripping and transfers to the designated unloading point, and at the same time confirms the success of the gripping through the end force sensor and position feedback, and sends the execution result back to the cloud.

[0076] Figure 3 This is a schematic diagram of a digital twin intelligent control system according to an embodiment of the present invention, such as... Figure 3As shown, the system is deployed on a PC and connected via data cables to three SCARA robot controllers 307, a LiDAR 304, an encoder 306, and a through-beam sensor 305. This allows the system to acquire robot status, LiDAR point cloud data, encoder pulse signals, and through-beam sensor detection signals. The system uploads these timestamped data or signals to the cloud module 312 for data processing. Finally, task instructions are sent to the robot controllers 307 to control the robot to perform loading and unloading tasks, and simultaneously, the data is transmitted to the simulation platform 313 for real-time simulation of the production line 301. By connecting the multimodal sensing devices (LiDAR 304, encoder 306, through-beam sensor 305, etc.) to the three SCARA robot controllers 307, the PC processing unit, and the cloud module 312 via wired connections, the system ensures low-latency, high-synchronization transmission of signals throughout the "perception-processing-decision-execution" chain. Furthermore, by attaching precise timestamps to each data point, it supports the necessary time consistency for subsequent motion compensation, coordinate alignment, and closed-loop feedback.

[0077] In addition, such as Figure 3 As shown, the material rack 308 and the robot support 309 are arranged adjacent to each other. Materials can be placed on the material rack 308, and the robot can be placed on the robot support 309. The conveyor belt controller 303 is located next to the conveyor belt 302 and is used to control the conveyor belt 302.

[0078] Figure 3 The data processing and simulation module 311 may include a cloud module 312 and a simulation platform 313; wherein, the cloud module 312 includes: a data storage module 314, a record backtracking module 315, a task scheduling algorithm 316, an intelligent classification algorithm 317, a point cloud processing algorithm 318, a coordinate transformation algorithm 319, a motion compensation algorithm 320, a motion correction algorithm 321, and a velocity modeling algorithm 322; and the simulation platform 313 includes: a robot motion simulation module 323 and a conveyor belt motion simulation module 324. Figure 4 This is a diagram of the architecture of a digital twin intelligent control system according to an embodiment of the present invention, such as... Figure 4As shown, the system is divided into five layers: the execution layer (robot controller and conveyor belt controller) is responsible for executing scheduling instructions; the algorithm layer (point cloud processing algorithm, intelligent classification algorithm, task scheduling algorithm, coordinate transformation algorithm, motion correction algorithm, and motion compensation algorithm) is responsible for data computation; the data layer (feature preprocessing and data caching) is responsible for data processing and caching; the communication layer (ModbusTCP, PTP timestamp synchronization) is responsible for providing hardware and software communication support and hardware timestamp synchronization mechanism; and the perception layer (LiDAR, encoder, and through-beam sensor) is responsible for detecting material information and collecting material data. By decomposing the system into the perception layer, communication layer, data layer, algorithm layer, and execution layer, modular decoupling of hardware acquisition, data transmission, information processing, intelligent decision-making, and physical execution is achieved. This allows each layer to be independently optimized and upgraded, improving the system's maintainability and scalability, and ensuring high reliability and low latency throughout the entire process from raw signals to control instructions.

[0079] Among them, such as Figure 3 As shown, the following operations can be completed through the timestamp synchronization module 310: after the master clock is initialized, the PTP network is established, the master clock Sync message is received from the device for time calibration, the network transmission delay is compensated through the PTP transparent clock mechanism, each module collects and carries a 64-bit microsecond-level timestamp according to the specification (the material triggering time collected by the through-beam sensor, the pulse-position correlation time collected by the encoder, the point cloud frame collected by the lidar, etc.), and finally, the data is correlated and verified.

[0080] In this method, the robot controller receives the motion control command generated in step S208, drives the servo motor to move the robot end effector to the compensated position along a precise trajectory, synchronously controls the gripper to complete the grasping action, and monitors the motion state and contact force in real time through feedback devices such as encoders and force sensors during the execution process to confirm whether the material has been successfully grasped and the unloading and transfer have been completed.

[0081] By implementing this control method, a precise closed loop from perception and decision-making to physical execution is achieved, ensuring that instructions are reliably implemented in the real environment. At the same time, the execution results, such as whether the capture is successful, the execution time, and the material ID, are transmitted back to the cloud in real time, providing real data support for production traceability and retraining of classification models, thereby improving the overall operational stability and long-term intelligence level.

[0082] As described above, in this embodiment, when material is detected arriving on the industrial production line, three-dimensional point cloud data of the material is collected, and a conveyor belt operation model is generated based on the instantaneous speed of the conveyor belt on the industrial production line. Feature extraction is performed on the three-dimensional point cloud data to obtain the material's classification label. The position of the material is compensated using the conveyor belt operation model to obtain the compensated position information of the material. Motion control commands are generated based on the classification label, the compensated position information, and the current working state of the robot corresponding to the industrial production line. The robot is controlled to perform a grasping action on the material according to the motion control commands to perform the unloading task. This achieves the goal of multi-modal collaborative perception with lidar and encoder, cross-coordinate system rigid body transformation positioning, cloud-based intelligent classification, and "perception-decision-execution-feedback" full-link digital twin closed-loop control. It realizes the technical effects of dynamic material millimeter-level high-precision positioning, automatic intelligent sorting of irregularly shaped and multi-sized materials, efficient simulation debugging of the production line, and system self-optimization, significantly improving the grasping success rate, production efficiency, and flexible manufacturing capabilities.

[0083] Therefore, the technical solution provided by the above embodiments of the present invention solves the technical problem in the related art that the low accuracy of dynamic material sensing and poor coordination between components in industrial production line control systems lead to inaccurate positioning and low reliability.

[0084] According to the above embodiments of the present invention, the industrial assembly line control method further includes: after the industrial assembly line is started, calibrating the clocks of various components on the industrial assembly line using the clock of the robot controller on the industrial assembly line as the master clock, wherein the components include at least one of the following: a data acquisition component, a sensing component, and a control component.

[0085] In this embodiment, the timestamp synchronization method adopts a hierarchical architecture of "master clock-slave device". The robot controller with integrated high-stability crystal oscillator (accuracy ≤1ppm) is used as the master clock module. Through the PTP IEEE 1588v2 precise time protocol, synchronization signals are distributed to slave devices such as through-beam sensors, encoders, lidar, and cloud processing nodes via an independent industrial-grade PTP time synchronization module to achieve microsecond-level time standardization of the entire system. Figure 5 This is a schematic diagram of the timestamp synchronization structure according to an embodiment of the present invention, as shown below. Figure 5 As shown, the synchronization process is as follows: After the master clock is initialized, the PTP network is established. The slave device receives the master clock Sync message for time calibration. The network transmission delay is compensated through the PTP transparent clock mechanism. Each module collects and carries a 64-bit microsecond-level timestamp according to the specifications (the material triggering time collected by the through-beam sensor, the pulse-position correlation time collected by the encoder, the point cloud frame collected by the lidar, etc.). Finally, through data association and verification, the accurate time association of the entire "trigger-collection-processing-execution" data is ensured, providing a reliable time reference for subsequent motion compensation.

[0086] When the industrial production line starts up, a high-precision master clock (with a built-in ≤1ppm stable crystal oscillator) deployed in the robot controller is used as the global time reference. Synchronization messages are broadcast to all key components in the system, including data acquisition components such as lidar, encoders, and through-beam sensors, as well as control and computing components such as the robot controller and cloud processing nodes, through the PTP (IEEE 1588v2) precise time protocol. Each slave device automatically calibrates its local clock according to the timestamp in the message and the network latency compensation mechanism, so as to achieve microsecond-level time synchronization of the entire system.

[0087] By implementing this control method, a unified time coordinate system is provided for the accurate correlation of multimodal sensor data (such as point cloud frames, pulse signals, and trigger events), ensuring that the data throughout the process is strictly aligned in the time dimension. This is a prerequisite for the reliable operation of core algorithms such as motion compensation, cross-coordinate system positioning, and closed-loop feedback. It fundamentally solves the positioning deviation and control misordering problems caused by clock drift, and significantly improves the stability and consistency of the system in high-speed dynamic environments.

[0088] According to the above embodiments of the present invention, when material is detected arriving on an industrial production line, the three-dimensional point cloud data of the material is collected, including: detecting the position of the material by a through-beam sensor on the industrial production line; when the through-beam sensor detects that the material has entered the scanning area of ​​the lidar on the industrial production line, the lidar is triggered to start, so as to perform a data acquisition operation on the material and obtain three-dimensional point cloud data.

[0089] In this embodiment, Figure 6 This is a schematic diagram of the pipeline scene and coordinate system mapping according to an embodiment of the present invention, such as... Figure 6 As shown, the relative positions of the lidar and the robot are fixed. This section only explains the coordinate mapping relationship between the lidar shown in section 6 and its associated robot; other combinations are similar. Assuming the lidar is installed directly above the conveyor belt at a vertical distance H, and the robot support is at the same height as the conveyor belt (i.e., assuming the world coordinate system and the robot base coordinate system coincide), then the coordinates of the origin of the world coordinate system in the lidar coordinate system are: Then the point in the lidar coordinate system Position mapped to the world coordinate system The relationship exists as follows: That is, the mapping relationship between the lidar coordinate system and the world coordinate system is as follows: ,in, Represents coordinates in the world coordinate system. This represents the coordinates in the lidar coordinate system. This represents the rotation matrix, here a 4×4 identity matrix. Let represent the translation matrix. Assume the DH transformation matrix from the robot base to the robot end effector is . So, the lidar coordinate system With robot coordinate system The mapping relationship is as follows: This mapping relationship enables the conversion between points in the lidar coordinate system and points in the robot coordinate system.

[0090] This method utilizes a through-beam sensor as a high-response threshold triggering device to monitor in real time whether the material enters the preset scanning area of ​​the lidar. When the material blocks the through-beam, the sensor immediately outputs a trigger signal, which simultaneously activates the lidar to start single-frame or multi-frame point cloud acquisition, ensuring that high-power laser scanning is only started when the material enters the effective observation area, thus achieving on-demand triggering and accurate sampling.

[0091] By implementing this control method, the ineffective operating time and data redundancy of the lidar are significantly reduced, and the system's energy efficiency and data processing efficiency are improved. At the same time, the data overload and computational pressure caused by continuous scanning are avoided, ensuring that the point cloud data is highly correlated with the actual position of the material, and providing a high-quality, time-accurate input source for subsequent positioning, classification and compensation.

[0092] According to the above embodiments of the present invention, feature extraction of three-dimensional point cloud data to obtain material classification labels includes: uploading three-dimensional point cloud data to a point cloud data processing module in the cloud, using the point cloud data processing module to extract features from the three-dimensional point cloud data to obtain material feature data; obtaining the material type based on the feature data; and obtaining classification labels according to the material type.

[0093] In this embodiment, Figure 7 This is a schematic diagram of an intelligent classification algorithm model according to an embodiment of the present invention, such as... Figure 7 As shown, the material's shape, orientation, and size data are first input into the data anomaly detection model to check for any data outside the normal data threshold range. If any data is found, it indicates that the material is not waste. If all data are within the normal data threshold range, the model proceeds to the classification algorithm module. Based on a custom data priority factor, the module determines the material's classification, generates label data, and inputs the classification results into the task scheduling algorithm. Simultaneously, the data is stored in a file such as... Figure 8 The data storage table shown is for data backtracking.

[0094] It should be noted that, Figure 8 This is a schematic diagram of the cloud data storage table structure according to an embodiment of the present invention, such as... Figure 8As shown, with globally unique record IDs (RI) and material IDs (MI) as the core, a one-item-one-code full lifecycle tracking of materials is achieved; timestamps (TS) accurately record the data generation time, data types (DT) distinguish data from different stages such as perception, processing, decision-making, execution, and feedback, raw data (SD) and structured data (PD) store raw information such as LiDAR point clouds and encoder speed, and processed information such as coordinates and dimensions after compensation transformation, respectively; intelligent classification results (AR) and task execution results (TR) record classification labels and capture execution status, respectively; at the same time, the digital twin model and virtual debugging task are associated through virtual model ID (DTMID) and simulation task ID (STI), supporting simulation analysis and production line adaptation, and providing complete data support for system optimization, fault tracing, and digital twin closed loop.

[0095] In this method, the raw 3D point cloud data collected by LiDAR is uploaded to the cloud point cloud processing module through the communication layer. The geometric features of the material are automatically extracted using deep learning or geometric analysis algorithms, including structured feature vectors such as contour volume, principal axis direction, surface curvature, centroid distribution, and minimum bounding box size. Then, the material type is identified by matching the historical sample library with a pre-trained classification model, and finally, a classification label with confidence is output.

[0096] By implementing this control method, we can break through the limitations of traditional rule-based or single-vision classification, achieve high-precision automatic identification of irregularly shaped, multi-sized, and low-contrast materials, and significantly improve the system's flexibility and adaptability. By relying on cloud computing power to realize complex model inference, we can avoid the bottleneck of edge computing power. At the same time, we can support online model iteration and cross-production line knowledge sharing, so that the classification accuracy can be continuously optimized with the accumulation of data, and provide a reliable semantic foundation for subsequent intelligent scheduling and adaptive task allocation.

[0097] According to the above embodiments of the present invention, generating a conveyor belt operation model based on the instantaneous speed of the conveyor belt in an industrial production line includes: acquiring the rotation pulse signal of the drive motor of the conveyor belt through an encoder on the industrial production line; calculating the instantaneous speed of the conveyor belt based on the rotation pulse signal; and generating the conveyor belt operation model according to the characteristic information of the conveyor belt and the instantaneous speed of the conveyor belt.

[0098] Figure 9 This is a schematic diagram of the motion compensation algorithm according to an embodiment of the present invention, as shown below. Figure 9As shown, the motion compensation algorithm includes coordinate compensation for target point offset caused by conveyor belt movement, and motion correction for path offset caused by uncontrollable factors during real-time robot movement. A through-beam sensor triggers a lidar to collect material point cloud data, which is then processed by a point cloud processing unit to obtain the target position in the lidar coordinate system. Simultaneously, an encoder collects the conveyor belt pulse count, and a velocity modeling unit constructs a conveyor belt velocity model. Both are input into a position prediction module to obtain the predicted position of the material. The predicted position is converted into the target position in the robot coordinate system by a coordinate transformation algorithm, and then input into a task scheduling algorithm to generate initial motion commands, which are sent to the robot execution unit. The robot execution unit feeds back the real-time position to the motion correction algorithm, generating correction commands that are sent back to the task scheduling algorithm, forming a closed-loop control. This achieves precise motion compensation and grasping control of dynamic materials. The core computation of the entire process is supported by cloud storage and computing units, ensuring control accuracy and response efficiency in dynamic scenarios.

[0099] In this embodiment, the encoder pulse-displacement coefficient K, the radius R of the circular conveyor belt, the radar-robot extrinsic parameter matrix T, and the fixed delay (point cloud processing, robot motion) are obtained in advance to provide basic parameters for the algorithm; the encoder pulse count and synchronization timestamp are collected at a frequency of 1kHz, and the conveyor belt speed v(t) is calculated and smoothed by Kalman filtering to establish a speed-time model under uniform / variable speed scenarios.

[0100] This method uses an industrial encoder to collect the rotation pulse signal of the motor driving the conveyor belt in real time. Combined with a pre-calibrated pulse-displacement coefficient, the instantaneous linear velocity of the conveyor belt in each sampling period (e.g., 1ms) is calculated. Based on the structural characteristics of the conveyor belt (e.g., the radius of the circular path, the transmission ratio, and the inertial characteristics), a dynamic operation model is constructed. This model can characterize the position evolution of the conveyor belt under uniform, accelerated, or variable speed conditions. Combined with Kalman filtering, the velocity is smoothed to form a high-precision, disturbance-resistant velocity-time function.

[0101] By implementing this control method, a real, continuous, and reliable dynamic state input of the conveyor belt is provided for the subsequent motion compensation algorithm. This enables the system to accurately predict the spatial displacement of the material between the time of lidar sampling and the time of robot grasping, effectively eliminating the positioning deviation caused by the movement of the conveyor belt and significantly improving the accuracy of the robot's grasping position.

[0102] According to the above embodiments of the present invention, the position compensation of the material is performed using a conveyor belt running model to obtain the compensated position information of the material, including: obtaining the initial position of the material in the lidar coordinate system and the point cloud detection timestamp of the three-dimensional point cloud data; processing the conveyor belt running model within the time interval to obtain the total displacement and displacement components of the material in the direction of movement of the conveyor belt; and superimposing the displacement components onto the initial position to obtain the compensated position information.

[0103] In this embodiment, the motion correction method for path deviation caused by uncontrollable factors during real-time robot movement employs model predictive control. Based on the system dynamic model, it predicts the state over a future period and optimizes the control input to minimize the path error. The optimization objective function is as follows: ,in, This represents the lateral error from the predicted state at step k to the reference path. This represents the angular error from the predicted state at step k to the reference path. This represents the linear velocity at step k. This represents the angular velocity at step k. , and These are the weighting coefficients.

[0104] This method is based on the initial position of the material (in the radar coordinate system) collected by lidar and the corresponding point cloud detection timestamp. Combined with the previously constructed conveyor belt operation model, and using the timestamp as a reference, it accumulates the instantaneous speed of the conveyor belt through numerical integration within the dynamic time interval from the "point cloud acquisition time" to the "actual robot grasping time". The total displacement of the material in the direction of the conveyor belt movement is calculated, and then decomposed into displacement components in three-dimensional space according to the conveyor belt trajectory (such as a straight line or a circular path). Subsequently, the displacement components are superimposed on the original point cloud position in vector form to complete the dynamic compensation for the "motion fuzziness" error caused by the movement of the conveyor belt. Finally, a high-precision target position in the robot coordinate system is output.

[0105] By implementing this control method, the problem of misgrabbing caused by high-speed conveyor belts is completely eliminated, enabling the robot to achieve millimeter-level precise positioning and stable grasping in dynamic environments, significantly improving yield and system throughput. This compensation mechanism does not rely on additional hardware and can be achieved solely through timing synchronization and motion modeling, offering advantages of low cost and high accuracy.

[0106] According to the above embodiments of the present invention, the industrial production line control method further includes: expanding the three-dimensional point cloud data coordinates in the lidar coordinate system to homogeneous coordinate form based on the compensated position information and the calibration rigid body transformation matrix between the lidar and the robot, so as to obtain the spatial position information of the material in the robot base coordinate system.

[0107] In this embodiment, the total time difference is calculated based on the effective time t1 of the lidar point cloud and the robot grasping time t2. The material displacement is obtained through numerical integration, and the displacement vector is corrected by combining the tangent direction θ of the circular track. Finally, the displacement vector is converted into the target position in the robot coordinate system through a coordinate transformation algorithm.

[0108] After obtaining the material position (in the lidar coordinate system) after motion compensation, this method represents it in homogeneous coordinate form (4×1 vector). Then, using the rigid body transformation matrix (including rotation matrix R and translation vector T) between the lidar and the robot base obtained in advance through calibration process (such as hand-eye calibration or three-coordinate measurement), the point is mapped from the lidar coordinate system to the robot base coordinate system without loss through matrix multiplication, thereby achieving the unification of spatial reference.

[0109] By implementing this control method, the cross-coordinate system control problem of inconsistency between "perceived coordinates" and "execution coordinates" in multi-sensor systems is solved, enabling the robot to directly analyze the material position based on its own motion reference without manual intervention or repeated calibration, significantly improving the accuracy and reliability of grasping command generation.

[0110] According to the above embodiments of the present invention, motion control instructions are generated based on classification labels, compensated position information, and the current working state of the robot corresponding to the industrial production line. This includes: determining a material grasping strategy by matching a preset grasping strategy library with the spatial position information of the classification labels and compensated position information in the robot base coordinate system through a task scheduling algorithm; selecting a target robot based on the current working state; and generating motion control instructions based on the material grasping strategy and the feature information of the target robot.

[0111] In this embodiment, "classification label" refers to the material type identifier output by the cloud-based intelligent classification model, such as "round metal part" or "irregular plastic shell", which reflects the shape, size and posture characteristics of the material and is used to determine which grasping method is suitable for it; "motion control command" is the final generated command sequence containing specific action commands such as path coordinates, movement speed and end force control parameters, which drives the robot to complete the grasping.

[0112] The task scheduling algorithm in this method receives classification tags (such as material type, size, and center of gravity distribution) from the cloud and compensated position information in the robot coordinate system. It combines the current working status of all robots (idle, running, task queue, end-effector type, etc.) to match the optimal execution plan from the preset grasping strategy library (such as "suction cup grasping thin sheet parts", "gripper stabilizing irregular parts", "multi-point adsorption of large volume objects"). Based on the robot's load capacity, kinematic constraints, and path priority, it dynamically allocates the most suitable robot to perform the task, and finally generates high-precision motion control instructions that include path planning, speed curves, and end-effector force control parameters.

[0113] By implementing this control method, the system's adaptability to flexible production of multiple varieties and small batches is greatly improved, and manual programming intervention is reduced. Intelligent scheduling avoids robot conflicts and resource idleness, optimizes the overall throughput efficiency of the production line, and upgrades the system from fixed process execution to intelligent task orchestration.

[0114] According to the above embodiments of the present invention, the industrial production line control method further includes: after determining that the robot's feeding has failed, dynamically correcting the robot's movement path to obtain a corrected path; controlling the robot to move according to the corrected path to grab the material, and performing the feeding task on the material.

[0115] In this embodiment, the real-time speed of the conveyor belt is updated at a frequency of 100Hz, and the remaining time is recalculated. The target position is corrected, and the robot confirms the grasp by using the end effector force sensor after arriving at the target position, thus forming a closed-loop compensation.

[0116] In this method, after the robot completes the grasping action, if the unloading failure is confirmed by the end force sensor or visual feedback (such as failure to clamp, grasping off-center, or material slippage), the system immediately triggers the motion correction mechanism. Based on the model predictive control (MPC) algorithm, combined with the robot's current pose, target position deviation, environmental constraints, and dynamic conveyor belt status, the optimal trajectory is recalculated, and a corrected path including position compensation, speed adjustment, and attitude fine-tuning is generated and sent to the robot controller for execution in real time. This process is completed in a closed loop within milliseconds without interrupting the production line.

[0117] By implementing this control method, the system's self-healing ability to abnormal operating conditions is significantly improved, and the material leakage rate and downtime are greatly reduced. At the same time, by continuously accumulating failure samples, the cloud classification and scheduling models are fed back, which promotes the system to change from passive response to proactive prediction, greatly enhancing the stability and intelligence level of the production line.

[0118] According to the above embodiments of the present invention, the industrial assembly line control method further includes: using a simulation platform, and based on the assembly line characteristic information of the industrial assembly line and the robot characteristic information of the robot, performing simulation processing on the industrial assembly line and the robot to obtain a material unloading simulation environment for the robot; controlling the robot model corresponding to the robot to perform efficiency tasks on the material in the material unloading simulation environment to obtain material unloading simulation results; and generating unloading prompt information based on the material unloading simulation results.

[0119] In this embodiment, Figure 10 This is a schematic diagram of the simulation platform structure according to an embodiment of the present invention, as shown below. Figure 10 As shown, the simulation platform automatically scans the equipment ID when the production line starts up, matches it with a pre-set digital twin template library to generate a 1:1 virtual model; it drives the virtual model's physics engine by synchronizing the dynamic parameters (conveyor speed, material position, industrial robot status, etc.) collected by the LiDAR-encoder in real time through ModbusTCP; users can interactively adjust the robot path and grasp parameters through VR devices, and the system generates simulation results and provides optimization suggestions in real time.

[0120] Specifically, the above simulation operations can be performed on the simulation platform 313 via, for example... Figure 3 The data processing and simulation module 311 shown is used to complete this task. In this method, the simulation platform is based on digital twin technology, which automatically identifies the physical layout of the industrial production line (such as conveyor belt length, LiDAR and robot mounting positions) and robot model parameters (such as degrees of freedom, working radius, and end-effector load). It calls a pre-set 1:1 high-fidelity virtual model library to construct a virtual production line environment that is completely consistent with the physical entity. It also uses ModbusTCP to synchronize the dynamic data of the real equipment in real time (such as conveyor belt speed, material position, and robot joint status) to drive the virtual robot model to perform material unloading tasks synchronized with the physical entity. The simulation system simulates grasping dynamics, collision detection, and trajectory errors in the virtual environment, outputs simulation results such as material unloading success rate, path time, and energy consumption, and generates visualized material unloading prompts by combining rule engine or AI analysis.

[0121] By implementing this control method, the on-site commissioning cycle and equipment wear and tear are significantly reduced. Engineers can interactively optimize parameters through VR / PC terminals and discover potential conflicts and bottlenecks in advance. At the same time, simulation results can be used to train cloud-based decision-making models, forming a closed-loop evolutionary system of "simulation optimization - physical deployment - data feedback - model iteration", which significantly improves production line deployment efficiency, process adaptability and system reliability.

[0122] Figure 11 This is an optional digital twin intelligent control system operation flowchart according to an embodiment of the present invention, such as... Figure 11As shown, after system startup, hardware timestamp synchronization is performed, followed by simultaneous radar point cloud processing and material offset compensation calculation. Radar point cloud processing: The through-beam sensor detects the material and sends the detection signal to the lidar. The lidar collects data and uploads it to the cloud-based point cloud data processing algorithm to acquire information such as the material's position (in the lidar coordinate system), shape, orientation, and size. This information is then processed by intelligent classification and coordinate transformation algorithms to obtain the robot's moving target point position information (position before offset compensation). Material offset compensation calculation: The encoder acquires pulse signals to calculate the conveyor belt speed and sends this speed to the motion compensation algorithm to calculate the material coordinate offset caused by the conveyor belt movement. The coordinate transformation algorithm then obtains the precise position information of the robot's moving target point (position after offset compensation, in the robot's coordinate system). Finally, the task scheduling algorithm controls the robot to move and grasp the material, providing real-time feedback on the robot's movement path to determine if motion correction is needed. It also determines if material unloading was successful. If motion correction is needed, the robot moves according to the corrected data after the motion correction algorithm. If material unloading is successful, the robot system operation is complete. By using two parallel branches, "radar point cloud processing" and "material offset compensation", asynchronous acquisition, synchronous processing and fusion decision-making of sensing data and motion state information are realized, solving the problems of latency accumulation and response lag caused by traditional serial processing.

[0123] It should be noted that the entire process of system operation is accompanied by real-time pipeline simulation of cloud data storage and simulation platform.

[0124] As described above, the technical solutions provided by the embodiments of this invention construct a closed-loop architecture of "perception-decision-execution-feedback" across the entire chain. It pioneers a high-precision timestamp alignment mechanism between the LiDAR and encoder, combined with a dynamic motion compensation algorithm, to correct positioning offsets caused by conveyor belt movement in real time, achieving millimeter-level dynamic material spatial positioning. A rigid body transformation model between the LiDAR and robot coordinate systems is established, and precise extrinsic parameter matrices are obtained through calibration, enabling high-precision mapping of cross-coordinate system point cloud data and providing a unified spatial reference for robot grasping. An intelligent classification model based on shape, size, and posture features is deployed in the cloud to achieve automatic identification and adaptive task allocation for irregularly shaped and multi-sized materials. A "strongly adaptable simulation platform" is constructed, achieving high-fidelity, real-time interactive debugging and optimization of the production line through a 1:1 virtual model library and dynamic data mapping. A closed-loop data chain is established, transmitting execution feedback data back to the cloud for retraining the classification model, enabling system self-evolution and full-process traceability of production.

[0125] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0127] Example 2

[0128] According to embodiments of the present invention, an industrial assembly line control device for implementing the above-described industrial assembly line control method is also provided. Figure 12 This is a schematic diagram of an industrial production line control device according to an embodiment of the present invention, such as... Figure 12 As shown, the device includes: an acquisition unit 121, a feature extraction unit 123, a compensation unit 125, an instruction generation unit 127, and a first control unit 129. The device will now be described in detail.

[0129] The acquisition unit 121 is used to acquire three-dimensional point cloud data of materials when the arrival of materials on the industrial production line is detected, and to generate a conveyor belt operation model based on the instantaneous speed of the conveyor belt on the industrial production line.

[0130] The feature extraction unit 123 is used to extract features from the three-dimensional point cloud data to obtain the classification label of the material.

[0131] The compensation unit 125 is used to perform position compensation on the material using the conveyor belt running model to obtain the compensated position information of the material.

[0132] The instruction generation unit 127 is used to generate motion control instructions based on the classification label, the compensated position information, and the current working status of the robot corresponding to the industrial production line.

[0133] The first control unit 129 is used to control the robot to perform grasping actions on materials according to motion control instructions in order to perform the unloading task.

[0134] It should be noted that the above-mentioned acquisition unit 121, feature extraction unit 123, compensation unit 125, instruction generation unit 127 and first control unit 129 correspond to steps S202 to S210 in the above embodiments. The five units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0135] As can be seen from the above, in the solution described in the above embodiments of the present invention, a data acquisition unit is used to acquire three-dimensional point cloud data of the material when the arrival of the material on the industrial production line is detected, and to generate a conveyor belt operation model based on the instantaneous speed of the conveyor belt on the industrial production line; a feature extraction unit is used to extract features from the three-dimensional point cloud data to obtain the classification label of the material; a compensation unit is used to compensate the position of the material using the conveyor belt operation model to obtain the compensated position information of the material; an instruction generation unit is used to generate motion control instructions based on the classification label, the compensated position information, and the current working state of the robot corresponding to the industrial production line; and a first control unit is used to control the robot to perform a grasping action on the material according to the motion control instructions to perform the unloading task. The above solution achieves the goals of multimodal collaborative sensing with lidar and encoder, cross-coordinate system rigid body transformation positioning, cloud-based intelligent classification, and full-link digital twin closed-loop control of "perception-decision-execution-feedback". It realizes the technical effects of high-precision positioning of dynamic materials at the millimeter level, automatic intelligent sorting of irregularly shaped and multi-sized materials, efficient simulation debugging of production lines and system self-optimization, and significantly improves the success rate of grasping, production efficiency and flexible manufacturing capabilities.

[0136] Therefore, the technical solution provided by the above embodiments of the present invention solves the technical problem in the related art that the low accuracy of dynamic material sensing and poor coordination between components in industrial production line control systems lead to inaccurate positioning and low reliability.

[0137] Optionally, the industrial production line control device further includes: a calibration unit, used to calibrate the clocks of various components on the industrial production line by using the clock of the robot controller on the industrial production line as the master clock after the industrial production line is started, wherein the components include at least one of the following: a data acquisition component, a sensing component, and a control component.

[0138] Optionally, the acquisition unit includes: a detection module, used to detect the position of materials through a through-beam sensor on the industrial production line. When the through-beam sensor detects that the materials have entered the scanning area of ​​the lidar on the industrial production line, it triggers the lidar to start, so as to perform data acquisition operations on the materials and obtain three-dimensional point cloud data.

[0139] Optionally, the feature extraction unit includes: an upload module for uploading 3D point cloud data to a point cloud data processing module in the cloud, so as to use the point cloud data processing module to extract features from the 3D point cloud data to obtain feature data of the material; a first acquisition module for obtaining the material type of the material based on the feature data; and a second acquisition module for obtaining a classification label according to the material type.

[0140] Optionally, the acquisition unit includes: an acquisition module for acquiring rotational pulse signals of the drive motor of the conveyor belt through an encoder on an industrial production line; a calculation module for calculating the instantaneous speed of the conveyor belt based on the rotational pulse signals; and a first generation module for generating a conveyor belt operation model based on the characteristic information of the conveyor belt and the instantaneous speed of the conveyor belt.

[0141] Optionally, the compensation unit includes: a third acquisition module for acquiring the initial position of the material in the lidar coordinate system and the point cloud detection timestamp of the three-dimensional point cloud data; a processing module for processing the conveyor belt operation model within the time interval to obtain the total displacement and displacement components of the material in the direction of movement of the conveyor belt; and a superposition module for superimposing the displacement components onto the initial position to obtain the compensated position information.

[0142] Optionally, the industrial production line control device further includes: a calibration unit, used to calculate the spatial position information of the material in the robot base coordinate system by using the compensated position information and the calibration rigid body transformation matrix between the lidar and the robot; and an expansion unit, used to expand the three-dimensional point cloud data coordinates in the lidar coordinate system into homogeneous coordinates.

[0143] Optionally, the instruction generation unit includes: a matching module, used to determine the material grasping strategy by matching a preset grasping strategy library with the spatial position information of the robot base coordinate system based on the classification label and the compensated position information through a task scheduling algorithm; a selection module, used to select the target robot according to the current operation status; and a second generation module, used to generate motion control instructions based on the material grasping strategy and the feature information of the target robot.

[0144] Optionally, the industrial production line control device further includes: a correction unit, used to dynamically correct the robot's movement path after determining that the robot's unloading has failed, so as to obtain a corrected path; and a second control unit, used to control the robot to move according to the corrected path to grab materials and perform unloading tasks on the materials.

[0145] Optionally, the industrial assembly line control device further includes: a simulation unit, used to perform simulation processing on the industrial assembly line and the robot through a simulation platform, based on the assembly line characteristic information of the industrial assembly line and the robot characteristic information of the robot, to obtain the robot's unloading simulation environment; a third control unit, used to control the robot model corresponding to the robot to perform efficiency tasks on materials in the unloading simulation environment, so as to obtain the material unloading simulation results; and a third generation module, used to generate unloading prompt information based on the material unloading simulation results.

[0146] According to one aspect of the present invention, an industrial assembly line control system is provided, which uses any of the above-described industrial assembly line control methods.

[0147] According to one aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the industrial production line control method described above.

[0148] According to one aspect of the present invention, a processor is provided for running a program, wherein the program executes any of the above-described industrial production line control methods during runtime.

[0149] According to one aspect of the present invention, a computer program product is provided, including computer instructions, which, when executed by a processor, perform any of the above-described industrial production line control methods.

[0150] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any communication device in a group of communication devices.

[0151] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: when material is detected arriving on the industrial production line, three-dimensional point cloud data of the material is collected, and a conveyor belt operation model is generated based on the instantaneous speed of the conveyor belt on the industrial production line; features are extracted from the three-dimensional point cloud data to obtain a classification label for the material; the position of the material is compensated using the conveyor belt operation model to obtain the compensated position information of the material; motion control instructions are generated based on the classification label, the compensated position information, and the current working state of the robot corresponding to the industrial production line; the robot is controlled to perform a grasping action on the material according to the motion control instructions to perform the unloading task.

[0152] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: after the industrial production line is started, calibrating the clocks of various components on the industrial production line with the clock of the robot controller on the industrial production line as the master clock, wherein the components include at least one of the following: a data acquisition component, a sensing component, and a control component.

[0153] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: detecting the position of materials using a through-beam sensor on an industrial production line; when the through-beam sensor detects that the materials have entered the scanning area of ​​a lidar on the industrial production line, triggering the lidar to start, thereby performing data acquisition operations on the materials to obtain three-dimensional point cloud data.

[0154] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: uploading three-dimensional point cloud data to a point cloud data processing module in the cloud, using the point cloud data processing module to extract features from the three-dimensional point cloud data to obtain feature data of the material; obtaining the material type of the material based on the feature data; and obtaining a classification label according to the material type.

[0155] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring rotational pulse signals of the drive motor of the conveyor belt through an encoder on an industrial production line; calculating the instantaneous speed of the conveyor belt based on the rotational pulse signals; and generating a conveyor belt operation model based on the characteristic information of the conveyor belt and the instantaneous speed of the conveyor belt.

[0156] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining the initial position of the material in the lidar coordinate system and the point cloud detection timestamp of the three-dimensional point cloud data; processing the conveyor belt running model within the time interval to obtain the total displacement and displacement components of the material in the direction of movement of the conveyor belt; and superimposing the displacement components onto the initial position to obtain the compensated position information.

[0157] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: based on the compensated position information and the calibration rigid body transformation matrix between the lidar and the robot, the coordinates of the three-dimensional point cloud data in the lidar coordinate system are expanded to homogeneous coordinates to obtain the spatial position information of the material in the robot base coordinate system.

[0158] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining a material grasping strategy by matching a preset grasping strategy library with the spatial position information of the robot base coordinate system based on the classification label and the compensated position information using a task scheduling algorithm; selecting a target robot based on the current working state; and generating motion control instructions based on the material grasping strategy and the feature information of the target robot.

[0159] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: after determining that the robot's feeding has failed, dynamically correcting the robot's movement path to obtain a corrected path; controlling the robot to move along the corrected path to grab the material, and performing the feeding task on the material.

[0160] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: using a simulation platform, and based on the production line characteristics of the industrial assembly line and the robot's characteristics, to perform simulation processing on the industrial assembly line and the robot to obtain a robot unloading simulation environment; controlling the robot model corresponding to the robot in the unloading simulation environment to perform efficiency tasks on the material to obtain material unloading simulation results; and generating unloading prompt information based on the material unloading simulation results.

[0161] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0166] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0167] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An industrial pipeline control method, characterized by, include: When material is detected arriving on the industrial production line, three-dimensional point cloud data of the material is collected, and a conveyor belt operation model is generated based on the instantaneous speed of the conveyor belt on the industrial production line. Feature extraction is performed on the three-dimensional point cloud data to obtain the classification label of the material; The material is positionally compensated using the conveyor belt operation model to obtain the compensated position information of the material. Motion control commands are generated based on the classification labels, the compensated position information, and the current operating status of the robot corresponding to the industrial production line. The robot is controlled to perform a grasping action on the material according to the motion control command in order to perform the unloading task.

2. The industrial flowline control method of claim 1, wherein, The industrial assembly line control method also includes: After the industrial production line is started, the clocks of each component on the industrial production line are calibrated using the clock of the robot controller on the industrial production line as the master clock. The components include at least one of the following: a data acquisition component, a sensing component, and a control component.

3. The industrial flowline control method of claim 1, wherein, When material is detected arriving on the industrial production line, the three-dimensional point cloud data of the material is collected, including: The material is located by a through-beam sensor on the industrial production line. When the through-beam sensor detects that the material has entered the scanning area of ​​the lidar on the industrial production line, the lidar is triggered to start and perform data acquisition on the material to obtain the three-dimensional point cloud data.

4. The industrial flowline control method of claim 1, wherein, Feature extraction is performed on the three-dimensional point cloud data to obtain the classification labels of the materials, including: The three-dimensional point cloud data is uploaded to the point cloud data processing module in the cloud, so as to extract features from the three-dimensional point cloud data to obtain the feature data of the material. The material type of the material is obtained based on the feature data; The classification label is obtained based on the material type.

5. The industrial flowline control method of claim 1, wherein, Based on the instantaneous speed of the conveyor belt in the industrial production line, a conveyor belt operation model is generated, including: The encoder on the industrial production line collects the rotation pulse signal of the drive motor of the conveyor belt. The instantaneous speed of the conveyor belt is calculated based on the rotating pulse signal; The conveyor belt operation model is generated based on the characteristic information of the conveyor belt and the instantaneous speed of the conveyor belt.

6. The industrial flowline control method of claim 1, wherein, The material is positionally compensated using the conveyor belt operation model to obtain the compensated position information of the material, including: Obtain the preliminary position of the material in the lidar coordinate system and the point cloud detection timestamp of the three-dimensional point cloud data; The conveyor belt operation model is processed within a time interval to obtain the total displacement and displacement components of the material in the direction of movement of the conveyor belt. The displacement components are superimposed on the initial position to obtain the compensated position information.

7. The industrial flowline control method of claim 1, wherein, The industrial assembly line control method also includes: Based on the compensated position information and the calibration rigid body transformation matrix between the lidar and the robot; The coordinates of the three-dimensional point cloud data in the lidar coordinate system are expanded into homogeneous coordinates to obtain the spatial position information of the material in the robot base coordinate system.

8. The industrial flowline control method according to any one of claims 1 to 7, characterized in that, Motion control commands are generated based on the classification labels, the compensated position information, and the current operating state of the robot corresponding to the industrial production line, including: The material grasping strategy is determined by matching the spatial position information of the classification label and the compensated position information in the robot base coordinate system with a preset grasping strategy library through a task scheduling algorithm. Select the target robot based on the current operation status; The motion control commands are generated based on the material grasping strategy and the feature information of the target robot.

9. The industrial flowline control method according to any one of claims 1 to 7, wherein, The industrial assembly line control method also includes: After determining that the robot's unloading has failed, the robot's movement path is dynamically corrected to obtain a corrected path; The robot is controlled to move along the corrected path to grab the material and perform the unloading task on the material.

10. The industrial flowline control method according to any one of claims 1 to 7, wherein, The industrial assembly line control method also includes: Using a simulation platform, and based on the production line characteristics of the industrial assembly line and the robot characteristics of the robot, the industrial assembly line and the robot are simulated to obtain the robot's unloading simulation environment. In the material feeding simulation environment, the robot model corresponding to the robot is controlled to perform efficiency tasks on the material to obtain the material feeding simulation results; Based on the material feeding simulation results, a feeding prompt message is generated.

11. An industrial pipeline control device, characterized by include: The acquisition unit is used to acquire three-dimensional point cloud data of the material when it is detected that the material has arrived on the industrial production line, and to generate a conveyor belt operation model based on the instantaneous speed of the conveyor belt of the industrial production line. The feature extraction unit is used to extract features from the three-dimensional point cloud data to obtain the classification label of the material; The compensation unit is used to perform position compensation on the material using the conveyor belt running model to obtain the compensated position information of the material. The instruction generation unit is used to generate motion control instructions based on the classification label, the compensated position information, and the current working state of the robot corresponding to the industrial production line. The first control unit is used to control the robot to perform a grasping action on the material according to the motion control command, so as to perform the unloading task.

12. An industrial pipeline control system characterized by, The industrial production line control system uses the industrial production line control method according to any one of claims 1 to 10.

Images