Flexible production line computer control system based on visual machine learning

Abstract

The application discloses a flexible production line computer control system based on visual machine learning, which comprises a visual perception layer, a feature extraction and learning layer, a decision and planning layer and a computer control execution layer. The visual perception layer captures images or point cloud data of workpieces and environment in real time through a multi-station image acquisition unit and a 3D visual sensor. The feature extraction and learning layer extracts multi-dimensional state features of the workpieces, including contour, pose and offset, by using a pre-trained deep learning model, and can continuously update the recognition mapping relationship based on actual feedback data of the production line through incremental learning or small sample transfer learning. The decision and planning layer combines feature information, order scheduling and process requirements. Through deep integration of visual perception, continuous learning and adaptive control, the application significantly improves the changeover efficiency, positioning accuracy and running stability of the flexible production line in the multi-variety mixed flow production scene.

Description

Technical Field

[0001] This invention relates to the field of industrial automation and intelligent manufacturing technology, and in particular to a flexible production line computer control system based on visual machine learning. Background Technology

[0002] As consumer demand becomes increasingly personalized and product lifecycles shorten, modern manufacturing is shifting from large-scale, rigid production to a flexible production model characterized by diverse varieties and small batches. Traditional production line control systems typically rely on fixed tooling fixtures and preset teaching trajectories. When product models are changed, manual intervention is required to complete the changeover adjustments, which is not only time-consuming and labor-intensive but also difficult to cope with random disturbances during production line operation—such as workpiece arrival position deviations, pallet positioning errors, and spatial interference between adjacent workstations. In recent years, vision-guided technology has been widely used in the field of industrial robots. However, most existing solutions treat the vision system as an independent "eye," only performing simple workpiece positioning or quality inspection functions. The output lacks deep information fusion with subsequent control processes such as robot path planning, multi-machine collaborative scheduling, and adaptive adjustment of process parameters. More importantly, existing systems generally adopt an "offline training, online fixed use" model. Once deployed, deep learning models are no longer updated, failing to utilize incremental data accumulated during actual production line operation to continuously optimize recognition accuracy and generalization ability. Existing computer control systems either employ purely rule-driven logic or rely excessively on end-to-end black-box policy networks. Therefore, there is an urgent need for a high-precision method for calculating the theoretical line loss of transformer substations that is compatible with bidirectional power flow. Summary of the Invention

[0003] One object of this invention is to propose a flexible production line computer control system based on visual machine learning, comprising the following steps: Visual perception layer: The visual perception layer is deployed at key workstations on the production line, including but not limited to loading stations, assembly stations, inspection stations, and unloading stations. Each workstation is equipped with at least one high-resolution industrial camera. For scenarios that require depth information, a 3D structured light camera or laser contour scanner is additionally deployed. This layer also integrates a programmable light source controller in hardware, which can automatically adjust the lighting parameters according to the material and surface reflection characteristics of the current workpiece to suppress the degradation of image quality caused by ambient light interference and workpiece reflection. It should be noted that, in a preferred embodiment of the present invention, the visual perception layer also integrates a force sensor and a proximity sensor at the same workstation to form a multimodal perception unit. The force sensor is installed between the robot's end flange and the gripping tool to sense the magnitude and direction of the contact force during workpiece picking and assembly insertion. The proximity sensor is used to provide millimeter-level distance feedback when the end effector approaches the workpiece surface. The data from these multi-source heterogeneous sensors are uploaded to the feature extraction and learning layer after being timestamped, providing richer environmental interaction information for subsequent compliant control.

[0004] Feature extraction and learning layer: This layer is equipped with a pre-trained deep learning model, which adopts a heterogeneous architecture of convolutional neural network and point cloud processing network. Two-dimensional image data is fed into ResNet or EfficientNet series backbone networks to extract texture and edge features, while three-dimensional point cloud data is fed into PointNet++ or a sparse convolution-based three-dimensional feature extraction network to obtain geometric structure features. The two types of features are aligned and aggregated through a cross-modal attention module in the mid-term fusion stage, and finally output a unified state representation of the current workpiece.

[0005] This state representation includes at least three aspects of information: first, the workpiece's category label and confidence score; second, the workpiece's 6D pose; and third, the contour integrity or surface defect index of key parts of the workpiece. This information is not output once and then stops, but is continuously updated at a set frequency (usually 10Hz to 30Hz), forming a dynamic tracking of the entire process of the workpiece from "entering the field of view" to "completing the operation".

[0006] More importantly, this deep learning model is not fixed, but is configured to perform incremental learning or few-shot transfer learning based on feedback data generated during actual production line operation. Specifically, when the system detects that the confidence level of the current workpiece recognition is lower than the set threshold, or when the success rate of grasping feedback from the subsequent execution layer decreases significantly, the feature extraction and learning layer automatically triggers the model update process: using recently accumulated labeled image data as training samples, and employing elastic weight consolidation or meta-learning strategies to quickly fine-tune the model, so that the model can adapt to new workpiece types or changes in production line lighting without forgetting existing knowledge.

[0007] In a preferred embodiment, the deep learning model at this layer also incorporates a feature focusing submodule based on a self-attention mechanism. This submodule calculates the importance weights of each spatial location in the input image or point cloud, automatically focusing computational resources on salient feature regions of the workpiece—such as chamfered edges, hole locations, or abrupt texture changes. This design is particularly effective when dealing with "weak feature workpieces" (such as smooth cylinders or textureless black plastic parts), compensating for insufficient texture information by emphasizing subtle variations in geometric contours.

[0008] Decision-making and planning level: The decision and planning layer receives workpiece state features from the feature extraction and learning layer, and integrates them with order scheduling information issued by the production line manufacturing execution system, the health status of each device, and the currently available process parameter library to ultimately generate an executable flexible control strategy.

[0009] This invention employs a hybrid architecture at the decision-making level: the rule-based reasoning part is responsible for handling the hard constraints of the production line—including equipment kinematic limits, safety interlock logic, and mandatory process sequence requirements. These constraints are solidified into "guardrails" of the decision-making framework in the form of production rules. The data-driven optimization part is responsible for seeking the optimal solution within the range allowed by the guardrails. Specifically, it uses an optimizer based on Monte Carlo tree search or population evolution algorithm to perform multi-objective optimization with the shortest cycle time, lowest energy consumption, or minimum force control fluctuation as the objective function.

[0010] The advantage of this hybrid architecture lies in its ability to retain the interpretability and security necessary for industrial scenarios while also possessing the adaptability to complex and unstructured chemical conditions (the data-driven part can cover edge cases that are difficult to exhaustively cover with manual rules). The generated flexible control strategy is specifically manifested as a set of structured instructions, which includes, but is not limited to: adjusting parameters for the gripping point position and end-effector posture of the current workpiece, selecting the assembly sequence and interleaving timing of each part during the assembly process, and dividing tasks and coordinating avoidance paths among multiple robots.

[0011] To further enhance the reliability of decision-making, this invention integrates a digital twin model into the decision-making and planning layer. This digital twin model maintains real-time mirror synchronization with the physical production line, performing virtual simulation within the digital twin environment before decision commands are issued to the execution layer. If anomalies such as collisions, overtravel, or cycle timeouts are detected during simulation, the decision-making and planning layer automatically corrects the control strategy and re-simulates and verifies until all safety checks are passed. Simultaneously, simulation deviation data—such as the difference between the virtual workpiece pose and the perceived pose, and the degree of deviation between the simulated force control curve and the historical real curve—is recorded and fed back to the feature extraction and learning layer as a supplementary training source for incremental learning. This is equivalent to transforming the digital twin into a "virtual teacher," continuously providing physically consistent supervisory signals to the machine learning model.

[0012] Computer control execution layer: The computer control execution layer serves as the interface bridge between the system and physical devices. This layer includes industrial-grade motion controllers, programmable logic controller communication modules, and drive interfaces for various actuators (including servo drives, pneumatic solenoid valves, robot controllers, etc.). Its core function is to parse the flexible control strategies issued by the decision-making and planning layer into position commands, speed commands, and force / torque commands that each actuator can recognize, and send them to each execution node according to a precise time sequence.

[0013] It is important to emphasize that the execution layer of this invention is not a simple instruction repeater; it is internally configured with an independent feedforward-feedback composite control module. The feedforward control section pre-calculates the compensation amount to the driver based on the "expected position-time" trajectory in the decision instruction, enabling the actuator to begin acceleration or deceleration before reaching the target position, effectively shortening response lag. The feedback control section performs millisecond-level dynamic fine-tuning based on the workpiece position deviation information transmitted in real-time from the visual perception layer, overcoming execution errors caused by mechanical backlash, load variations, and external disturbances. The feedforward and feedback control quantities are dynamically weighted and fused in the collaborative decision module—the weighting coefficients are automatically adjusted based on the error change rate and cumulative error under the current operating conditions—ultimately generating a single execution instruction. This composite control structure enables the system to possess both rapid response capability and high steady-state accuracy, making it particularly suitable for scenarios with stringent trajectory accuracy requirements, such as high-speed pick-and-place and precision assembly.

[0014] At the hardware level, the computer control execution layer also has reserved expansion interfaces for connecting other auxiliary equipment on the production line—such as vibrating feeders, conveyor belt frequency converters, pneumatic clamp pressure regulating valves, etc.—so that the system can coordinate all controllable resources on the production line in a unified manner.

[0015] Other preferred configurations: In some application scenarios, production lines need to switch to entirely new workpiece types within a very short time. In such cases, there is neither sufficient time to collect a large amount of training data nor immediate manual teaching. To address this, this invention also includes a human-machine interface. Operators input basic parameters of the new workpiece type (such as size range, material, and expected assembly clearance) or directly drag and drop a standard action for teaching. The system then performs few-sample transfer learning based on this teaching data—utilizing the general feature extraction capabilities of the pre-trained model—requiring only a few gradient updates to quickly adapt to the new workpiece, thereby reducing changeover time from hours to minutes.

[0016] Furthermore, in the multi-robot collaborative solution generated by the decision-making and planning layer, the obstacle avoidance path planning is not fixed after a single calculation, but is dynamically adjusted based on the real-time perception of the production line space occupancy status by the vision perception layer (including the positions of other workpieces on the workbench, the current posture of other robots, and the accumulation status of materials on the conveyor belt). When a potential collision risk is detected, the system uses a speed-damping modulation strategy to replan the path online, avoiding sudden stops or collisions caused by unexpected situations.

[0017] The beneficial effects of this invention are: By deeply coupling visual perception with the incremental update mechanism of machine learning models, the visual capabilities of the production line are continuously enhanced as the running time increases. Small sample transfer learning supports rapid changeover and significantly improves the production line's response efficiency to multi-variety orders. The decision-making level adopts a rule-data hybrid driven architecture, which not only preserves the necessary safety boundaries and decision interpretability in industrial production, but also covers complex working conditions that are difficult to exhaustively cover through data-driven optimization. Digital twin models are not only used for policy pre-validation, but also actively feed simulation biases back to the learning layer, forming a "physical-virtual" closed loop, which effectively alleviates the problem of insufficient real labeled data and improves the model's generalization ability and physical consistency. The feedforward-feedback composite control structure of the execution layer, combined with real-time visual deviation feedback, enables rapid suppression and high-precision tracking of random disturbances in flexible production lines, reducing scrap rate while ensuring production efficiency. Multimodal sensing provides a reliable information foundation for force-sensitive processes such as compliant insertion and thread tightening in precision assembly. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a flexible production line computer control system based on visual machine learning proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] This embodiment provides a flexible production line computer control system based on visual machine learning, and the specific steps are as follows: Visual perception layer: The visual perception layer is deployed at key workstations on the production line, including but not limited to loading stations, assembly stations, inspection stations, and unloading stations. Each workstation is equipped with at least one high-resolution industrial camera. For scenarios that require depth information, a 3D structured light camera or laser contour scanner is additionally deployed. This layer also integrates a programmable light source controller in hardware, which can automatically adjust the lighting parameters according to the material and surface reflection characteristics of the current workpiece to suppress the degradation of image quality caused by ambient light interference and workpiece reflection. It should be noted that, in a preferred embodiment of the present invention, the visual perception layer also integrates a force sensor and a proximity sensor at the same workstation to form a multimodal perception unit. The force sensor is installed between the robot's end flange and the gripping tool to sense the magnitude and direction of the contact force during workpiece picking and assembly insertion. The proximity sensor is used to provide millimeter-level distance feedback when the end effector approaches the workpiece surface. The data from these multi-source heterogeneous sensors are uploaded to the feature extraction and learning layer after being timestamped, providing richer environmental interaction information for subsequent compliant control.

[0021] Feature extraction and learning layer: This layer is equipped with a pre-trained deep learning model, which adopts a heterogeneous architecture of convolutional neural network and point cloud processing network. Two-dimensional image data is fed into ResNet or EfficientNet series backbone networks to extract texture and edge features, while three-dimensional point cloud data is fed into PointNet++ or a sparse convolution-based three-dimensional feature extraction network to obtain geometric structure features. The two types of features are aligned and aggregated through a cross-modal attention module in the mid-term fusion stage, and finally output a unified state representation of the current workpiece.

[0022] This state representation includes at least three aspects of information: first, the workpiece's category label and confidence score; second, the workpiece's 6D pose; and third, the contour integrity or surface defect index of key parts of the workpiece. This information is not output once and then stops, but is continuously updated at a set frequency (usually 10Hz to 30Hz), forming a dynamic tracking of the entire process of the workpiece from "entering the field of view" to "completing the operation".

[0023] More importantly, this deep learning model is not fixed, but is configured to perform incremental learning or few-shot transfer learning based on feedback data generated during actual production line operation. Specifically, when the system detects that the confidence level of the current workpiece recognition is lower than the set threshold, or when the success rate of grasping feedback from the subsequent execution layer decreases significantly, the feature extraction and learning layer automatically triggers the model update process: using recently accumulated labeled image data as training samples, and employing elastic weight consolidation or meta-learning strategies to quickly fine-tune the model, so that the model can adapt to new workpiece types or changes in production line lighting without forgetting existing knowledge.

[0024] In a preferred embodiment, the deep learning model at this layer also incorporates a feature focusing submodule based on a self-attention mechanism. This submodule calculates the importance weights of each spatial location in the input image or point cloud, automatically focusing computational resources on salient feature regions of the workpiece—such as chamfered edges, hole locations, or abrupt texture changes. This design is particularly effective when dealing with "weak feature workpieces" (such as smooth cylinders or textureless black plastic parts), compensating for insufficient texture information by emphasizing subtle variations in geometric contours.

[0025] Decision-making and planning level: The decision and planning layer receives workpiece state features from the feature extraction and learning layer, and integrates them with order scheduling information issued by the production line manufacturing execution system, the health status of each device, and the currently available process parameter library to ultimately generate an executable flexible control strategy.

[0026] This invention employs a hybrid architecture at the decision-making level: the rule-based reasoning part is responsible for handling the hard constraints of the production line—including equipment kinematic limits, safety interlock logic, and mandatory process sequence requirements. These constraints are solidified into "guardrails" of the decision-making framework in the form of production rules. The data-driven optimization part is responsible for seeking the optimal solution within the range allowed by the guardrails. Specifically, it uses an optimizer based on Monte Carlo tree search or population evolution algorithm to perform multi-objective optimization with the shortest cycle time, lowest energy consumption, or minimum force control fluctuation as the objective function.

[0027] The advantage of this hybrid architecture lies in its ability to retain the interpretability and security necessary for industrial scenarios while also possessing the adaptability to complex and unstructured chemical conditions (the data-driven part can cover edge cases that are difficult to exhaustively cover with manual rules). The generated flexible control strategy is specifically manifested as a set of structured instructions, which includes, but is not limited to: adjusting parameters for the gripping point position and end-effector posture of the current workpiece, selecting the assembly sequence and interleaving timing of each part during the assembly process, and dividing tasks and coordinating avoidance paths among multiple robots.

[0028] To further enhance the reliability of decision-making, this invention integrates a digital twin model into the decision-making and planning layer. This digital twin model maintains real-time mirror synchronization with the physical production line, performing virtual simulation within the digital twin environment before decision commands are issued to the execution layer. If anomalies such as collisions, overtravel, or cycle timeouts are detected during simulation, the decision-making and planning layer automatically corrects the control strategy and re-simulates and verifies until all safety checks are passed. Simultaneously, simulation deviation data—such as the difference between the virtual workpiece pose and the perceived pose, and the degree of deviation between the simulated force control curve and the historical real curve—is recorded and fed back to the feature extraction and learning layer as a supplementary training source for incremental learning. This is equivalent to transforming the digital twin into a "virtual teacher," continuously providing physically consistent supervisory signals to the machine learning model.

[0029] Computer control execution layer: The computer control execution layer is the interface bridge between the system and physical devices. This layer includes industrial-grade motion controllers, programmable logic controller communication modules, and drive interfaces for various actuators. Its core function is to parse the flexible control strategy issued by the decision and planning layer into position commands, speed commands, and force / torque commands that each actuator can recognize, and send them to each execution node according to a precise time sequence.

[0030] It is important to emphasize that the execution layer of this invention is not a simple instruction repeater. It is internally configured with an independent feedforward-feedback composite control module. The feedforward control section calculates the compensation amount in advance based on the "expected position-time" trajectory in the decision instruction and outputs it to the driver, enabling the actuator to begin acceleration or deceleration before entering the target position, effectively shortening the response lag. The feedback control section performs millisecond-level dynamic fine-tuning based on the workpiece position deviation information transmitted in real time from the visual perception layer, overcoming execution errors caused by mechanical backlash, load variations, and external disturbances. The feedforward and feedback control quantities are dynamically weighted and fused in the collaborative decision module—the weighting coefficients are automatically adjusted based on the error change rate and cumulative error under the current operating conditions—ultimately generating a single execution instruction. This composite control structure enables the system to possess both rapid response capability and high steady-state accuracy, making it particularly suitable for scenarios with stringent trajectory accuracy requirements, such as high-speed pick-and-place and precision assembly.

[0031] At the hardware level, the computer control execution layer also has reserved expansion interfaces for connecting other auxiliary equipment on the production line—such as vibrating feeders, conveyor belt frequency converters, pneumatic clamp pressure regulating valves, etc.—so that the system can coordinate all controllable resources on the production line in a unified manner.

[0032] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A flexible production line computer control system based on visual machine learning, comprising a visual perception layer, a feature extraction and learning layer, a decision-making and planning layer, and a computer control execution layer, characterized in that, The visual perception layer is deployed at key workstations on the production line and includes multiple image acquisition units and at least one 3D vision sensor to capture raw image data or point cloud data containing information about the target workpiece and its surrounding environment in real time. The feature extraction and learning layer, which is communicatively connected to the visual perception layer, is configured with a pre-trained deep learning model for extracting multi-dimensional state features of the workpiece from the original image data or point cloud data. The multi-dimensional state features include at least workpiece contour information, surface quality features, pose parameters and their offset relative to the production line reference. Furthermore, the deep learning model is configured to perform incremental learning or few-shot transfer learning based on feedback data generated during actual production line operation to dynamically update the mapping relationship between workpiece recognition and state representation. It receives the multi-dimensional state features output by the feature extraction and learning layer, and combines the current production line's order information, equipment status, and process requirements to generate a flexible control strategy through a hybrid decision model that integrates rule reasoning and data-driven optimization algorithms. The flexible control strategy includes at least: gripping pose adjustment parameters for the current workpiece, assembly sequence selection instructions, and task allocation schemes for multi-robot collaborative operations. Its communication connection is to the decision and planning layer, which includes a motion controller and an I / O interface unit. It is used to parse the flexible control strategy into drive instructions for each actuator, and to adjust the production line conveying speed, robot motion trajectory and end effector action in real time, so as to realize closed-loop computer control of the flexible production line.

2. The computer control system for a flexible production line based on visual machine learning according to claim 1, characterized in that, The visual perception layer also includes a multimodal perception unit deployed at the same workstation. In addition to the image acquisition unit, the multimodal perception unit also includes a force sensor or a distance sensor, which is used to simultaneously acquire contact force information and proximity information during workpiece gripping or assembly. The feature extraction and learning layer is configured to perform spatiotemporal alignment and fusion processing on the multi-source heterogeneous data acquired by the multimodal perception unit to generate multimodal fusion features for guiding compliant operation.

3. The flexible production line computer control system based on visual machine learning according to claim 1, characterized in that, The attention mechanism is configured to automatically focus on local salient regions of the workpiece in point cloud data or image data, so as to achieve high-precision recognition and 6D pose estimation in scenarios where the workpiece cut features are weak or the inter-class differences are small.

4. The computer control system for a flexible production line based on visual machine learning according to claim 1, characterized in that, The decision-making and planning layer is equipped with a digital twin model, which is synchronized with the physical production line. It is used to perform virtual simulation verification of the flexible control strategy and to send the deviation data generated during the simulation back to the feature extraction and learning layer as training samples for the incremental learning.

5. A flexible production line computer control system based on visual machine learning according to claim 1, characterized in that, The computer control execution layer adopts a feedforward and feedback composite control architecture. The feedforward control part performs advance compensation on the actuator according to the pre-adjustment instructions issued by the decision and planning layer, and the feedback control part performs dynamic fine-tuning according to the workpiece position deviation information fed back by the visual perception layer in real time. The outputs of the feedforward control and feedback control are weighted and fused by the collaborative decision module to generate the final execution instruction.

6. The computer control system for a flexible production line based on visual machine learning according to claim 1, characterized in that, The feature extraction and learning layer is further configured to determine the type of the current workpiece based on the workpiece recognition result in the original image data, and retrieve the corresponding process parameter set from the preset process parameter library according to the type determination result. The generation of the flexible control strategy further depends on the process parameter set.

7. A computer control system for a flexible production line based on visual machine learning according to claim 1, characterized in that, The task allocation scheme for multi-robot collaborative operation generated by the decision and planning layer includes obstacle avoidance path planning for at least two robot actuators. The obstacle avoidance path planning is dynamically updated based on the production line space occupancy status perceived in real time by the visual perception layer.

8. A computer control system for a flexible production line based on visual machine learning according to claim 6, characterized in that, The system also includes a human-machine interface configured to receive external input production line changeover instructions or teaching data of new workpieces. The feature extraction and learning layer responds to the changeover instructions or teaching data by performing small-sample transfer learning to update the recognition and positioning capabilities of new workpieces.

Images