A machine vision production line efficiency evaluation and optimization management system

Abstract

The application relates to the technical field of digital management of industrial manufacturing, in particular to a machine vision production line efficiency evaluation and optimization management system, which comprises a data acquisition module, a high-speed industrial camera array, a vibration sensor and an RFID reader are triggered through a central synchronous controller, product image, equipment operation and material flow data are synchronously acquired, a data processing and fusion module extracts product quality features based on CNN, multi-modal data are fused through time sequence alignment normalization and an attention mechanism, a dynamic efficiency evaluation module calculates OEE, FPY and production line balance rate in real time through a deep neural network, an optimization strategy generation module outputs optimization instructions such as equipment parameter adjustment through a reinforcement learning intelligent agent, a control execution module converts the instructions into an industrial protocol format and issues the PLC, and the effect is verified to form a closed loop. The application improves data correlation and evaluation real-time performance, optimizes the dynamic adaptation of a production line and helps to improve efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital management of industrial manufacturing, in particular to a machine vision production line efficiency evaluation and optimization management system. BACKGROUND

[0002] The current production line management scheme involving machine vision has many limitations and cannot meet the efficient evaluation and optimization requirements. The specific problems are as follows:

[0003] Poor synchronization of machine vision data collection: Although some production lines have introduced high-speed industrial cameras and other machine vision equipment to collect product image data, a unified synchronization mechanism has not been established, resulting in inconsistent timestamps between the product image data obtained by machine vision and the device operation state data collected by vibration sensors and the material flow data collected by RFID readers. The corresponding relationship between "product visual features-device state-material flow" cannot be accurately associated, and the root cause of product quality problems (such as surface defects) cannot be located (such as device vibration abnormalities);

[0004] Machine vision is disconnected from efficiency evaluation and optimization: In existing systems, machine vision data is mainly used for single product quality inspection, and it is not deeply combined with production line efficiency evaluation (such as first pass yield FPY and overall equipment effectiveness OEE). Efficiency evaluation still relies on manual statistics, which is lagging and subjective. At the same time, optimization strategies are not based on real-time feedback from quality data, and device parameters and production rhythm cannot be dynamically adjusted according to changes in product visual features (such as rising defect rate), which means that the real-time advantage of machine vision has not been translated into production line optimization power.

[0005] Therefore, a machine vision production line efficiency evaluation and optimization management system is proposed to solve the above problems. SUMMARY

[0006] The purpose of the present application is to provide a machine vision production line efficiency evaluation and optimization management system to solve the problems raised in the background.

[0007] To achieve the above purpose, the present application provides the following technical solutions:

[0008] A machine vision production line efficiency evaluation and optimization management system, comprising:

[0009] A data collection module, including a high-speed industrial camera array, vibration sensors and RFID readers deployed on the production line, for synchronously collecting image data of products, operation state data of devices and flow data of materials;

[0010] A data processing and fusion module connected to the data collection module, comprising:

[0011] an image processing unit configured to extract features of the image data based on a deep learning convolutional neural network model to obtain a product quality feature vector;

[0012] a data normalization unit configured to perform time alignment and normalization processing on the running state data and the flow data to generate standardized equipment state data;

[0013] a multi-modal data fusion unit configured to fuse the product quality feature vector and the standardized equipment state data using a fusion algorithm based on an attention mechanism to generate a production line fusion feature set;

[0014] a dynamic performance evaluation module connected to the data processing and fusion module, configured to input the production line fusion feature set into a trained performance evaluation model to calculate, in real time, key performance indicators including at least an overall equipment effectiveness (OEE), a first pass yield (FPY), and a production line balancing rate;

[0015] an optimization strategy generation module connected to the dynamic performance evaluation module, which is internally provided with a reinforcement learning agent, the reinforcement learning agent taking the key performance indicators as state inputs of a reward function, and learning an output optimization action strategy by interacting with the environment, the optimization action strategy including device parameter adjustment instructions, production rhythm control instructions, or re-planning instructions for a feeding robot path;

[0016] a control execution module connected to the optimization strategy generation module and a programmable logic controller (PLC) on the production line, configured to convert the optimization action strategy into control instructions conforming to an industrial communication protocol, and issue the control instructions to corresponding execution units to form a closed-loop optimization control system.

[0017] As a preferred solution, in the data acquisition module, the high-speed industrial camera array, the vibration sensor, and the RFID reader perform data acquisition through a synchronous triggering mechanism, wherein the synchronous triggering mechanism is managed by a central synchronization controller, the central synchronization controller generates a synchronization pulse signal to simultaneously trigger the high-speed industrial camera array to capture multi-angle image data of the product, the vibration sensor to collect device vibration frequency and amplitude data, and the RFID reader to read material RFID tag data, ensuring that all collected data have consistent timestamps; the data acquisition module further packages the collected image data, running state data, and flow data into time series data packets, and transmits the time series data packets to the data processing and fusion module through an industrial Ethernet.

[0018] As a preferred solution, the image processing unit extracts features of the image data based on a deep learning convolutional neural network model to obtain a product quality feature vector, including:

[0019] preprocessing the collected image data, including image denoising, size adjustment, and pixel value normalization, to generate standardized image data;

[0020] The standardized image data is input into a pre-trained deep learning convolutional neural network model, and is processed by forward propagation through multiple convolutional layers, activation function layers and pooling layers to generate multi-level feature maps;

[0021] The multi-level feature maps are subjected to feature fusion processing, high-level semantic features are extracted through a fully connected layer or a global pooling layer in the convolutional neural network, and an initial feature vector is generated;

[0022] The initial feature vector is subjected to dimension reduction and standardization processing to obtain a final feature vector representing product quality.

[0023] As a preferred solution, the data standardization unit is configured to perform time series alignment and normalization processing on the running state data and the flow data to generate standardized equipment state data, including:

[0024] The running state data and the flow data are received from the data acquisition module, wherein the running state data includes device vibration frequency and amplitude data, and the flow data includes material RFID tag data;

[0025] The running state data and the flow data are subjected to timestamp analysis to extract original time information of each data point;

[0026] Based on the synchronization time reference provided by the central synchronization controller, the running state data and the flow data after timestamp analysis are subjected to time series alignment processing, and the data is resampled onto a unified time sequence through a linear interpolation method to ensure that all data points have consistent time intervals;

[0027] The running state data and the flow data after time series alignment are subjected to normalization processing, and a minimum-maximum normalization algorithm is used to map the data values to the range of [0, 1] to generate normalized data;

[0028] The normalized data is combined into standardized equipment state data and output to the multi-modal data fusion unit.

[0029] As a preferred solution, the multi-modal data fusion unit uses a fusion algorithm based on an attention mechanism to fuse the product quality feature vector and the standardized equipment state data to generate a production line fusion feature set, including:

[0030] The product quality feature vector from the image processing unit and the standardized equipment state data from the data standardization unit are received;

[0031] The product quality feature vector and the standardized equipment state data are respectively input into a linear mapping layer to generate corresponding query vectors, key vectors and value vectors;

[0032] A dot product similarity between the query vector and the key vector is calculated, and the similarity is normalized by a softmax operation to generate attention weights;

[0033] The value vectors are weighted and summed using the attention weights to generate a weighted context feature vector;

[0034] The weighted context feature vector is combined with the product quality feature vector to generate a preliminary fusion feature vector;

[0035] The preliminary fusion feature vector is processed by a fully connected layer and an activation function to generate a final production line fusion feature set.

[0036] As a preferred solution, the dynamic performance evaluation module is configured to input the production line fusion feature set into a trained performance evaluation model to calculate key performance indicators including at least overall equipment effectiveness OEE, first pass yield FPY, and line balancing rate, including:

[0037] Receiving the production line fusion feature set from the multi-modal data fusion unit;

[0038] The production line fusion feature set is divided into a time window to generate a fixed-length feature sequence;

[0039] The feature sequence is input into the trained performance evaluation model, which is based on a deep neural network architecture. The feature sequence is received by the input layer, and non-linear transformation and feature extraction are performed through multiple hidden layers to output a preliminary performance indicator vector. The preliminary performance indicator vector is de-normalized to map the model output value to the actual physical range to generate a standardized performance indicator value;

[0040] Based on the standardized performance indicator value, the numerical values of the overall equipment effectiveness OEE, the first pass yield FPY, and the line balancing rate are analyzed;

[0041] The analyzed key performance indicators are output to the optimization strategy generation module.

[0042] As a preferred solution, the optimization strategy generation module is built-in with a reinforcement learning agent. The reinforcement learning agent takes the key performance indicators as the state input of the reward function, and learns the output optimization action strategy by interacting with the environment. The optimization action strategy includes device parameter adjustment instructions, production rhythm control instructions, or re-planning instructions for the mechanical arm path, including:

[0043] Receiving the key performance indicators from the dynamic performance evaluation module;

[0044] The key performance indicators are input into the state space of the reinforcement learning agent as the current environment state;

[0045] The reinforcement learning agent calculates an action value function based on the current environment state through a policy network to generate a candidate action set;

[0046] The candidate action set is processed for exploration and utilization balance, and an optimal action is selected using an ε-greedy strategy;

[0047] The optimal action is mapped to an original action policy, including device parameter adjustment, production rhythm adjustment value, or mechanical arm path point sequence;

[0048] The original action policy is checked for compliance to ensure that it meets the production line safety constraints and operating range;

[0049] The compliant original action policy is encoded into specific device parameter adjustment instructions, production rhythm control instructions, or unloading mechanical arm path re-planning instructions;

[0050] The optimized action policy is output to the control execution module.

[0051] As a preferred solution, the control execution module is connected to the optimized policy generation module and the programmable logic controller (PLC) on the production line, which is used to convert the optimized action policy into control instructions that meet the industrial communication protocol, and then send them to the corresponding execution unit, including:

[0052] The optimized action policy received from the optimized policy generation module includes device parameter adjustment instructions, production rhythm control instructions, or unloading mechanical arm path re-planning instructions;

[0053] The received optimized action policy is parsed and processed to extract the action type and specific parameter values;

[0054] According to the extracted action type, a corresponding industrial communication protocol template is selected, including Modbus, Profinet, or EtherNet / IP;

[0055] The extracted specific parameter values are filled into the data fields of the selected industrial communication protocol template to generate an original control instruction frame;

[0056] The original control instruction frame is processed for checksum calculation and encapsulation to generate a standard control instruction that meets the industrial communication protocol;

[0057] The standard control instruction is sent to the programmable logic controller (PLC) on the production line through industrial Ethernet;

[0058] After the programmable logic controller (PLC) parses and executes the standard control instruction, the control execution module acquires the executed production line data through the data acquisition module to verify the instruction execution effect and form a closed-loop optimization control system.

[0059] From the above technical solutions provided by the present application, the machine vision production line efficiency evaluation and optimization management system provided by the present application has the beneficial effects that:

[0060] Data acquisition and processing are more accurate and comprehensive: through the central synchronous controller, synchronous data acquisition of high-speed industrial cameras, vibration sensors, and RFID readers is realized, ensuring consistent data timestamps and avoiding time misalignment; with the help of deep learning CNN, product quality features are extracted, time sequence alignment and normalization processing of equipment / flow data are performed, and multi-modal data are fused through the attention mechanism, breaking through the limitations of single data and providing high-quality data support for subsequent analysis;

[0061] Efficiency evaluation is real-time and accurate: based on the trained deep neural network model, OEE, FPY, and production line balance rate are calculated in real time according to the preset time window, replacing traditional post-event manual statistics, and efficiency fluctuations can be detected in time; the evaluation process is fully automated, and through model calculation and reverse normalization processing, human subjective errors are reduced, ensuring the accuracy and consistency of the indicators;

[0062] Optimization strategy is intelligent and safe: the built-in reinforcement learning agent takes real-time efficiency indicators as input and dynamically generates optimization actions such as device parameter adjustment and production rhythm control, which can adapt to dynamic changes in the production line and balance strategy exploration and utilization; before generating the strategy, compliance checks are performed to ensure compliance with production line safety constraints and operating ranges, avoiding the risk of illegal adjustments;

[0063] Form a closed-loop optimization to ensure effectiveness: the control execution module converts the optimization strategy into instructions conforming to industrial protocols (Modbus, Profinet, etc.) and issues them to the PLC, and then obtains production line data through the data acquisition module to verify the effectiveness, forming a closed loop of "collection-processing-evaluation-optimization-execution-feedback", ensuring that the optimization actions are accurately implemented and effectively improving the efficiency of the production line. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 Figure 1 is a schematic diagram of the overall structure of the machine vision production line efficiency evaluation and optimization management system of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0066] In order to better understand the above technical solutions, the above technical solutions will be described in detail in combination with the drawings and specific embodiments of the specification.

[0067] As Figure 1As shown, the embodiment of the present application provides a machine vision production line efficiency evaluation and optimization management system, comprising:

[0068] A data acquisition module, comprising a high-speed industrial camera array, a vibration sensor and an RFID reader deployed on the production line, for synchronously collecting image data of products, running state data of equipment and flow data of materials;

[0069] A data processing and fusion module connected to the data acquisition module, comprising:

[0070] An image processing unit for feature extraction of the image data based on a deep learning convolutional neural network model, to obtain a product quality feature vector;

[0071] A data standardization unit for time sequence alignment and normalization processing of the running state data and the flow data, to generate standardized equipment state data;

[0072] A multi-modal data fusion unit for fusion of the product quality feature vector and the standardized equipment state data by using a fusion algorithm based on an attention mechanism, to generate a production line fusion feature set;

[0073] A dynamic efficiency evaluation module connected to the data processing and fusion module, for inputting the production line fusion feature set into a trained efficiency evaluation model, to calculate key performance indicators including at least equipment comprehensive efficiency OEE, first pass yield FPY and production line balancing rate in real time;

[0074] An optimization strategy generation module connected to the dynamic efficiency evaluation module, with a reinforcement learning agent built-in, the reinforcement learning agent taking the key performance indicators as state inputs of a reward function, learning an output optimization action strategy by interacting with the environment, the optimization action strategy including equipment parameter adjustment instructions, production rhythm control instructions or re-planning instructions of a feeding mechanical arm path;

[0075] A control execution module connected to the optimization strategy generation module and a programmable logic controller PLC on the production line, for converting the optimization action strategy into control instructions conforming to an industrial communication protocol, and issuing the control instructions to corresponding execution units to constitute a closed-loop optimization control system.

[0076] In this embodiment, the data acquisition module, the high-speed industrial camera array, the vibration sensor and the RFID reader collect data through a synchronous triggering mechanism, wherein the synchronous triggering mechanism is managed by a central synchronization controller, the central synchronization controller generates a synchronization pulse signal to simultaneously trigger the high-speed industrial camera array to capture multi-angle image data of the product, the vibration sensor to collect vibration frequency and amplitude data of the equipment, and the RFID reader to read RFID tag data of the material, so as to ensure that all collected data have consistent timestamps; the data acquisition module also packs the collected image data, running state data and flow data into time sequence data packets and transmits them to the data processing and fusion module through an industrial Ethernet;

[0077] Further, the data acquisition module mainly consists of a high-speed industrial camera array, a vibration sensor and an RFID reader deployed on the production line, and the core function is to synchronously collect three types of key data: first, multi-angle image data of the product (for subsequent extraction of product quality features); second, running state data of the equipment (specifically, vibration frequency and amplitude data of the equipment, reflecting the health of the equipment); and third, flow data of the material (specifically, RFID tag data of the material, recording the identity and flow location of the material); the module ensures that the data collected by the three types of equipment have consistent timestamps through a synchronous triggering mechanism managed by a central synchronization controller, avoiding time misalignment; at the same time, the collected image data, running state data and flow data are packed into time sequence data packets and transmitted in real time to the data processing and fusion module through an industrial Ethernet, providing complete and accurate raw data support for subsequent data analysis and performance evaluation of the system;

[0078] The high-speed industrial camera array includes:

[0079] Core hardware components: composed of multiple industrial high-speed cameras, deployed at key observation points (such as after product processing, after assembly completion, etc.) of the product flow according to the product transmission path and detection requirements of the production line, to ensure that images of the product from different perspectives can be captured;

[0080] Data acquisition function: only start collecting when receiving the synchronization pulse signal issued by the central synchronization controller, real-time capture multi-angle image data of the product (such as visual information of product surface defects, dimensional accuracy, assembly state, etc.); during the collection process, each frame of image is automatically labeled with the timestamp corresponding to the current synchronization pulse, ensuring the time consistency of the image data with other types of data;

[0081] Data output: real-time transmission of the collected multi-angle image data to the data packing and transmission unit as the raw data source for the subsequent image processing unit to extract the product quality feature vector;

[0082] The vibration sensor includes:

[0083] Core hardware configuration: adopt industrial high-precision vibration sensor, install in the vibration sensitive part (such as equipment spindle, machine base, etc.) of the key equipment (such as machining machine tool, conveying belt motor, assembly mechanical arm, etc.) of production line, ensure that the vibration signal of equipment running can be captured accurately;

[0084] Data acquisition function: in response to the synchronization pulse signal of central synchronous controller, collect the vibration frequency and amplitude data of equipment (the data directly reflects the equipment running state, such as whether there is abnormal vibration, component wear and other problems) at the pulse trigger time; the collected data is marked with real-time synchronization timestamp to avoid time deviation with other data;

[0085] Data output: transmit the collected equipment vibration frequency and amplitude data (i.e. equipment running state data) to data packaging and transmission unit, provide equipment state original data for time alignment and normalization processing of data standardization unit;

[0086] RFID reader includes:

[0087] Core hardware configuration: deployed in the key nodes of production line material flow (such as material entrance, process conversion place, finished product outlet, etc.), equipped with high sensitivity radio frequency identification module, supporting reading material surface or internal integrated RFID tag information;

[0088] Data acquisition function: under the synchronization pulse trigger of central synchronous controller, read the RFID tag data of materials flowing through the identification area, obtain the unique identification (such as material number), production batch, flow transfer progress, belonging process and other flow transfer information (i.e. material flow transfer data) of materials; if there is no material passing in the current period, record "no effective flow transfer data" and mark the corresponding timestamp to ensure the integrity of data recording;

[0089] Data output: transmit the collected material RFID tag data (or "no effective flow transfer data" identification) to data packaging and transmission unit as the flow transfer data source for processing of data standardization unit;

[0090] Central synchronous control unit:

[0091] Core hardware configuration: composed of industrial central synchronous controller, with built-in high-precision clock module and pulse generation circuit, can preset the generation period of synchronization pulse (the period is flexibly set according to the production line beat, such as 100ms / time, 200ms / time, etc.);

[0092] Synchronization trigger function: generate unified synchronization pulse signal according to the preset period, the signal is sent to high-speed industrial camera array, vibration sensor and RFID reader through special signal line at the same time, ensure that three kinds of collection equipment start data collection operation at the same time, eliminate the time difference of data collected by different equipment from the source;

[0093] Timestamp management: Assign a unique timestamp (accurate to the millisecond level) to each synchronization pulse and send it synchronously to the data packaging and transmission unit, ensuring that all collected data can be marked with the corresponding unified timestamp, providing a reference for subsequent time alignment of data processing;

[0094] Data packaging and transmission unit:

[0095] Data receiving function: Real-time reception of image data transmitted by high-speed industrial camera array, operation state data transmitted by vibration sensor, and flow data transmitted by RFID reader, as well as corresponding timestamp information issued by the central synchronization controller;

[0096] Data packaging function: According to the format of "unified timestamp + data type identification + raw data", integrate three types of data (or part of valid data) under the same timestamp into a structured time series data packet; for example, a data packet under a certain timestamp contains "timestamp T1 + image data (device A perspective 1-3) + operation state data (vibration frequency f1, amplitude A1) + flow data (material number M001)", ensuring the time correlation of data within the data packet;

[0097] Data transmission function: Through industrial Ethernet link, real-time transmission of packaged time series data packets to the data processing and fusion module; during transmission, industrial-level data verification mechanism (such as CRC cyclic redundancy check) is adopted to ensure that data is not lost or tampered with during transmission, and to adapt to the complex electromagnetic environment of industrial field.

[0098] In this embodiment, the data processing and fusion module is connected to the data acquisition module, and the core consists of an image processing unit, a data standardization unit, and a multi-modal data fusion unit. The overall function is divided into two stages:

[0099] Data processing stage: Process the two types of core data transmitted by the data acquisition module respectively - through the image processing unit to extract features from product image data and generate feature vectors representing product quality; through the data standardization unit to align and normalize device operation state data (vibration frequency, amplitude) and material flow data (RFID tag data), eliminating data dimension differences and time misalignment, and generating standardized device state data;

[0100] Multi-modal fusion stage: Through the multi-modal data fusion unit, using a fusion algorithm based on attention mechanism, the product quality feature vector and the standardized device state data are deeply fused to generate a production line fusion feature set containing product, device, and material correlation information, which is finally output to the dynamic performance evaluation module, providing high-quality feature input for real-time calculation of key performance indicators (OEE, FPY, production line balance rate).

[0101] The data processing and fusion module comprises:

[0102] The image processing unit comprises:

[0103] The core function of the image processing unit is to extract a feature vector representing the quality state from product image data based on a deep learning convolutional neural network (CNN) model. The image processing unit extracts features from image data based on a deep learning convolutional neural network model to obtain a product quality feature vector, including:

[0104] The collected image data is preprocessed, including image denoising, size adjustment, and pixel value normalization, to generate standardized image data;

[0105] The standardized image data is input into a pre-trained deep learning convolutional neural network model, and is processed by multiple convolutional layers, activation function layers, and pooling layers for forward propagation to generate multi-level feature maps;

[0106] The multi-level feature maps are processed for feature fusion, and high-level semantic features are extracted through the fully connected layer or global pooling layer in the convolutional neural network to generate an initial feature vector;

[0107] The initial feature vector is processed for dimension reduction and standardization to obtain a final feature vector representing product quality;

[0108] The specific process and functions are as follows:

[0109] Image preprocessing: receiving product multi-angle image data transmitted by the data acquisition module, performing three basic processes:

[0110] Image denoising: using an industrial-level denoising algorithm (such as Gaussian filtering, median filtering) to remove industrial environment noise (such as workshop light interference, equipment reflection) in the image;

[0111] Size adjustment: uniformly scaling images of different angles and different resolutions to a preset size (such as 224x224 pixels) to adapt to the input requirements of the pre-trained CNN model;

[0112] Pixel value normalization: mapping the image pixel value from the range [0, 255] to the range [0, 1] or [-1, 1] (according to the pre-trained model parameters), eliminating the influence of pixel value magnitude difference on model training and feature extraction, and finally generating standardized image data;

[0113] CNN model forward propagation: inputting the standardized image data into a pre-trained deep learning convolutional neural network model (such as ResNet, VGG series model), and processing it through the multi-layer structure of the model in turn:

[0114] Convolutional layer: Extract local features of the image (such as product edges, surface texture, defect contours) through multiple convolution kernel sliding calculations;

[0115] Activation function layer: Use ReLU (Rectified Linear Unit) activation function to introduce non-linear relationship and enhance the model's ability to express complex features;

[0116] Pooling layer: Reduce feature map dimension and calculation amount while preserving key feature information through max pooling or average pooling operations, and finally generate multi-level feature maps (different levels correspond to different abstract degree of features, shallow layer is edge feature, deep layer is semantic feature);

[0117] High-level semantic feature extraction: Perform feature fusion processing on multi-level feature maps, compress high-dimensional feature maps into low-dimensional vectors through CNN model's fully connected layer or global pooling layer (such as global average pooling), extract high-level semantic features that can reflect product quality core information, and generate initial feature vectors (such as 1024-dimensional, 2048-dimensional vectors);

[0118] Feature vector optimization: Perform dimension reduction (using principal component analysis PCA or linear discriminant analysis LDA) and standardization processing on the initial feature vector to remove redundant features and reduce vector dimension, and finally obtain a product quality feature vector with unified dimension and clear features, which is output to the multi-modal data fusion unit;

[0119] Data standardization unit:

[0120] The core function of the data standardization unit is to perform time series alignment and normalization on the equipment running state data and material flow data transmitted by the data acquisition module, generating standardized equipment state data. The data standardization unit is used to perform time series alignment and normalization processing on the running state data and flow data, generating standardized equipment state data, including:

[0121] Receive running state data and flow data from the data acquisition module, where the running state data includes device vibration frequency and amplitude data, and the flow data includes material RFID tag data;

[0122] Timestamp analysis of running state data and flow data to extract original time information of each data point;

[0123] Based on the synchronization time reference provided by the central synchronous controller, perform time series alignment processing on the running state data and flow data after timestamp analysis, resample the data to a unified time sequence through linear interpolation method, and ensure that all data points have consistent time intervals;

[0124] The normalized data is combined as standardized equipment state data and output to the multi-modal data fusion unit.

[0125] The normalized data is combined as standardized equipment state data and output to the multi-modal data fusion unit.

[0126] The specific process and functions are as follows:

[0127] Data reception and analysis: receiving two types of data transmitted by the data acquisition module:

[0128] Equipment operating state data: including equipment vibration frequency (Hz), vibration amplitude (mm);

[0129] Material flow data: including material RFID tag information (such as material number, flow node, timestamp);

[0130] At the same time, the original timestamps of the two types of data are analyzed, and the collection time information of each data point is extracted (based on the central synchronous controller time reference of the data acquisition module);

[0131] Time sequence alignment processing: taking the synchronous time reference provided by the central synchronous controller of the data acquisition module as the standard, the analyzed operating state data and flow data are time sequence aligned:

[0132] For data points with unmatched timestamps (such as vibration sensor collection frequency of 10Hz, RFID reader collection frequency of 5Hz), linear interpolation method is used to resample the data to a unified time sequence (such as every 200ms a data point);

[0133] Ensure that all data points have consistent time intervals, eliminate time misalignment caused by collection frequency differences, and realize the data correlation of "equipment state-material flow at the same timestamp";

[0134] Data normalization processing: using the Min-Max Scaling algorithm, the operating state data and flow data after time sequence alignment are normalized, the calculation formula is: , wherein, is the normalized equipment operating state data (such as vibration frequency, vibration amplitude) or material flow data (RFID tag associated data mapped to numerical form), with a value range of [0, 1]; is the original data value of the equipment operating state data or material flow data (mapped to numerical form) (such as vibration frequency original value of 50Hz, vibration amplitude original value of 0.1mm); The minimum value in the same type of data set (such as the minimum frequency value in all vibration frequency data, the minimum amplitude value in all vibration amplitude data); The maximum value in the same type of data set (such as the maximum frequency value in all vibration frequency data, the maximum amplitude value in all vibration amplitude data);

[0135] The data value is uniformly mapped to the range [0, 1] by the formula, eliminating the dimensional difference of different data types (such as the order of magnitude difference between vibration frequency "Hz" and vibration amplitude "mm"), avoiding the influence of different data magnitudes on subsequent fusion effect;

[0136] Standardized data generation: combine the normalized device running state data and material flow conversion data in chronological order to generate structured standardized device state data (such as "timestamp T1: vibration frequency 0.6 (normalized), vibration amplitude 0.4 (normalized), RFID tag M001 (normalized result after mapping to numerical value)"), output to the multi-modal data fusion unit;

[0137] Multi-modal data fusion unit:

[0138] The core function of the multi-modal data fusion unit is to use the attention mechanism-based fusion algorithm to fuse the product quality feature vector and the standardized device state data to generate the production line fusion feature set. The multi-modal data fusion unit uses the attention mechanism-based fusion algorithm to fuse the product quality feature vector and the standardized device state data to generate the production line fusion feature set, including:

[0139] Receive the product quality feature vector from the image processing unit and the standardized device state data from the data standardization unit;

[0140] Input the product quality feature vector and the standardized device state data into the linear mapping layer respectively to generate the corresponding query vector, key vector and value vector;

[0141] Calculate the dot product similarity between the query vector and the key vector, and perform softmax normalization on the similarity to generate the attention weight;

[0142] Use the attention weight to perform weighted summation operation on the value vector to generate the weighted context feature vector;

[0143] Concatenate the weighted context feature vector and the product quality feature vector to generate the preliminary fusion feature vector;

[0144] Perform full connection layer processing and activation function transformation on the preliminary fusion feature vector to generate the final production line fusion feature set;

[0145] The specific process and functions are as follows:

[0146] Data receiving: synchronously receiving product quality feature vectors (such as 256-dimensional vectors) from the image processing unit and standardized equipment state data (time-series multi-dimensional data, such as containing 3 feature dimensions every 200 ms) from the data standardization unit;

[0147] Linear mapping processing: inputting the two types of data into independent linear mapping layers respectively, and generating three groups of vectors corresponding thereto through linear transformation:

[0148] For the product quality feature vectors, generating a query vector (Query) and a key vector (Key);

[0149] For the standardized equipment state data, generating a value vector (Value);

[0150] The purpose of linear mapping is to convert the features of different dimensions and different semantics of the two types of data into the same feature space, laying a foundation for subsequent similarity calculation;

[0151] Attention weight calculation: calculating the dot product similarity between the query vector and the key vector (measuring the degree of association between the two types of data, such as the matching degree of “a product quality feature and a device state at a certain time”), and performing softmax normalization processing on the similarity result, the calculation formula is: wherein, is the attention weight matrix calculated by the query vector and the key vector, each element in the matrix corresponds to an association weight of “product quality feature-equipment state feature”, and the value range is [0, 1]; is the query vector obtained by converting the product quality feature vector through the linear mapping layer, the dimension is consistent with the key vector, and is used to represent the “query dimension” of the product quality feature; is the key vector obtained by converting the product quality feature vector through the linear mapping layer, and is used to represent the “key dimension” of the product quality feature, which is used to calculate the association degree together with the query vector; is the transpose matrix of the key vector ; is the dimension of the key vector (such as 64 dimensions, 128 dimensions), the parameter is introduced to scale the dot product result of the large-dimensional vector, so as to avoid the weight being too concentrated or the gradient vanishing after softmax normalization due to the too large numerical value; is the softmax normalization function,

[0152] After normalization, the attention weight is generated, and the greater the weight value is, the higher the corresponding data association degree is (such as the association weight of “device vibration anomaly and product surface defect” at a certain time is higher);

[0153] Context feature generation: using attention weight to perform weighted sum operation on the value vector (after normalization of the equipment state data mapping), the formula is (wherein is the value vector), giving higher weight to data features with high correlation and lower weight to low correlation features, generating a weighted context feature vector that reflects the "equipment state-product quality" correlation relationship;

[0154] Feature splicing and optimization: splice and combine the weighted context feature vector and the product quality feature vector (such as 256-dimensional product vector + 64-dimensional context vector = 320-dimensional preliminary vector), generate a preliminary fusion feature vector; then input the preliminary vector into the fully connected layer, and perform nonlinear transformation through the activation function such as ReLU, further integrate feature information, and finally generate a production line fusion feature set (time-sequenced fusion features, such as a 320-dimensional fusion vector every 200ms), output to the dynamic performance evaluation module.

[0155] In this embodiment, the dynamic performance evaluation module is used to input the production line fusion feature set into the trained performance evaluation model to calculate key performance indicators including at least equipment comprehensive efficiency OEE, first pass yield FPY and production line balance rate, including:

[0156] Receiving the production line fusion feature set from the multi-modal data fusion unit;

[0157] Performing time window division processing on the production line fusion feature set to generate a feature sequence of fixed time length;

[0158] Input the feature sequence into the trained performance evaluation model, the performance evaluation model is based on a deep neural network architecture, receives the feature sequence through the input layer, performs nonlinear transformation and feature extraction through multiple hidden layers, and outputs a preliminary performance indicator vector. The preliminary performance indicator vector is subjected to inverse normalization processing, the model output value is mapped to the actual physical range, and a standardized performance indicator value is generated;

[0159] Based on the standardized performance indicator value, the numerical values of the equipment comprehensive efficiency OEE, the first pass yield FPY and the production line balance rate are analyzed;

[0160] Output the analyzed key performance indicators to the optimization strategy generation module;

[0161] Further, the dynamic performance evaluation module is a key hub connecting the data processing and fusion module and the optimization strategy generation module, and the core function is to convert the production line feature data after multi-modal fusion into quantifiable and interpretable key performance indicators; it receives the production line fusion feature set output by the data processing and fusion module, calculates the three core indicators of equipment overall efficiency (OEE), first pass yield (FPY), and production line balance rate in real time through the trained deep neural network performance evaluation model, provides accurate performance “diagnosis results” for the optimization strategy generation module, and is the key decision basis for the system to realize the “evaluation-optimization” closed loop; the following is a detailed description of this module from the whole to the details:

[0162] I. Overall function overview:

[0163] The dynamic performance evaluation module is connected to the data processing and fusion module, and the overall function is divided into three core links of “feature preprocessing-model calculation-index analysis”:

[0164] Feature preprocessing: time window division is performed on the received production line fusion feature set, and the continuous feature data is cut into feature sequences of fixed time length, ensuring that the data input into the model has time sequence correlation and uniformity;

[0165] Model calculation: input the feature sequence into the trained performance evaluation model (based on deep neural network architecture), and output the normalized preliminary performance indicator vector through the nonlinear transformation and feature extraction of the model;

[0166] Index analysis: perform inverse normalization on the preliminary performance indicator vector, map the dimensionless value output by the model back to the actual physical range, generate standardized performance indicator values, and then analyze the specific values of equipment overall efficiency (OEE), first pass yield (FPY), and production line balance rate, and finally output to the optimization strategy generation module, providing data support for the formulation of subsequent optimization action strategies;

[0167] II. Submodule composition and function:

[0168] (I) Feature sequence generation unit:

[0169] The core function of the feature sequence generation unit is to divide the production line fusion feature set into time windows, generate feature sequences that adapt to the input format of the performance evaluation model, and the specific process and function are as follows:

[0170] Feature set reception: real-time reception of the production line fusion feature set transmitted by the data processing and fusion module; the feature set is time-series data, containing multi-dimensional fusion features (such as 320-dimensional vectors, covering product quality, equipment status, and material flow correlation information) corresponding to each timestamp (such as 200ms / each);

[0171] Time window parameter configuration: based on the production line beat and performance evaluation requirements, a fixed time window length (such as 10s, 30s, which needs to be consistent with the window parameter during model training) is preset, and the window sliding step and the feature set timestamp interval are kept synchronous (such as 200ms / step), ensuring that there is no data omission or repetition;

[0172] Feature sequence cutting: according to the preset time window, the continuous production line fusion feature set is cut in a sliding manner; for example, when the window length is 10s and the timestamp interval is 200ms, each window contains 50 continuous timestamp fusion feature vectors, which are combined to form a feature sequence of "50x320" dimensions;

[0173] Sequence verification and output: the integrity of the cut feature sequence is verified (such as checking whether the number of feature vectors in the window meets the standard and whether there are missing values), and after the verification is passed, the feature sequence is output to the performance evaluation model calculation unit, and if the verification fails, the data processing and fusion module is fed back to retransmit the fusion feature set of the corresponding period;

[0174] (2) Performance evaluation model calculation unit:

[0175] The performance evaluation model calculation unit is the core of the module, which converts the feature sequence into a preliminary performance index vector through a trained deep neural network model. The specific process and functions are as follows:

[0176] Model loading and initialization: load the pre-trained performance evaluation model at startup. The model is based on a deep neural network architecture (such as multi-layer perceptron MLP, long short-term memory network LSTM, which needs to be consistent with the architecture during training), and the loaded parameters include input layer dimension (matched with feature sequence dimension, such as "50x320"), number of hidden layers / neurons, and output layer dimension (fixed at 3, corresponding to OEE, FPY, and production line balance rate);

[0177] Feature sequence input: receive the compliant feature sequence output by the feature sequence generation unit and flatten it into a vector format that the model input layer can accept (such as flattening the "50x320" two-dimensional sequence into a "16000" dimensional one-dimensional input vector);

[0178] Model forward propagation: the input vector is processed by each layer of the model in turn:

[0179] Input layer: receives the flattened feature sequence vector and passes it to the first hidden layer;

[0180] Hidden layer: linear transformation wherein, is the intermediate result of the hidden layer linear transformation, is the weight matrix of the hidden layer (used to quantify the influence of each dimension of the input vector on the output of the hidden layer), Input vector for hidden layer (i.e. output result of previous layer), Bias term for hidden layer (used to adjust linear transformation result, avoid fitting deviation caused by model relying on input vector) and activation function (such as ReLU, Siqmoid) are used for non-linear transformation, gradually extracting deep features related to performance indicators;

[0181] Output layer: output a preliminary performance indicator vector with dimension 3 through linear transformation, each element in the vector is a normalized value in the range of [0, 1], corresponding to the preliminary prediction results of OEE, FPY and line balance rate respectively;

[0182] Preliminary indicator output: transmit the preliminary performance indicator vector generated by the output layer to the indicator denormalization and analysis unit, and record the model calculation timestamp to ensure time consistency with the subsequent indicator analysis results;

[0183] (Three) Indicator denormalization and analysis unit:

[0184] The core function of the indicator denormalization and analysis unit is to convert the normalized indicator value output by the model into the actual physical performance indicator and analyze the specific value. The specific process and functions are as follows:

[0185] Preliminary indicator receiving: receive the preliminary performance indicator vector output by the performance evaluation model calculation unit (such as [0.82, 0.95, 0.88]), and load the "indicator-maximum value mapping table" saved in the model training stage (record the historical minimum value and historical maximum value of OEE, FPY and line balance rate respectively);

[0186] Denormalization processing: use the denormalization formula to map each normalized value in the preliminary performance indicator vector back to the actual physical range, the formula is: , where is the normalized performance indicator value after denormalization (with actual physical meaning, such as OEE of 82%, FPY of 95%), is an element in the preliminary performance indicator vector (normalized value, range [0, 1], such as OEE corresponding to ), is the historical minimum value of the performance indicator in the model training data set (such as OEE , FPY ), is the historical maximum value of the performance indicator in the model training data set (such as OEE , FPY ); for example, if OEE , , , then the reverse normalization ;

[0187] Index analysis: based on the dimension order of the normalized performance index value (consistent with the model output layer definition: the first dimension corresponds to OEE, the second dimension corresponds to FPY, and the third dimension corresponds to the line balance rate), the specific values of the three indexes are extracted, such as the analysis result is "OEE=92.8%, FPY=95.2%, line balance rate=88.5%";

[0188] Index output: the analyzed key performance indicators (including specific values and calculation timestamps) are output to the optimization strategy generation module in real time, and are stored in the local index log library for subsequent performance trend analysis and historical data tracing;

[0189] Three, key technical principles:

[0190] (1) Time window division principle:

[0191] The core purpose of time window division is to ensure that the feature data input into the model has "time sequence correlation"; the calculation of production line performance indicators (such as OEE, line balance rate) needs to be based on equipment operation, product quality, and material flow data within a certain time period. Single timestamp feature data cannot reflect the dynamic changes of performance; by dividing the fixed time window (such as 10s), the continuous production line fusion feature set is converted into a feature sequence containing multiple time sequence nodes, which not only retains the time dimension information of the data (such as the correlation between the vibration trend of the equipment and the product defect rate within a certain 10s), but also makes the data format of the input model uniform (fixed length, fixed dimension), avoiding model calculation errors caused by inconsistent data lengths, and matching the needs of "periodic performance evaluation" in actual industrial scenarios (such as updating performance indicators every 10s);

[0192] (2) Deep neural network performance evaluation principle:

[0193] The performance evaluation model adopts a deep neural network architecture, and the core principle is to realize the complex mapping of "features-index" through multiple layers of nonlinear transformation:

[0194] Input layer adaptation: the flattened feature sequence vector is taken as input to ensure that the model can receive multi-dimensional, time-sequenced fusion features (covering product, equipment, and material correlation information);

[0195] Hidden layer feature extraction: through the weight matrix of multiple hidden layers (W1, W2, …, Wn), the feature extraction of the input data is realized, and the high-level abstract representation of the performance index is obtained; ) and an activation function (such as ReLU), step by step, from the fused features, abstract features strongly related to the performance indicators are extracted; for example, from "device vibration frequency feature + product defect feature", the "device failure affects production efficiency" related feature is extracted, and from "material flow speed feature + process time-consuming feature", the "production line bottleneck process" feature is extracted; wherein the linear transformation of the hidden layer follows (Wherein, is the intermediate result of the hidden layer linear transformation, is the weight matrix of the hidden layer, is the input vector of the hidden layer, is the bias term of the hidden layer), and the activation function introduces nonlinearity through (ReLU function, wherein, is the output value of the activation function, is the intermediate result of the hidden layer linear transformation) so that the model can fit the complex feature-indicator mapping relationship;

[0196] Output layer mapping: the output layer outputs three normalized preliminary performance indicator values through linear transformation, realizing the precise conversion of "multi-dimensional fused features → single indicator value";

[0197] Model training basis: in the model training phase, the "production line fused feature sequence" is taken as the input, and the "actually measured OEE / FPY / production line balance rate" is taken as the label. First, the actual label is mapped to the [0, 1] range through "min-max normalization", and then the weight matrix ( ) and the bias term ( ) are iteratively optimized through the back propagation algorithm (such as gradient descent) to minimize the error between the model prediction value and the normalized label, ensuring that the trained model has high-precision performance prediction capability;

[0198] (Three) denormalization principle:

[0199] In the model training phase, to avoid the influence of the order of magnitude difference of different indicators on the training effect (such as OEE taking the value range of 60%-100%, and production line balance rate taking the value range of 50%-90%), the actual indicator value is mapped to the [0, 1] range through "min-max normalization"; therefore, the preliminary performance indicator vector output by the model is a dimensionless value in the [0, 1] interval, which cannot directly reflect the actual performance; denormalization through inverse calculation (formula: ), restores the normalized preliminary indicator value to an indicator value with actual physical meaning (such as restored to ), ensuring that the output OEE, FPY, and production line balance rate meet the conventional quantification standards in the industrial scene, facilitating the understanding and use of the optimization strategy generation module.

[0200] In this embodiment, the optimization strategy generation module is built-in reinforcement learning agent. The reinforcement learning agent takes the key performance indicators as the state input of the reward function, learns the output optimization action strategy through interaction with the environment, and the optimization action strategy includes device parameter adjustment instructions, production rhythm control instructions, or unloading mechanical arm path re-planning instructions, including:

[0201] Receiving the key performance indicators from the dynamic performance evaluation module;

[0202] Input the key performance indicators as the current environment state to the state space of the reinforcement learning agent;

[0203] The reinforcement learning agent calculates the action value function based on the current environment state through the policy network, and generates a candidate action set;

[0204] Balancing exploration and utilization of the candidate action set, and selecting the optimal action using the ε-greedy strategy;

[0205] Map the optimal action to the original action strategy, including device parameter adjustment amount, production rhythm adjustment value, or mechanical arm path point sequence;

[0206] Conduct compliance check on the original action strategy to ensure that it meets the production line safety constraints and operating range;

[0207] Encode the compliant original action strategy into specific device parameter adjustment instructions, production rhythm control instructions, or unloading mechanical arm path re-planning instructions;

[0208] Output the optimization action strategy to the control execution module;

[0209] Further, the optimization strategy generation module is connected to the dynamic performance evaluation module, and the core function is to receive three types of key performance indicators, including device comprehensive efficiency (OEE), first pass yield (FPY), and line balance rate, and convert them into the environment state input of the reinforcement learning agent. The agent learns through interaction with the production line environment, calculates and generates a candidate optimization action set, and after exploration and utilization balancing and compliance checking, maps and encodes the effective action into specific device parameter adjustment instructions, production rhythm control instructions, or unloading mechanical arm path re-planning instructions, and finally outputs to the control execution module to promote the line performance to the optimal direction. At the same time, the module will continuously iterate the agent decision logic based on the production line feedback data after control execution, to ensure the adaptability and effectiveness of the strategy, and the specific operation steps are as follows:

[0210] Key performance indicator receiving: real-time receiving of key performance indicator data output by the dynamic performance evaluation module, including specific values of equipment overall efficiency OEE (such as 92.8%), first pass yield FPY (such as 95.2%), and line balancing rate (such as 88.5%), and simultaneously receiving the calculation time stamp corresponding to the indicators, to ensure that the state data matches the actual running period of the production line;

[0211] Indicator standardization processing: using the normalization algorithm corresponding to the inverse normalization process of the dynamic performance evaluation module, the actual performance indicator value received is mapped to the [0, 1] range adapted to the state space of the reinforcement learning agent, and the calculation formula is (Wherein, is the standardized state value of the th key performance indicator, corresponding to OEE, FPY, and line balancing rate, respectively; is the actual received value of the th indicator; is the historical minimum effective value of the th indicator, such as the of OEE; is the theoretical maximum effective value of the th indicator, such as the of OEE), eliminating the interference of different indicator magnitudes on agent decision-making;

[0212] State vector construction: combining the three standardized state values in the fixed order of “OEE FPY line balancing rate”, generating a current environment state vector with a dimension of 3, for example , and adding a time stamp label to the state vector, and outputting to the reinforcement learning agent decision unit;

[0213] Strategy network loading and initialization: when the module starts, load the pre-trained reinforcement learning strategy network model, the model parameters include the input layer dimension (fixed at 3, matching the state vector dimension), the number of hidden layers (such as 2 layers), the number of hidden layer neurons (such as 64 per layer), the output layer dimension (matching the number of candidate actions, such as 6 candidate actions corresponding to an output layer dimension of 6); At the same time, initialize the exploration rate (such as an initial value of 0.1, used to balance exploration and utilization), the discount factor (such as 0.9, used to measure the weight of future rewards), and other decision-making parameters;

[0214] Action value function calculation: inputting the state and the state vector Input policy network, through linear transformation of hidden layer and ReLU activation function processing, calculate the value of each candidate action, output action value vector , the calculation formula is ReLU ReLU (wherein, is the value of the action executed in the state ; , respectively, the weight matrix and bias term of the first layer of hidden layer; , respectively, the weight matrix and bias term of the second layer of hidden layer; ReLU is a rectified linear unit activation function, used to introduce nonlinear decision-making ability

[0215] - Greedy policy selection: based on action value vector and exploration rate , the optimal action selection: when the randomly generated probability value is greater than (as ), select the action with the maximum action value (that is, use the known optimal policy); when , randomly select an action in the candidate action set (that is, explore new potential optimization actions), the selection probability formula is , wherein, is the selection probability of action ; is the total number of candidate action set; is the action with the maximum action value, which ensures that the existing experience is used while avoiding falling into local optimum;

[0216] Optimal action mapping: map the optimal action output by the reinforcement learning intelligent agent decision unit to specific original action policy parameters:

[0217] If the optimal action is "equipment parameter adjustment", it is mapped to equipment ID (such as "lathe #3"), parameter type (such as "spindle speed"), and adjustment amount (such as "+50 r / min");

[0218] If the optimal action is "production pace control", it is mapped to the pace adjustment object (such as "assembly process") and adjustment value (such as "-2s", i.e. the pace is shortened from 20s to 18s);

[0219] If the optimal action is "replanning of the blanking robot path", it is mapped to the robot ID (e.g., "Robot #2") and the new sequence of path points (e.g., "(X1, Y1, Z1)→(X2, Y2, Z2)→(X3, Y3, Z3)", a total of 5 path points);

[0220] Safety constraint compliance check: Based on the safety constraint thresholds preset by the production line, the original action strategy is checked for compliance:

[0221] Device parameter adjustment must meet the condition "parameter value ∈ [lower safety limit, upper safety limit]", such as the safety range of the lathe spindle speed is 500-3000 r / min, if the adjustment value is "+50 r / min" and the speed is 3020 r / min after adjustment, it is determined to be non-compliant;

[0222] Production rhythm control must meet the condition "adjusted rhythm ≥ minimum production time", such as the minimum production time of the assembly process is 15s, if the adjustment value is "-6s" and the rhythm is 14s after adjustment, it is determined to be non-compliant;

[0223] Blanking robot path must meet the condition "all path points ∈ robot working space", if a path point (Z3) exceeds the maximum lifting height of the robot, it is determined to be non-compliant;

[0224] Strategy correction and confirmation: For non-compliant original action strategies, modify them according to the principle of "close to the optimal value and comply with the constraints", such as adjusting the spindle speed from "+50 r / min" to "+30 r / min" (making the adjusted speed 3000 r / min, which complies with the upper limit constraint); After modification, check again until compliance, and generate the confirmed original action strategy;

[0225] Optimization instruction structured coding: Encode the confirmed original action strategy into a specific instruction format that meets the parsing requirements of the control execution module:

[0226] Device parameter adjustment instruction: The encoding format is "device ID | parameter type | adjusted value | effective time", such as "Lathe #3 | spindle speed | 3000 r / min | immediately";

[0227] Production rhythm control instruction: The encoding format is "process ID | pre-adjustment rhythm | post-adjustment rhythm | switching opportunity", such as "Assembly Process #2 | 20s | 18s | when the next batch of materials starts";

[0228] The blanking mechanical arm path re-planning instruction: the coding format is "mechanical arm ID | path point number | path point 1 coordinate | path point 2 coordinate |... | path point n coordinate", such as "mechanical arm #2 | 5 | (100, 200, 50) | (120, 220, 50) | (140, 240, 50) | (160, 220, 50) | (180, 200, 50)";

[0229] Instruction verification and output: the coded optimized action strategy instruction is subjected to integrity verification (such as checking whether the instruction field is missing or the coordinate format is correct), and after the verification is passed, the instruction and the corresponding timestamp are packaged and output in real time to the control execution module through the industrial Ethernet; if the verification fails, it is fed back to the action strategy mapping and compliance checking unit for recoding.

[0230] In this embodiment, the control execution module is connected to the optimized strategy generation module and the programmable logic controller (PLC) on the production line, for converting the optimized action strategy into control instructions conforming to the industrial communication protocol, and issuing them to the corresponding execution unit, including:

[0231] Receiving the optimized action strategy from the optimized strategy generation module, the optimized action strategy including device parameter adjustment instructions, production rhythm control instructions or blanking mechanical arm path re-planning instructions;

[0232] Analyzing and processing the received optimized action strategy, extracting the action type and specific parameter value;

[0233] According to the extracted action type, selecting a corresponding industrial communication protocol template, the industrial communication protocol template including Modbus, Profinet or EtherNet / IP;

[0234] Filling the extracted specific parameter value into the data field of the selected industrial communication protocol template to generate an original control instruction frame;

[0235] Carrying out check code calculation and encapsulation processing on the original control instruction frame to generate a standard control instruction conforming to the industrial communication protocol;

[0236] Issuing the standard control instruction to the programmable logic controller (PLC) on the production line through the industrial Ethernet;

[0237] After the programmable logic controller (PLC) analyzes and executes the standard control instruction, the control execution module acquires the executed production line data through the data acquisition module to verify the instruction execution effect and constitute a closed-loop optimization control system;

[0238] Further, the control execution module is connected to the optimization strategy generation module at one end and interfaces with the PLC on the production line at the other end, and the core functions cover four links: first, strategy receiving and analysis, receiving the optimization action strategy and extracting the action type (device parameter adjustment, production rhythm control, and re-planning of the unloading mechanical arm path) and specific parameter values; second, industrial protocol adaptation, selecting a matching industrial communication protocol template (Modbus, Profinet, or EtherNet / IP) according to the action type; third, instruction generation and packaging, filling the parameter values into the protocol template, generating control instruction frames conforming to the industrial standard and completing the verification; fourth, instruction issuing and closed-loop verification, issuing the instruction to the corresponding PLC for execution, while obtaining the production line data after execution through the data acquisition module to verify the instruction effect, and finally forming a closed-loop optimization control system of “strategy-instruction-execution-feedback” to ensure accurate landing and effectiveness of the optimization action; the specific operation steps are as follows:

[0239] Optimization strategy receiving: real-time receiving of the optimization action strategy from the optimization strategy generation module through industrial Ethernet, and the strategy format is a structured instruction (such as “device ID|parameter type|adjusted value|effective time” “process ID|pre-adjustment rhythm|post-adjustment rhythm|switching time” “mechanical arm ID|path point number|path point coordinate sequence”), while synchronously receiving the timestamp corresponding to the strategy to ensure matching with the production line operation period;

[0240] Action type identification: keyword analysis of the received optimization action strategy to identify the action type:

[0241] If the strategy contains “device ID” and “parameter type” (such as “main shaft speed” and “feed amount”), it is determined as a “device parameter adjustment instruction”;

[0242] If the strategy contains “process ID” and “post-adjustment rhythm”, it is determined as a “production rhythm control instruction”;

[0243] If the strategy contains “mechanical arm ID” and “path point coordinate”, it is determined as a “re-planning of the unloading mechanical arm path instruction”;

[0244] Specific parameter extraction: for different action types, extract the core parameter values:

[0245] Device parameter adjustment instruction: extract “device ID” (such as “lathe #3”), “adjusted parameter value” (such as “3000 r / min”), and “effective time” (such as “immediately” and “at the start of the next batch of materials”);

[0246] Production rhythm control instruction: extract “process ID” (such as “assembly process #2”), “post-adjustment rhythm value” (such as “18 s”), and “switching time” (such as “after the current batch is completed”);

[0247] Unloading robot path re-planning instruction: extract "robot ID" (such as "robot #2"), "number of path points" (such as "5"), "path point coordinate sequence" (such as "(100, 200, 50), (120, 220, 50), (140, 240, 50), (160, 220, 50), (180, 200, 50)").

[0248] Parameter format verification: verify whether the extracted parameter value format meets the requirements of industrial equipment, such as equipment parameter value should contain units ("r / min", "mm / s"), path point coordinates should be integers or have 1 decimal place, and beat value should be positive. If the format is incorrect (such as parameter value without units, negative coordinates), feedback to the optimization strategy generation module to resend, ensuring parameter validity.

[0249] Protocol template library loading: when the module is initialized, load the preset industrial communication protocol template library, including standard templates of three types of protocols:

[0250] Modbus protocol template: suitable for parameter adjustment of small and medium-sized equipment (such as single machine tool, small conveyor belt), template contains fixed format of address field, function code (such as "06H" represents writing single register), data field, and verification field;

[0251] Profinet protocol template: suitable for control of high-speed synchronous equipment (such as multi-axis robot, continuous production line), template supports real-time data transmission (RT) and non-real-time data transmission (IRT), contains field definition of device name, data identifier, and data length;

[0252] EtherNet / IP protocol template: suitable for communication between cross-brand equipment (such as PLC and sensors from different manufacturers), template is based on CIP protocol specification, contains structure framework of message header, data segment, and message tail;

[0253] Protocol automatic matching: according to the action type identified by the strategy receiving and analyzing unit, automatically select the corresponding industrial communication protocol:

[0254] Device parameter adjustment instruction (such as lathe spindle speed adjustment): if the device is single machine independent control, match Modbus protocol; if the device is a production line synchronous control node, match Profinet protocol;

[0255] Production beat control instruction (such as whole assembly line beat adjustment): due to the need to synchronize multiple process equipment, match Profinet protocol (high real-time requirement) or EtherNet / IP protocol (high cross-device compatibility requirement);

[0256] Replanning of the blanking mechanical arm path instructions (such as multi-axis mechanical arm path update): due to the need for high-speed transmission of path point coordinate sequences, the Profinet protocol is matched (supporting real-time path data delivery);

[0257] Protocol parameter configuration: configure communication parameters for matched protocols:

[0258] Modbus protocol: configure slave address (such as "1" corresponding to the Modbus slave address of lathe #3), baud rate (such as "115200bps"), data bits ("8 bits"), stop bits ("1 bit"), and check mode ("even check");

[0259] Profinet protocol: configure the PROFINET device name of the device (such as "Robot_2" corresponding to mechanical arm #2), real-time channel priority ("high priority" for path data transmission);

[0260] EtherNet / IP protocol: configure the IP address of the device (such as "192.168.1.102" corresponding to the PLC of assembly process #2), CIP object identifier (such as "0x6B" indicating motion control object);

[0261] Original instruction frame filling: fill the specific parameter values extracted by the strategy receiving and analysis unit into the protocol template data domain matched by the industrial protocol adaptation unit:

[0262] Modbus protocol (device parameter adjustment): if the spindle speed of lathe #3 is adjusted to 3000r / min, and the Modbus register address corresponding to the spindle speed is "40001", then fill "40001" (address), "06H" (function code), and "0BB8H" (hexadecimal value of 3000) in the template data domain;

[0263] Profinet protocol (mechanical arm path replanning): fill the mechanical arm ID "Robot_2", the number of path points "5", and the X / Y / Z coordinates of each path point (such as "100,200,50" corresponding to hexadecimal "64,C8,32") in the template data domain;

[0264] EtherNet / IP protocol (production beat control): fill the process ID "Assembly_2", the adjusted beat "18s" (corresponding to hexadecimal "12"), and the switching timing "01" (indicating after the current batch is completed) in the template data segment;

[0265] Check code calculation: calculate the check code according to the protocol requirements to ensure that the instruction transmission process is tamper-free and lossless:

[0266] Modbus protocol: CRC-16 check algorithm is used to calculate the binary data check value of address field, function code and data field, and fill it into the template check field;

[0267] Profinet protocol: CRC-32 check algorithm is used to calculate the check value of the entire message (device name + data identifier + data field), which is attached at the end of the message;

[0268] EtherNet / IP protocol: Based on the TCP / IP protocol stack check mechanism, calculate the IP header checksum and TCP checksum to ensure message integrity;

[0269] Instruction frame packaging: The protocol template filled with data field and check field is packaged into standard control instruction frame according to protocol standard format:

[0270] Modbus instruction frame format: [address field (1 byte) + function code (1 byte) + data field (N bytes) + CRC check code (2 bytes)];

[0271] Profinet instruction frame format: [message header (8 bytes, including version, length) + device name (16 bytes) + data identifier (2 bytes) + data field (M bytes) + CRC-32 check code (4 bytes)];

[0272] EtherNet / IP instruction frame format: [TCP header (20 bytes) + IP header (20 bytes) + CIP message header (8 bytes) + data segment (K bytes) + message tail (4 bytes)];

[0273] Instruction validity verification: The integrity and compliance of the packaged standard control instruction frame are verified, such as checking whether the frame length meets the protocol requirements (Modbus instruction frame is at least 8 bytes), whether the data field value is within the PLC recognizable range (such as Profinet path coordinate value does not exceed the maximum storage value of PLC register), and if the verification is passed, it enters the instruction issuing link, and if it fails, it returns to the original instruction frame filling link for reprocessing;

[0274] Instruction directional issuing: Through industrial Ethernet, the standard control instruction frame verified is directed to the corresponding production line PLC:

[0275] Device parameter adjustment instruction: issued to the local PLC corresponding to the target device (such as lathe #3);

[0276] Production rhythm control instruction: issued to the main PLC of the production line (such as assembly line main controller), and then issued to each process PLC by the main PLC;

[0277] Replan the path of the unloading robot: send the instruction to the robot's dedicated PLC (e.g. the motion controller of robot #2);

[0278] Receive the execution status: after the PLC receives the standard control instruction frame, analyze and execute the instruction, and feedback the execution status ("execution success", "execution failure", "execution in progress") to the control execution module. The feedback information includes PLC ID, instruction timestamp, and execution result code (e.g. "00H" for success, "01H" for parameter exceeding PLC control range);

[0279] Verify the closed-loop effect: if the PLC feedbacks "execution success", the control execution module obtains the production line data after the instruction execution through the data acquisition module (high-speed industrial camera array, vibration sensor, RFID reader):

[0280] After adjusting the device parameters: obtain the device vibration frequency / amplitude data (verify whether the device is stable after adjusting the spindle speed) and product image data (verify whether the product quality meets the standard after adjusting the parameters);

[0281] After production rhythm control: obtain RFID material flow data (verify whether the material flow is smooth after adjusting the rhythm) and production line balance rate related data (verify the rhythm optimization effect);

[0282] After robot path replanning: obtain robot motion trajectory data (verify whether the path is executed according to the instruction) and material grasping success rate data (verify the path optimization effect);

[0283] Archive the feedback results: package the instruction sending records (sending time, instruction content, target PLC), execution status feedback, and closed-loop effect verification data, and store them in the local database of the module. At the same time, feedback the verification results (e.g. "device parameter adjustment improves OEE by 2%" and "robot path optimization improves grasping success rate by 5%") to the optimization strategy generation module, which provides the basis for subsequent strategy iteration optimization, and completes the closed-loop control.

[0284] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A machine vision production line performance evaluation and optimization management system, characterized in that: Comprise: a data acquisition module comprising a high-speed industrial camera array, a vibration sensor and an RFID reader deployed on the production line for synchronously collecting image data of products, running state data of equipment and flow data of materials; a data processing and fusion module connected to the data acquisition module, comprising: an image processing unit for feature extraction of image data based on a deep learning convolutional neural network model to obtain product quality feature vectors; a data standardization unit for time alignment and normalization processing of running state data and flow data to generate standardized equipment state data; a multi-modal data fusion unit for fusing product quality feature vectors and standardized equipment state data using a fusion algorithm based on an attention mechanism to generate a production line fusion feature set; a dynamic performance evaluation module connected to the data processing and fusion module for inputting the production line fusion feature set into a trained performance evaluation model to calculate key performance indicators including at least equipment overall efficiency OEE, first pass yield FPY and production line balance rate in real time, comprising: real-time receiving of the production line fusion feature set transmitted by the data processing and fusion module; the production line fusion feature set is time-series data containing multi-dimensional fusion features corresponding to each timestamp; based on the production line beat and performance evaluation requirements, a fixed time window length is preset, the window sliding step and the feature set timestamp interval are kept synchronous to ensure no data omission or repetition; according to the preset time window, the continuous production line fusion feature set is cut by sliding; the integrity of the cut feature sequence is checked, and after the check passes, the feature sequence is output to the performance evaluation model calculation unit, and if the check fails, the data processing and fusion module is fed back to retransmit the fusion feature set of the corresponding period; the pre-trained performance evaluation model is loaded at startup, the performance evaluation model is based on a deep neural network architecture, and the loaded parameters include input layer dimension, number of hidden layers / neurons and output layer dimension; compliant feature sequences output by the feature sequence generation unit are received and flattened into a vector format acceptable by the model input layer; model forward propagation: the input vector is processed by each layer of the model in turn: input layer: receiving the flattened feature sequence vector, passing it to the first hidden layer; hidden layer: performing linear transformation and non-linear transformation through activation function to gradually extract deep features related to performance indicators; output layer: performing linear transformation to output a preliminary performance indicator vector with a dimension of 3, each element in the preliminary performance indicator vector is a normalized value in the range of [0, 1] corresponding to the preliminary prediction results of OEE, FPY and production line balance rate; preliminary indicator output: transmitting the preliminary performance indicator vector generated by the output layer to the indicator denormalization and analysis unit, and recording the model calculation timestamp to ensure time consistency with subsequent indicator analysis results; receiving the preliminary performance indicator vector output by the performance evaluation model calculation unit, and loading the indicator-maximum value mapping table saved during model training; Each normalized value in the preliminary performance indicator vector is mapped back to the actual physical range using the denormalization formula: wherein, is the denormalized normalized performance indicator value, is a certain element in the preliminary performance indicator vector, is the historical minimum value of the performance indicator for the model training dataset, is the historical maximum value of the performance indicator for the model training dataset. based on the dimension order of the standardized performance indicator values, the specific values of the three indicators are extracted from the mapping table; The parsed key performance indicators are output to an optimization strategy generation module in real time and stored in a local indicator log library for subsequent performance trend analysis and historical data tracing; The optimization strategy generation module is connected to the dynamic performance evaluation module and has a reinforcement learning agent built therein. The reinforcement learning agent takes the key performance indicators as state inputs of a reward function, learns an optimization action strategy by interacting with the environment, and the optimization action strategy includes device parameter adjustment instructions, production rhythm control instructions, or re-planning instructions for the path of a feeding mechanical arm. The control execution module is connected to the optimization strategy generation module and a programmable logic controller (PLC) on the production line, and is configured to convert the optimization action strategy into control instructions conforming to an industrial communication protocol and send the control instructions to corresponding execution units to form a closed-loop optimization control system.

2. The machine vision production line performance evaluation and optimization management system of claim 1, wherein: In the data acquisition module, the high-speed industrial camera array, the vibration sensor, and the RFID reader perform data acquisition through a synchronous triggering mechanism. The synchronous triggering mechanism is managed by a central synchronization controller, which generates a synchronization pulse signal to simultaneously trigger the high-speed industrial camera array to capture multi-angle image data of the product, the vibration sensor to collect device vibration frequency and amplitude data, and the RFID reader to read material RFID tag data, ensuring that all collected data have consistent timestamps. The data acquisition module also packages the collected image data, running state data, and flow data into time series data packets and transmits them to the data processing and fusion module through an industrial Ethernet.

3. The machine vision production line performance evaluation and optimization management system of claim 1, wherein: The image processing unit extracts features from the image data based on a deep learning convolutional neural network model to obtain a product quality feature vector, including: The collected image data is preprocessed, including image denoising, size adjustment, and pixel value normalization, to generate standardized image data; The standardized image data is input into a pre-trained deep learning convolutional neural network model, which is processed through multiple convolution layers, activation function layers, and pooling layers to generate multi-level feature maps; The multi-level feature maps are processed for feature fusion, high-level semantic features are extracted through a fully connected layer or a global pooling layer in the convolutional neural network, and an initial feature vector is generated; The initial feature vector is processed for dimension reduction and standardization to obtain a final feature vector representing product quality.

4. The machine vision production line performance evaluation and optimization management system of claim 1, wherein: The data standardization unit is configured to perform time series alignment and normalization processing on the running state data and the flow data to generate standardized device state data, including: The running state data and the flow data are received from the data acquisition module, where the running state data includes device vibration frequency and amplitude data, and the flow data includes material RFID tag data; The timestamps of the running state data and the flow data are analyzed to extract the original time information of each data point; Based on the synchronization time reference provided by the central synchronization controller, the running state data and the flow data after timestamp analysis are processed for time series alignment, and the data is resampled onto a unified time series through linear interpolation to ensure that all data points have consistent time intervals; The running state data and the flow data after time alignment are normalized, a minimum-maximum normalization algorithm is adopted, data values are mapped to a range of [0, 1], and normalized data is generated; The normalized data is combined into standardized equipment state data and output to a multi-modal data fusion unit.

5. The machine vision production line performance evaluation and optimization management system, according to claim 1, wherein: The multi-modal data fusion unit adopts a fusion algorithm based on an attention mechanism to fuse the product quality feature vector and the standardized equipment state data, and generates a production line fusion feature set, including: receiving the product quality feature vector from the image processing unit and the standardized equipment state data from the data standardization unit; inputting the product quality feature vector and the standardized equipment state data into a linear mapping layer respectively to generate corresponding query vectors, key vectors and value vectors; calculating the dot product similarity between the query vectors and the key vectors, and performing softmax normalization processing on the similarity to generate attention weights; performing weighted summation operation on the value vectors using the attention weights to generate weighted context feature vectors; splicing and combining the weighted context feature vectors and the product quality feature vector to generate a preliminary fusion feature vector; performing full connection layer processing and activation function transformation on the preliminary fusion feature vector to generate a final production line fusion feature set.

6. The machine vision production line performance evaluation and optimization management system, according to claim 1, wherein: The optimization strategy generation module has a reinforcement learning agent built-in, the reinforcement learning agent takes key performance indicators as state inputs of a reward function, learns output optimization action strategies by interacting with the environment, the optimization action strategies include device parameter adjustment instructions, production rhythm control instructions or feeding mechanical arm path re-planning instructions, including: receiving key performance indicators from the dynamic performance evaluation module; inputting the key performance indicators as the current environment state into the state space of the reinforcement learning agent; the reinforcement learning agent calculates the action value function through the policy network based on the current environment state to generate a candidate action set; performing exploration and utilization balancing processing on the candidate action set, and selecting the optimal action using an ε-greedy strategy; mapping the optimal action into an original action strategy, including device parameter adjustment amount, production rhythm adjustment value or mechanical arm path point sequence; performing compliance checking on the original action strategy to ensure that it meets the production line safety constraints and operating range; encoding the compliant original action strategy into specific device parameter adjustment instructions, production rhythm control instructions or feeding mechanical arm path re-planning instructions; outputting the optimization action strategy to the control execution module.

7. The machine vision production line performance evaluation and optimization management system, according to claim 1, wherein: The control execution module is connected to the optimization strategy generation module and the programmable logic controller (PLC) on the production line, used to convert the optimization action strategy into control instructions conforming to the industrial communication protocol, and issue them to the corresponding execution unit, including: receiving the optimization action strategy from the optimization strategy generation module, the optimization action strategy including device parameter adjustment instructions, production rhythm control instructions or feeding mechanical arm path re-planning instructions; performing parsing processing on the received optimization action strategy to extract the action type and specific parameter value; According to the extracted action type, a corresponding industrial communication protocol template is selected, the industrial communication protocol template including Modbus, Profinet or EtherNet / IP; Specific parameter values extracted are filled into data fields of the selected industrial communication protocol template to generate an original control instruction frame; The original control instruction frame is subjected to check code calculation and packaging processing to generate a standard control instruction conforming to the industrial communication protocol; The standard control instruction is issued to a programmable logic controller PLC on a production line through industrial Ethernet; After the programmable logic controller PLC parses and executes the standard control instruction, a control execution module acquires production line data after execution through a data acquisition module to verify instruction execution effect and constitute a closed-loop optimization control system.

Images