A method and system for real-time monitoring of web warp in a molding press

Abstract

The present application relates to the technical field of industrial automation and intelligent manufacturing, in particular to a kind of moulding press sheet warping real-time monitoring method and system.The method is aimed at the problems of material clamping and out-of-track caused by sheet warping in the prior art, by deploying diagonally arranged inductors, image acquisition devices and environmental data acquisition modules, real-time acquisition of sheet position, image and production environment data;Further, the pre-trained deep learning model is used to analyze the fusion of multi-source data, and the warping degree evaluation value and the out-of-track risk prediction value of the sheet are output;When the risk value exceeds the threshold value, the motion parameters of the moulding press actuator are adaptively adjusted.The corresponding system integrates the above hardware and processor to achieve efficient collaboration.The present application overcomes the shortcomings of traditional single-point detection, significantly improves the monitoring accuracy and reliability through intelligent prediction and dynamic control, effectively prevents production abnormalities and optimizes equipment operating efficiency.

Description

Technical Field

[0001] This invention relates to the field of technical fields, and specifically to a method and system for real-time monitoring of warpage of molding machine sheet. Background Technology

[0002] As a key piece of equipment in electronic packaging and semiconductor manufacturing, the stability and reliability of molding machines directly affect product quality and production efficiency. In the molding process, the wafers need to be precisely delivered to designated positions for pressing; any deviation can lead to jamming, incomplete filling, or equipment damage. Traditional technologies commonly use a single sensor to detect wafer positioning. This method is based on a simple trigger mechanism: when the sensor detects the wafer, it is considered positioned correctly, initiating subsequent operations. However, this single-sensor design has significant drawbacks: it cannot effectively handle wafer warpage caused by material properties, temperature and humidity changes, or mechanical stress. Wafer warpage, especially localized deformation introduced by the sampling process, can cause the wafer to deviate from its intended path during transport, failing to fully cover the sensor's detection area and leading to misjudgments. More seriously, warped wafers may break through track limiting devices (such as stoppers) at high speeds, causing jamming or mechanical collisions, affecting not only the current production run but also subsequent workstations, creating a chain reaction of abnormalities. Although industry efforts have attempted to improve detection reliability by modifying mechanical structures or increasing hardware redundancy (such as diagonally arranging dual sensors), these solutions remain limited to static or regular judgments and cannot adapt to dynamic changes in warping in real time. For example, while dual-sensor designs can reduce false negative rates through two-point detection, their response to asymmetric warping or slowly forming deformations remains lagging, and they lack the ability to predict potential risks. Therefore, existing technologies urgently need a monitoring solution that can deeply integrate real-time data and possess intelligent analysis and prediction capabilities to fundamentally address the challenges posed by warping.

[0003] Therefore, existing technologies still need further development. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a method and system for real-time monitoring of warpage of molding machine sheet, so as to solve the problems existing in the prior art.

[0005] To achieve the above-mentioned technical objectives, according to a first aspect of the present invention, the present invention provides a method for real-time monitoring of warpage of molding machine sheet, comprising: S1. At least two sensors are used to detect in real time whether the sheet material arrives at the diagonal position of the sheet material conveying track of the molding machine at the same time. The sensors are arranged diagonally to cover the sheet material area. S2. Image data of the material sheet is continuously acquired through an image acquisition device, and production environment data is continuously acquired through an environmental data acquisition module. The production environment data includes temperature, humidity and vibration parameters. S3. Input the detection signal of the sensor, the image data, and the production environment data into a pre-trained deep learning model; the deep learning model analyzes the input data in real time and outputs the warpage assessment value and the punching risk prediction value of the sheet; when the punching risk prediction value exceeds the preset risk threshold, generate control commands to adjust the motion parameters of the actuator in the molding press to achieve adaptive control.

[0006] Specifically, the step of continuously acquiring image data of the material sheet through an image acquisition device and simultaneously continuously acquiring production environment data through an environmental data acquisition module includes: The image acquisition device is an industrial camera, which is installed above or to the side of the sheet conveying track. The shooting direction is perpendicular to the sheet plane, and the shooting frequency is synchronized with the working rhythm of the molding machine to ensure the acquisition of continuous sheet movement sequence images. The environmental data acquisition module includes a temperature sensor, a humidity sensor, and a vibration sensor, which are respectively arranged at key positions around the molding machine to collect production environment parameters that affect sheet warping in real time. The sensor is a photoelectric sensor or a proximity switch, which is arranged at the two diagonal positions at both ends of the sheet conveying track, with its sensing surface facing the sheet feeding direction.

[0007] Specifically, before inputting data into a deep learning model, a data preprocessing step is also included: The sensor's detection signal is denoised and time-aligned to eliminate false triggering and signal delay; the image data is grayscaled, denoised, and normalized to eliminate illumination variations and image distortion; the production environment data is filtered and standardized to eliminate noise and dimensional differences, thereby generating a unified input data format for use by the deep learning model.

[0008] Specifically, the preprocessed sensor detection signal, the image data, and the production environment data are fused using multimodal data fusion, which includes: Visual features, including material outline features, edge features, and texture features, are extracted from the image data; temporal features, including temperature change rate, humidity fluctuation, and vibration spectrum, are extracted from the production environment data; the visual features, temporal features, and the detection signal of the sensor are fused at the feature level to generate a multi-dimensional feature vector containing multi-dimensional information, which is used as the input of the deep learning model.

[0009] Specifically, the deep learning model is a convolutional neural network, which includes an input layer, multiple convolutional layers, pooling layers, fully connected layers, and an output layer. The input layer receives the multidimensional feature vector, the convolutional and pooling layers are used to extract high-level features layer by layer, the fully connected layer is used for feature integration, and the output layer outputs the warpage assessment value and the runaway risk prediction value. The warpage assessment value is a classification label or regression value, representing the warpage level or specific warpage amount of the material, and the runaway risk prediction value is a probability value, representing the possibility of the material running off the track.

[0010] Specifically, the deep learning model is trained using historical production data, which includes normal sheet conveying cases and abnormal ejection cases. Each case includes sensor signals, image data, environmental data, and corresponding warpage and risk labels. During training, the model is optimized using a cross-entropy loss function or a mean squared error loss function, and the model parameters are updated using a stochastic gradient descent algorithm or an adaptive moment estimation algorithm until the model converges to a preset accuracy.

[0011] Specifically, when the predicted risk value exceeds the preset risk threshold, a control command is generated, including adjusting the movement speed, acceleration, or stopping position of the actuator to achieve adaptive control. The preset risk threshold is dynamically adjusted according to the production batch, specifically calculated by statistical methods based on the material characteristics, environmental conditions, or historical anomaly rate of the current batch of material.

[0012] Specifically, this also includes real-time monitoring steps: The prediction performance of the deep learning model is monitored in real time. The prediction error is calculated by comparing the predicted value with the actual result. When the prediction error continues to exceed the tolerance, the online model update mechanism is triggered. The model is incrementally learned using the current production data to maintain its adaptability. The incremental learning adopts an online learning algorithm to update some parameters without retraining the entire model.

[0013] Specifically, it also includes emergency response procedures: When the sensor detects that the material pieces are not in place simultaneously, it directly triggers an emergency stop command, suspends the operation of the molding machine, and issues an audible and visual alarm to prompt manual intervention; at the same time, it records the time, location, and sensor data of the abnormal event for subsequent analysis and model optimization.

[0014] According to a second aspect of the present invention, a real-time monitoring system for warpage of molding sheet is provided, comprising: At least two sensors are arranged at opposite positions on the sheet conveying track of the molding machine to detect in real time whether the sheets arrive at the same time. An image acquisition device is installed near the sheet conveying track to continuously acquire image data of the sheet; The environmental data acquisition module includes a temperature sensor, a humidity sensor, and a vibration sensor, used to continuously acquire production environment data; The processor is connected to the sensor, image acquisition device, and environmental data acquisition module. And a controller, connected to the processor, for adjusting the motion parameters of the actuator in the molding machine according to the control instructions generated by the processor.

[0015] Beneficial effects: This invention, by introducing deep learning technology and multimodal data fusion, achieves intelligent and predictive monitoring of warpage of molding machine sheet, bringing many significant benefits.

[0016] Firstly, regarding detection reliability, this invention surpasses the limitations of traditional single or dual-sensor systems that rely on fixed thresholds for judgment. By combining visual, environmental, and positional signals, it constructs a comprehensive monitoring network. The deep learning model can learn the complex correlation between warpage and various factors (such as temperature fluctuations and vibration spectra) from historical data, thereby identifying risk signs in the early stages of slight deformation of the material, greatly improving the accuracy and timeliness of anomaly identification. This capability allows the system not only to determine whether the material is "in place" but also to quantify its "postural stability," effectively avoiding missed detections or false alarms caused by warpage.

[0017] Secondly, in terms of control adaptability, this invention achieves a leap from passive response to active intervention. Based on real-time risk prediction values, the system dynamically adjusts actuator parameters (such as movement speed or stopping position) to form a flexible adaptive control strategy. This reduces the mechanical shock and production interruption caused by traditional emergency stops, making equipment operation smoother. At the same time, preventive measures reduce the probability of material jamming and ejection events, thereby improving overall production efficiency and equipment lifespan.

[0018] Furthermore, the multi-task learning architecture of this invention allows for the simultaneous assessment of warpage and runout risk, providing operators with richer decision support and facilitating quality traceability and process optimization.

[0019] Furthermore, the system's real-time monitoring and incremental learning mechanism ensures long-term applicability and can automatically adapt to changes in the production environment (such as changes in raw material batches or seasonal effects), avoiding the problem of model performance degradation.

[0020] Finally, by combining emergency response with intelligent analysis, this invention optimizes the data loop and promotes continuous improvement while ensuring safety redundancy.

[0021] Overall, these effects collectively contribute to the intelligent upgrading of the production process, significantly reducing maintenance costs and improving product quality consistency. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the real-time monitoring method for warpage of molding machine sheet provided in a specific embodiment of the present invention; Figure 2 This is a schematic diagram of the system composition of the real-time monitoring system for warpage of molding machine sheet provided in a specific embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments in this application, other similar embodiments obtained by those skilled in the art without creative effort should all fall within the scope of protection of this application. Furthermore, directional terms mentioned in the following embodiments, such as "up," "down," "left," and "right," are only for reference to the directions in the accompanying drawings; therefore, the directional terms used are for illustrative purposes and not for limiting the invention.

[0024] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments.

[0025] Please see Figure 1 This invention provides a method for real-time monitoring of warpage in molding machine sheet materials, comprising: S1. At least two sensors are used to detect in real time whether the sheet material arrives at the diagonal position of the sheet material conveying track of the molding machine at the same time. The sensors are arranged diagonally to cover the sheet material area.

[0026] It should be further explained that the sensors are two diffuse reflection photoelectric sensors (preferably Sick WTT12-2P2431) with a response time of less than 5 milliseconds. They are rigidly mounted at two opposite corners of the inlet end plane of the sheet conveying track, with the mounting points 10 mm from the edge of the track. The length of the line connecting the center points of the sensing surfaces of the two sensors is equal to the diagonal length of the sheet being produced. For example, for a 100mm x 150mm rectangular sheet, this distance is set to 180.28 mm to ensure that it can only be triggered simultaneously when the sheet completely covers the diagonal area of ​​the track.

[0027] S2. Image data of the material sheet is continuously acquired through an image acquisition device, and production environment data is continuously acquired through an environmental data acquisition module. The production environment data includes temperature, humidity and vibration parameters.

[0028] It should be further explained that the image acquisition device is a 5-megapixel industrial area scan camera (preferably a Baslerac A2500-20gc), equipped with a 12mm fixed-focus lens. This camera is mounted 800mm directly above the track, with its optical axis perpendicular to the material plane. It is synchronized with the main control PLC of the molding machine via an external trigger signal, capturing grayscale images of the material at a frequency of 25 frames per second. Third, the environmental data acquisition module includes a digital temperature sensor (model DS18B20, measurement range -55°C to +125°C, accuracy ±0.5°C), mounted 5cm from the heating plate of the molding machine; a digital humidity sensor (model SHT31, measurement range 0-100%RH, accuracy ±2%RH), mounted at the workshop ventilation opening; and a triaxial accelerometer (model ADXL345, measurement range ±16g), mounted on the drive pulley bearing of the material conveyor belt. All sensor data is uploaded via an RS485 bus at a sampling frequency of 10Hz.

[0029] S3. Input the detection signal of the sensor, the image data, and the production environment data into a pre-trained deep learning model; the deep learning model analyzes the input data in real time and outputs the warpage assessment value and the punching risk prediction value of the sheet; when the punching risk prediction value exceeds the preset risk threshold, generate control commands to adjust the motion parameters of the actuator in the molding press to achieve adaptive control.

[0030] It should be further explained that the deep learning model is a convolutional neural network implemented based on the PyTorch framework. Its input is a preprocessed 152-dimensional feature vector, and its output is a three-class warp severity assessment value (0: no warp, 1: slight warp, 2: severe warp) and a predicted probability value for the risk of expulsion between 0 and 1. The preferred value of the preset risk threshold is 0.82. This value was determined using a grid search method, aiming to maximize the F1 score (reaching 0.93), by analyzing the predicted probability distribution of 500 abnormal expulsion events and 5000 normal events in the past six months of production data. When the predicted risk value exceeds this threshold, the generated control command is sent to the servo drive through the PLC's PROFINET interface, linearly reducing the shifter's downward speed from the default 120 mm / s to (120 mm / s). (1-risk prediction value) mm / s, and an emergency stop is initiated when the risk prediction value is greater than 0.95.

[0031] Understandably, this method creatively upgrades simple two-point presence detection into intelligent, predictive monitoring that integrates multi-dimensional physical information. It not only inherits the basic judgment logic of "simultaneous arrival" from the original dual-sensor design, solving the problem of potential missed detection by a single signal source, but also, for the first time, achieves a quantitative assessment of the material's "posture" (degree of warping) by introducing image and temporal environmental data. The introduction of a deep learning model enables the system to learn complex nonlinear relationships in historical data (such as the fact that materials with certain contour features are more prone to warping under specific humidity and vibration pattern combinations), thereby making risk predictions far exceeding preset rules. Ultimately, dynamic parameter adjustments based on predicted risks achieve a fundamental shift from "downtime after a failure" to "pre-risk intervention," significantly improving quality control while ensuring production continuity.

[0032] Specifically, the continuous acquisition of image data of the material sheet via an image acquisition device and the continuous acquisition of production environment data via an environmental data acquisition module include: the image acquisition device being an industrial camera installed above or to the side of the material sheet conveying track, with the shooting direction perpendicular to the material sheet plane and the shooting frequency synchronized with the working rhythm of the molding machine to ensure the acquisition of continuous material sheet movement sequence images; the environmental data acquisition module including a temperature sensor, a humidity sensor, and a vibration sensor, respectively arranged at key positions around the molding machine, for real-time acquisition of production environment parameters affecting material sheet warping; and the sensors being photoelectric sensors or proximity switches arranged at diagonal positions at both ends of the material sheet conveying track, with their sensing surfaces facing the material sheet infeed direction.

[0033] Further explanation is needed regarding the image acquisition process: The industrial camera is mounted on a 3D adjustable bracket, and calibrated using a laser level to ensure that the angle between its optical axis and the normal to the material plane is less than 1 degree, thus eliminating perspective distortion. The camera exposure time is set to 2000 microseconds. Under typical workshop lighting (500 Lux), this setting ensures that the grayscale value of the material image is distributed between 80-180, with moderate contrast. The trigger synchronization signal comes from the PLC's I / O module. After a 50-millisecond delay following the initial sensor trigger, the camera begins to continuously capture 10 frames at 25fps, fully covering the entire process from the material entering the field of view to being pressed down by the shifter. The environmental data acquisition process is as follows: A temperature sensor is installed close to the lower mold plate of the molding machine to monitor the actual working temperature of the mold; a humidity sensor is placed in an open position 1 meter away from the track and 1.5 meters high to monitor the workshop's ambient humidity; a vibration sensor is magnetically attached to the drive motor housing of the conveyor belt, collecting vibration signals along the belt's movement direction (X-axis) and vertically (Z-axis) to monitor imbalances or wear in the transmission system. The specific installation requirements for the sensor are that its sensing distance is set to 15 mm, the minimum diameter of the object to be detected is 2 mm, its signal line is connected to the high-speed counter module of the PLC, and the PLC program is set to a 10-millisecond judgment window. Only when the rising edge signals of the two sensors arrive at the same time within this window is it determined to be "simultaneously in place".

[0034] Understandably, this highly specific deployment and parameter configuration ensures the accuracy, synchronization, and engineering practicality of data acquisition. Images perpendicular to the material sheet avoid shape misjudgments caused by viewing angle, and synchronized shooting with the production cycle guarantees data validity and economical processing. The sensors are positioned in physically direct "critical locations" reflecting the process status, ensuring a clear causal or strong correlation between the acquired environmental parameters and the material sheet warpage. This provides a high-quality, high signal-to-noise ratio input data foundation for subsequent deep learning models, a crucial prerequisite for the successful application of artificial intelligence in industrial scenarios.

[0035] Specifically, before inputting the data into the deep learning model, a data preprocessing step is also included: denoising and timing alignment of the sensor's detection signal to eliminate false triggering and signal delay; grayscale conversion, noise reduction, and size normalization of the image data to eliminate illumination variations and image distortion; and filtering and standardization of the production environment data to eliminate noise and dimensional differences, thereby generating a unified input data format for use by the deep learning model.

[0036] It should be further explained that the sensor signal preprocessing is as follows: The raw switching signal received by the PLC first undergoes a software anti-jitter filtering algorithm. This algorithm requires the signal to remain stable for more than 5 milliseconds before being acknowledged, in order to filter out glitches caused by electromagnetic interference. Timing alignment is achieved by adding high-precision timestamps (using the PLC's 1-microsecond resolution system clock) to the two sensor signals. The preferred alignment window value is 15 milliseconds. This is based on the maximum possible time difference (180 / 300=0.6 seconds) calculated from the distance between the two sensor installation positions (maximum 180 mm) and the maximum conveying speed of the material (300 mm / s), multiplied by a safety factor of 25 (0.6). The time (0.025 = 0.015 seconds) is determined to ensure a reasonable alignment tolerance. Image preprocessing specifically involves: First, convert the color image to grayscale using the formula. ,in Grayscale value , , These represent the pixel values ​​for the red, green, and blue channels, respectively. This coefficient reflects the human eye's highest sensitivity to green light. Then, a 3x3 Gaussian kernel is used to denoise the image, with a standard deviation of [missing value]. Setting it to 0.8 will smooth out noise while preserving edges relatively well.

[0037] Next, all image sizes were normalized to 224 pixels x 224 pixels using bilinear interpolation to accommodate the standard input size of subsequent CNN backbone networks (such as ResNet). Environmental data preprocessing specifically involved applying a first-order low-pass digital filter (transfer function: ...) to the temperature and humidity data. ) Filter out high-frequency fluctuations, among which It is the transfer function of the filter in the Z-domain. It is a unit delay operator with a cutoff frequency approximately one-tenth of the sampling frequency. For vibration data, it is first zero-mean normalized, and then its root mean square (RMS) value within a one-second window is calculated as a feature.

[0038] Finally, Z-score normalization was performed on all numerical environmental features: ,in It is the standardized value. It is the original value. It is the mean of this feature on the training set. It is the standard deviation of the feature on the training set, in order to eliminate the influence of different sensor dimensions and orders of magnitude.

[0039] Understandably, this complete preprocessing workflow is a "refinement" process for multi-source heterogeneous data from industrial sites. It's not simply a matter of piling on technologies, but rather a targeted processing solution designed for the physical characteristics and noise sources (such as electrical noise, lighting variations, and mechanical vibrations) of each data source. These steps significantly improve the quality of the input data, transforming raw, coarse physical signals into clean, well-organized data representations that can be effectively learned by the model. This is a crucial step in ensuring the stable and reliable operation of deep learning models in industrial applications, effectively avoiding the problem of "garbage in, garbage out."

[0040] Specifically, the preprocessed sensor detection signal, image data, and production environment data are fused using multimodal data fusion. This includes: extracting visual features from the image data, including material outline features, edge features, and texture features; extracting temporal features from the production environment data, including temperature change rate, humidity fluctuation, and vibration spectrum; and fusing the visual features, temporal features, and sensor detection signal at the feature level to generate a multidimensional feature vector containing multidimensional information, which is then used as input to a deep learning model.

[0041] It should be further explained that visual feature extraction uses a lightweight preprocessing CNN (such as the first 10 layers of MobileNetV2 with frozen weights) to perform forward propagation on the normalized 224x224 image, extracting a 128-dimensional high-level semantic feature vector. This vector encodes the contour, shape, and texture information of the material. The specific algorithm for temporal feature extraction is as follows: (1) For temperature, calculate the linear regression slope of the most recent 5 sampling points (i.e., within 0.5 seconds) as the rate of temperature change. The formula is ,in It is the i-th time point. It is the i-th temperature value. and These are the average values ​​of time and temperature, respectively.

[0042] (2) For humidity, calculate its standard deviation within the most recent 10-second window. .

[0043] (3) For vibration, perform a 1024-point FFT transform on the acceleration signal of each axis and extract the top 5 frequency components with the largest amplitude in the 0-100Hz frequency band. arrive and its corresponding amplitude arrive As a spectral feature, the sensor signal is directly treated as a 2-dimensional Boolean vector (e.g., [1,1] indicates that both sensors are triggered).

[0044] Furthermore, the final fusion is performed in a fusion layer, which combines 128-dimensional image features and 8-dimensional temporal features (1-dimensional...). 1D 5D 1D ,in The average amplitude of the main frequency components and the 2D sensor features are concatenated to form a 138-dimensional feature vector. For dimensionality reduction and decorrelation, this vector is then projected to 64 dimensions through a fully connected layer (with 64 neurons, using ReLU activation), which is the final multidimensional feature vector input to the downstream deep learning classifier / regressor.

[0045] Understandably, this multimodal fusion strategy is not a simple data splicing, but rather follows the principle of "taking the best from each and abstracting in layers." Image CNNs excel at extracting spatial morphological features, while manually calculated temporal features clearly express the changing trends and spectral characteristics of physical quantities, and sensor signals provide the most direct markers of motion events. Fusion at the feature level enables subsequent models to learn the complex interactions between different modal features (for example, a specific texture feature, combined with a specific vibration spectrum in a high-temperature and high-humidity environment, may indicate a higher risk of warping). This cross-modal correlation mining is impossible to achieve with a single sensor or simple rule-based judgment, and constitutes the core of the intelligence of this invention.

[0046] Specifically, the deep learning model is a convolutional neural network, which includes an input layer, multiple convolutional layers, pooling layers, fully connected layers, and an output layer. The input layer receives the multidimensional feature vector, the convolutional and pooling layers are used to extract high-level features layer by layer, the fully connected layer is used for feature integration, and the output layer outputs the warpage assessment value and the runaway risk prediction value. The warpage assessment value is a classification label or regression value, representing the warpage level or specific warpage amount of the material, and the runaway risk prediction value is a probability value, representing the possibility of the material running off the track.

[0047] It should be further explained that the specific architecture of the deep learning model is as follows: (1) Input layer: Receives a multidimensional feature vector with dimensions [1, 64], which is reshaped into an 8x8 “feature map” to be input to the convolutional layer.

[0048] (2) Convolution and Pooling: The first convolutional layer uses 16 3x3 convolutional kernels with a stride of 1 and "same" padding, followed by a ReLU activation function and a 2x2 max pooling layer. The second convolutional layer uses 32 3x3 convolutional kernels, with the same structure as above. After two layers of convolution and pooling, the feature map is flattened into a one-dimensional vector.

[0049] (3) Fully connected layer: The first fully connected layer has 128 neurons, using ReLU activation and Dropout regularization (dropout rate set to 0.3 to prevent overfitting). The second fully connected layer has 64 neurons, also using ReLU activation.

[0050] (4) Dual-task output layer: This is a multi-task learning architecture: For the warp assessment task (classification), the output layer uses the Softmax activation function and outputs a 3D vector. ,in , , These represent the predicted probabilities of belonging to the three categories of "no warping", "mild warping", and "severe warping", respectively. The final evaluation value is the category index with the highest probability.

[0051] For escape risk prediction tasks (regression), the output layer uses the Sigmoid activation function to output a scalar value. , This represents the probability of the model escaping. During model training, the total loss function... It is classification loss and regression loss Weighted sum: ,in It is a trade-off parameter, with an optimal value of 0.7, because the accuracy of the classification task is more fundamental for subsequent risk threshold judgment. Using cross-entropy loss, Mean squared error loss is used.

[0052] Understandably, a major advantage of this implementation method is its use of a shared low-level feature extraction architecture with top-level branches handling different tasks. This allows the two tasks (classification and regression) to share the common feature representations learned from the fused data during model training, mutually reinforcing each other and resulting in higher efficiency and stronger generalization ability compared to training two independent models. Simultaneously, separating the warp degree classification and risk probability regression outputs provides operators with richer decision-making information: classification results can be used for statistical analysis and process parameter tracking, while continuous risk probability values ​​enable more refined and sensitive gradient control (such as linearly adjusting speed based on risk values), avoiding the abrupt changes in action and mechanical shocks caused by binary "either / or" control.

[0053] Specifically, the deep learning model is trained using historical production data, which includes normal sheet conveying cases and abnormal ejection cases. Each case includes sensor signals, image data, environmental data, and corresponding warpage and risk labels. During training, the model is optimized using a cross-entropy loss function or a mean squared error loss function, and the model parameters are updated using a stochastic gradient descent algorithm or an adaptive moment estimation algorithm until the model converges to a preset accuracy.

[0054] Further explanation is needed regarding the specific process of constructing the historical dataset: Production cycle data from at least 10,000 pieces of material were collected over three consecutive months on the production line. Each case's data package included: the trigger timing of two sensors (accurate to milliseconds), a sequence of grayscale images (224x224) of the material taken from the trigger moment forward for 10 frames, a sampling sequence of temperature, humidity, and vibration within the same time period (10Hz, lasting 1 second), and two manually labeled tags. The warpage label was determined by quality inspectors offline based on post-production observation of the finished material or through high-precision laser scanning data, and was categorized into three levels: 0, 1, and 2. The risk label was a binary label; 1 represented that the material actually experienced punching or jamming abnormalities in subsequent molding processes, while 0 represented normal passage. To ensure data balance, oversampling was used to increase the proportion of abnormal cases in the training set, making the ratio of normal to abnormal cases approximately 3:1. Model training used the Adam optimizer, with parameters set as follows: initial learning rate... First-order moment estimation of attenuation rate Second-order moment estimation of attenuation rate The training is conducted in two phases: In the first stage, the weights of the convolutional layers are frozen, and only the fully connected layers are trained for 50 epochs. In the second stage, all network layers are unfrozen, end-to-end fine-tuning is performed, the learning rate is reduced to 0.0001, and then the network is trained for another 100 epochs.

[0055] Furthermore, during training, the macro-average F1 score on the validation set (representing 15% of the total data), a comprehensive indicator that examines both classification and regression tasks, is used as a monitoring metric for early stopping. If the score does not improve for 10 consecutive epochs, training is stopped and the best model is saved. The converged model is considered deployable only if it achieves a warped classification accuracy of over 95% on the independent test set (representing 15% of the total data) and an AUC (area under the curve) of overshoot risk prediction of over 0.97.

[0056] Understandably, this detailed and rigorous data construction and training scheme is the cornerstone of ensuring the model's industrial-grade reliability. It clearly defines data sources, annotation standards, balancing methods, optimizer selection and parameters, training phase strategies, and final acceptance metrics. This enables those skilled in the art to fully reproduce the model's training process, avoiding problems such as model performance instability or inoperability caused by unclear training details. In particular, the two-stage training and early stopping strategies are effective techniques designed to address the potentially limited amount of industrial data and the need to avoid overfitting, ensuring the generalization ability and robustness of the final deployed model.

[0057] Specifically, when the predicted risk value exceeds the preset risk threshold, a control command is generated, including adjusting the movement speed, acceleration, or stopping position of the actuator to achieve adaptive control. The preset risk threshold is dynamically adjusted according to the production batch, specifically calculated by statistical methods based on the material characteristics, environmental conditions, or historical anomaly rate of the current batch of material.

[0058] It should be further explained that the specific algorithm for generating control commands is as follows: Let the default shifter pressing speed be... (e.g., 120 mm / s), acceleration is (e.g., 500 mm / s²), the standard stop position is The current predicted risk value is Preset risk threshold (Initial value is 0.82). If Then, an adjustment instruction is generated, with the specific design as follows: (1) Speed ​​adjustment: ,in This is the velocity attenuation coefficient, with a preferred value of 1.5. When Below the safety limit (When set to 30mm / s) take .

[0059] (2) Acceleration adjustment: , It is the acceleration attenuation coefficient, with a preferred value of 1.0.

[0060] (3) Position fine-tuning: To prevent inaccurate positioning of the material sheet due to warping, the centroid offset is calculated based on the risk value and the material sheet profile. Compensation is performed on the stopping position: ,in This is the compensation gain, with a preferred value of 0.8. The threshold value... The dynamic adjustment is based on the Exponentially Weighted Moving Average (EWMA) method: after each production batch (e.g., every 2000 pieces), the actual anomaly rate of that batch is calculated. (Number of anomalies / Total number of films) and the model's average risk prediction for this batch. Then update the threshold: ,in The base threshold is 0.82. It is the target anomaly rate (e.g., 0.1%). It is an adjustment coefficient (preferred value is 0.1). This is a smoothing factor (preferred value is 0.8). This formula allows the threshold to be automatically tightened or loosened based on actual production performance; if the recent abnormality rate is high, the threshold will be slightly lowered. This makes the control system more sensitive; conversely, it raises the threshold to avoid excessive intervention that could affect efficiency.

[0061] Understandably, this risk-based, continuously adjustable control strategy enables precise "dose-response" intervention, rather than a simple "one-size-fits-all" shutdown. It not only considers whether the risk exceeds limits but also determines the intensity of control actions based on the degree of exceedance, making system behavior smoother and more intelligent. The dynamic threshold adjustment mechanism endows the system with self-learning and adaptive capabilities, enabling it to adapt to factors such as raw material fluctuations, equipment aging, and seasonal environmental changes over the long term, always maintaining an optimal risk control state—a superior advantage that fixed threshold schemes cannot match.

[0062] Specifically, it also includes a real-time monitoring step: real-time monitoring of the prediction performance of the deep learning model, calculating the prediction error by comparing the predicted value with the actual result, and triggering an online model update mechanism when the prediction error continues to exceed the tolerance, using the current production data to incrementally learn the model in order to maintain the model's adaptability; the incremental learning adopts an online learning algorithm to update some parameters without retraining the entire model.

[0063] It should be further explained that the specific implementation of real-time monitoring is as follows: During system operation, "real results" are continuously collected. For warpage classification, real results come from periodic offline sampling (e.g., sampling 1 piece out of every 100 pieces, using a laser height gauge for precise measurement); for risk labels, real results are whether the piece has actually been ejected (based on an emergency stop signal from the PLC or operator confirmation records). The model performance index is calculated once every N new samples collected (the preferred value for N is 50). For classification tasks, the proportion of predicted categories that do not match the true categories is calculated as the error rate. For regression tasks, calculate the predicted risk value. With the binary label of reality Cross-entropy loss between (0 or 1) Set tolerance values and If three consecutive evaluation cycles (i.e., after a cumulative total of 150 new samples) are completed... or This triggers a model update. The incremental learning process is as follows: (1) Fix the weights of all preceding convolutional and fully connected layers in the model.

[0064] (2) Only update the weights of the last fully connected layer (i.e. the 64-dimensional layer before the dual-task output layer) and the output layer.

[0065] (3) Use the M latest samples accumulated when the update is triggered (the preferred value of M is 200, which includes samples with normal historical performance and poor recent performance) to form a small batch dataset.

[0066] (4) The Stochastic Gradient Descent (SGD) algorithm is used for updating, with a learning rate set to a small value, such as 0.00001, to avoid overfitting on new data or forgetting old knowledge. (5) After iteratively updating for 10 epochs, the performance is evaluated on a small, retained validation set. If the performance recovers, the update is stopped, and the new weights replace the old weights; if the performance does not recover, an alarm is issued, indicating that a complete offline model retraining may be necessary. This update process is performed asynchronously in the background thread of the system and does not affect the real-time monitoring and control in the foreground.

[0067] Understandably, this real-time monitoring and incremental learning mechanism acts as an "immune system" to ensure the long-term stable operation of the system. Conceptual drift in industrial environments (such as changes in raw material suppliers or equipment component wear) is inevitable, and fixed models will gradually become ineffective. This invention, through clearly defined performance indicators, triggering conditions, and a secure incremental update process, enables the system to "learn as it goes," allowing it to continuously fine-tune itself to adapt to slow changes without shutting down or affecting production. This significantly extends the effective service life, reduces maintenance costs, and demonstrates the core advantages of an intelligent system.

[0068] Specifically, it also includes emergency handling steps: when the sensor detects that the material pieces are not in place simultaneously, it directly triggers an emergency stop command, suspends the operation of the molding machine, and issues an audible and visual alarm to prompt manual intervention; at the same time, it records the time, location, and sensor data of the abnormal event for subsequent analysis and model optimization.

[0069] It should be further explained that the emergency handling procedure is a safety redundancy mechanism independent of the deep learning prediction loop. Its implementation details are as follows: A separate hardware interrupt routine is set in the PLC program, which directly monitors the input points of the two sensors. Once it is detected that within the expected arrival time window of the material (e.g., from loading to triggering should be completed within 2 seconds), the two sensors do not trigger simultaneously within the pre-set "simultaneous arrival judgment window" (e.g., 15 milliseconds), for example, only one sensor triggers or neither triggers, the interrupt routine will take effect immediately (response time < 2 milliseconds). Its execution actions include: (1) Send the highest priority emergency stop command to the servo drive to cut off power and lock the mechanical brakes.

[0070] (2) A specific alarm pattern is emitted by a buzzer and an alarm light connected to the PLC (for example, the red light flashes rapidly 5 times per second, and the buzzer emits a 2000Hz high-frequency sound that lasts for 2 seconds with a 1-second interval).

[0071] (3) Upon triggering the emergency stop, the PLC captures and saves a frame from the current industrial camera, and records the data streams from all environmental sensors within 1 second before and 0.5 seconds after the emergency stop. This data, along with millisecond-accurate timestamps, sensor status, current batch number, mold ID, and other information, is packaged into a single data file and automatically uploaded to the server's database via FTP. This data is marked as "hardware detection anomalies" for subsequent root cause analysis. For example, the correlation between such anomalies and specific batches of materials, specific mold temperatures, or specific time periods can be analyzed. Simultaneously, these anomaly cases are added to the historical dataset for the next round of model training or incremental learning, enabling the model to learn these obvious anomaly patterns that can be determined without relying on image analysis, further enriching its knowledge.

[0072] Understandably, this emergency response procedure constructs a "failure-safety" defense layer. It relies on no software algorithms, responds extremely quickly, and provides the most basic guarantee for production safety. Furthermore, it is not only a safety measure but also a high-quality data collector. Deep learning models primarily handle predictions in the "grey area," while these explicit hardware-level anomalies provide valuable, well-labeled training data, forming a positive cycle of "hardware detection driving data collection, data driving model optimization, and model optimization reducing hardware anomalies." This makes the entire system increasingly intelligent and reliable over time.

[0073] Please see Figure 2 The present invention provides another embodiment, which provides a real-time monitoring system for warpage of molding machine sheet, the real-time monitoring system for warpage of molding machine sheet includes: (1) At least two sensors 100 are arranged at opposite corners of the molding machine sheet conveying track to detect in real time whether the sheets arrive at the same time.

[0074] It should be further noted that sensor 100 is a Sick brand diffuse reflection photoelectric sensor, which is connected to the DI high-speed input module (model 6ES7221-1BF32-0XB0) of Siemens S7-1200 PLC via an M8 connector.

[0075] (2) Image acquisition device 200, installed near the material conveying track, is used to continuously acquire image data of the material.

[0076] It should be further noted that the image acquisition device 200 is a Baslerac A2500-20gc industrial camera, which is connected to an industrial edge computing gateway (model Advantech MIC-770, equipped with an Intel i5 processor, 16GB of memory, and 256GB of SSD) via a gigabit Ethernet cable.

[0077] (3) Environmental data acquisition module 300, including temperature sensor, humidity sensor and vibration sensor, for continuously acquiring production environment data.

[0078] It should be further noted that the environmental data acquisition module 300 is a custom integrated unit containing DS18B20, SHT31 and ADXL345 chips, and is connected to the same gateway through an RS485 to Ethernet module.

[0079] (4) A processor 400 is connected to the sensor 100, the image acquisition device 200 and the environmental data acquisition module 300, and is configured to execute the steps of the method described in any of the above-mentioned items.

[0080] It should be further noted that processor 400 is the edge computing gateway, running the Ubuntu 18.04 real-time kernel operating system. The software stack includes: ① Data acquisition service (written in Python) is responsible for reading sensor status from the PLC's OPCUA server, capturing images from the camera, and reading environmental data from the serial port.

[0081] ② The preprocessing and inference service (written in C++ and Lib Torch) implements all the preprocessing, feature fusion and deep learning model inference steps, and sends the resulting risk prediction value and warpage classification to the control service.

[0082] ③ Control service: Generates control instructions based on risk values ​​and sends them to the PLC via Socket communication.

[0083] (5) Controller 500, connected to the processor, is used to adjust the motion parameters of the actuator in the molding machine according to the control instructions generated by the processor.

[0084] It should be further explained that the controller is an S7-1200 PLC, which runs the main control program, receives control commands from the processor (such as modifying speed setpoints), controls servo drives (such as V90) to drive shifter actions, and also independently runs the aforementioned emergency interrupt handling routine. All components are installed in a control cabinet that meets IP54 standards. The cabinet uses DIN rails and cable ties to secure cables, ensuring interference resistance and reliability in industrial environments.

[0085] Understandably, this system is a complete, hardware-software co-working industrial IoT solution. It tightly integrates multiple aspects such as sensing, computing, decision-making, and control. The selection of an edge computing gateway ensures low latency and privacy security in data processing, avoiding the network risks and latency associated with uploading all data to the cloud. The division of labor between the PLC and the processor is clear: the PLC focuses on highly reliable real-time logic and safety control, while the processor focuses on complex algorithm calculations. This architecture leverages the intelligent advantages of IT technology (AI) while adhering to the bottom-line requirements of reliability and real-time performance in the OT (Operational Technology) field. It is a typical practice of "IT / OT convergence" in the context of Industry 4.0, and has high replicability and promotional value.

[0086] In a preferred embodiment, this application also provides an electronic device, the electronic device comprising: The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the real-time monitoring method for warpage of molding machine sheet. The computer device can be broadly categorized as a server, terminal, or any other electronic device with the necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, memory, network interface, communication interface, etc., connected via a system bus. The processor of the computer device can be used to provide the necessary computing, processing, and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and internal memory. The non-volatile storage medium may store an operating system, computer programs, etc. The internal memory can provide an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface and communication interface of the computer device can be used to connect and communicate with external devices via a network. When the computer program is executed by the processor, it performs the steps of the method of the present invention.

[0087] This invention can be implemented as a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the steps of the methods of embodiments of the invention to be performed. In one embodiment, the computer program is distributed across multiple network-coupled computer devices or processors, such that the computer program is stored, accessed, and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be executed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be executed by one or more computer devices or processors, and one or more other method steps / operations may be executed by one or more other computer devices or processors. One or more computer devices or processors may execute a single method step / operation, or execute two or more method steps / operations.

[0088] It will be understood by those skilled in the art that the method steps of the present invention can be performed by a computer program instructing related hardware, such as a computer device or processor, which may be stored in a non-transitory computer-readable storage medium. When the computer program is executed, the steps of the present invention are performed. Depending on the context, any references herein to memory, storage, database, or other media may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0089] The technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification, provided that such combination does not contain contradictions.

[0090] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for real-time monitoring of warpage in molding machine sheet, characterized in that, Includes the following steps: S1. At least two sensors are used to detect in real time whether the sheet material arrives at the diagonal position of the sheet material conveying track of the molding machine at the same time. The sensors are arranged diagonally to cover the sheet material area. S2. Image data of the material sheet is continuously acquired through an image acquisition device, and production environment data is continuously acquired through an environmental data acquisition module. The production environment data includes temperature, humidity and vibration parameters. S3. Input the detection signal from the sensor, the image data, and the production environment data into a pre-trained deep learning model; The deep learning model analyzes the input data in real time and outputs the warping assessment value and the punching risk prediction value of the material sheet. When the predicted risk value of the punching exceeds the preset risk threshold, a control command is generated to adjust the motion parameters of the actuator in the molding machine to achieve adaptive control.

2. The method according to claim 1, characterized in that, The process of continuously acquiring image data of the material sheet through an image acquisition device, and simultaneously continuously acquiring production environment data through an environmental data acquisition module, includes: The image acquisition device is an industrial camera, which is installed above or to the side of the sheet conveying track. The shooting direction is perpendicular to the sheet plane, and the shooting frequency is synchronized with the working rhythm of the molding machine to ensure the acquisition of continuous sheet movement sequence images. The environmental data acquisition module includes a temperature sensor, a humidity sensor, and a vibration sensor, which are respectively arranged at key positions around the molding machine to collect production environment parameters that affect sheet warping in real time. The sensor is a photoelectric sensor or a proximity switch, which is arranged at the two diagonal positions at both ends of the sheet conveying track, with its sensing surface facing the sheet feeding direction.

3. The method according to claim 2, characterized in that, Before inputting data into a deep learning model, a data preprocessing step is also included: The detection signal from the sensor is denoised and time-aligned to eliminate false triggering and signal delay. The image data is subjected to grayscale conversion, noise reduction, and size normalization to eliminate illumination variations and image distortion. The production environment data is filtered and standardized to eliminate noise and dimensional differences, thereby generating a unified input data format for use by deep learning models.

4. The method according to claim 3, characterized in that, The preprocessed sensor detection signal, the image data, and the production environment data are fused using multimodal data fusion, specifically including: Visual features, including material outline features, edge features, and texture features, are extracted from the image data; temporal features, including temperature change rate, humidity fluctuation, and vibration spectrum, are extracted from the production environment data; the visual features, temporal features, and the detection signal of the sensor are fused at the feature level to generate a multi-dimensional feature vector containing multi-dimensional information, which is used as the input of the deep learning model.

5. The method according to claim 4, characterized in that, The deep learning model is a convolutional neural network, which includes an input layer, multiple convolutional layers, pooling layers, fully connected layers, and an output layer. The input layer receives the multidimensional feature vector. The convolutional and pooling layers are used to extract high-level features layer by layer. The fully connected layer is used for feature integration. The output layer outputs the warpage assessment value and the runaway risk prediction value. The warpage assessment value is a classification label or regression value, representing the warpage level or specific warpage amount of the material. The runaway risk prediction value is a probability value, representing the possibility that the material will run off the track.

6. The method according to claim 5, characterized in that, The deep learning model is trained using historical production data, which includes normal material conveying cases and abnormal ejection cases. Each case includes sensor signals, image data, environmental data, and corresponding warpage and risk labels. During training, the model is optimized using a cross-entropy loss function or a mean squared error loss function, and the model parameters are updated using a stochastic gradient descent algorithm or an adaptive moment estimation algorithm until the model converges to a preset accuracy.

7. The method according to claim 6, characterized in that, When the predicted risk value exceeds the preset risk threshold, a control command is generated, including adjusting the movement speed, acceleration, or stop position of the actuator to achieve adaptive control. The preset risk threshold is dynamically adjusted according to the production batch, specifically calculated by statistical methods based on the material characteristics, environmental conditions, or historical anomaly rate of the current batch of material.

8. The method according to claim 7, characterized in that, It also includes real-time monitoring steps: The prediction performance of the deep learning model is monitored in real time. The prediction error is calculated by comparing the predicted value with the actual result. When the prediction error continues to exceed the tolerance, the online model update mechanism is triggered. The model is incrementally learned using the current production data to maintain its adaptability. The incremental learning adopts an online learning algorithm to update some parameters without retraining the entire model.

9. The method according to claim 1, characterized in that, It also includes emergency response steps: When the sensor detects that the material pieces are not in place simultaneously, it directly triggers an emergency stop command, suspends the operation of the molding machine, and issues an audible and visual alarm to prompt manual intervention; at the same time, it records the time, location, and sensor data of the abnormal event for subsequent analysis and model optimization.

10. A real-time monitoring system for warpage of sheet metal in a molding machine, characterized in that, The real-time monitoring method for warpage of molding machine sheet as described in any one of claims 1-9 includes: At least two sensors are arranged at opposite positions on the sheet conveying track of the molding machine to detect in real time whether the sheets arrive at the same time. An image acquisition device is installed near the sheet conveying track to continuously acquire image data of the sheet; The environmental data acquisition module includes a temperature sensor, a humidity sensor, and a vibration sensor, used to continuously acquire production environment data; The processor is connected to the sensor, image acquisition device, and environmental data acquisition module. And a controller, connected to the processor, for adjusting the motion parameters of the actuator in the molding machine according to the control instructions generated by the processor.

Images