Intelligent optimization method and system for manufacturing process based on multi-modal data fusion

Abstract

The application discloses a kind of based on multi-modal data fusion's manufacturing process intelligent optimization method and system, belong to production intelligent control technical field, this method includes: infrared thermal imaging video stream in production process is split into multiple image frames, image frame is analyzed to obtain the gap width between product contour and mould;If there is gap width greater than critical threshold in last image frame, actual trend sequence that gap width changes with time is generated;Actual trend sequence is analyzed to determine the adjustment direction of abnormal heating partition and heating parameter;Based on adjustment direction, the heating parameter of abnormal heating partition is disturbed, and input to rheological prediction model, to locate out the target heating partition that causes actual trend sequence in reverse, and determine its corresponding heating parameter adjustment amount;Based on adjustment amount, the heating parameter of target heating partition is corrected.Through the application, intelligent precise optimization of manufacturing process parameters can be realized.

Description

Technical Field

[0001] This invention belongs to the field of production control technology, specifically relating to an intelligent optimization method and system for manufacturing processes based on multimodal data fusion. Background Technology

[0002] In industrialized intelligent manufacturing, products using hot bending processes typically involve placing the product on a graphite mold and undergoing multiple steps, including preheating, softening, pressurizing, and cooling under pressure, to soften the product at high temperatures and allow it to conform to the mold's shape. During hot bending, the product's contour fit is a key indicator determining the product yield. If the product edges fail to fit tightly to the mold, dimensional deviations in the finished product will occur.

[0003] To effectively optimize the hot bending process of products, various methods have been proposed in the existing technology. For example, PID control algorithms can be used to adjust the temperature in the heating furnace to achieve a preset temperature curve; or a 3D contour scanner can be used at the end of the production line to perform offline inspection of the cooled finished product, and the process parameters can be adjusted based on the inspection results using a data association algorithm; or finite element analysis can be used to simulate the forming process before production to determine the initial process parameters.

[0004] While the aforementioned existing technologies have achieved optimization of process parameters to some extent, hot bending equipment typically contains multiple closely arranged heating zones with complex thermodynamic crosstalk between them. Existing methods ignore the dynamic characteristics of the product's evolution over time during processing, making it difficult to distinguish whether the same final defect is caused by insufficient heating leading to softening lag or excessive preheating leading to thermal stress release. This results in a lack of targeted parameter correction, making it difficult to guarantee product quality, and the level of intelligence in the manufacturing process needs to be further improved. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method and system for intelligent optimization of manufacturing processes based on multimodal data fusion, thereby resolving the issues present in the background art.

[0006] To achieve the aforementioned objectives, this invention proposes an intelligent optimization method for manufacturing processes based on multimodal data fusion, comprising: Acquire infrared thermal imaging video streams during the production process, break the infrared thermal imaging video streams into multiple image frames, and analyze the image frames to obtain the gap width between the product outline and the mold. If the gap width in the last image frame is greater than the critical threshold, the gap width measured in each image frame is combined with the corresponding timestamp to generate an actual trend sequence of gap width changes over time. A rheological prediction model is established based on a time-series neural network. The rheological prediction model predicts the trend sequence of gap width changes based on the heating parameters of each heating zone and the material properties of the product. The actual trend sequence is divided into multiple flow intervals, and the flow intervals are analyzed to determine abnormal heating zones and the direction of heating parameter adjustment. Based on the adjustment direction, the heating parameters of the abnormal heating zone are perturbed, and the heating parameters of each heating zone after perturbation and the material properties of the product are input into the rheological prediction model to reverse locate the target heating zone that caused the actual trend sequence and determine its corresponding heating parameter adjustment amount. The heating parameters of the target heating zone are corrected based on the adjustment amount.

[0007] Further, analyzing image frames to obtain the gap width between the product outline and the mold includes the following steps: Each image frame is processed for grayscale and distortion correction to obtain a standard image. In the first standard image, the initial outline of the product and the mold outline are located based on the image recognition algorithm. Multiple feature points are located in the initial outline, and the Euclidean distance between the feature points and the mold outline is used as the initial gap width. Based on the moving speed of the production equipment and the pixel features of the feature points, the feature points are mapped to each subsequent standard image. The displacement distance of the same feature point in the vertical or horizontal direction is calculated based on adjacent standard images. The gap width of the feature point in the previous standard image is subtracted from the displacement distance. The gap width of the feature point in the current standard image is updated and obtained in a recursive manner.

[0008] Furthermore, mapping the feature points to each subsequent standard image includes the following steps: In the current standard image, the normal direction of the initial contour line position of the feature point is calculated, and the pixel grayscale sequence of the feature point is extracted along the normal direction. Based on the moving speed of the production equipment, the predicted coordinates of the feature point in the next standard image are estimated. With the predicted coordinates as the center, combined with the product flow inertia, a search area extending outward along the normal direction is constructed. Within the search area of ​​the next standard image, the pixel grayscale sequence of each pixel is traversed and obtained, and the similarity with the pixel grayscale sequence of the feature point in the current standard image is calculated. The pixel with the highest similarity is selected as the visual coordinate. The confidence weight is set according to the image sharpness. Based on the confidence weight, the predicted coordinates and the visual coordinates are weighted and fused to obtain the position of the feature point in the next standard image.

[0009] Furthermore, establishing a rheological prediction model based on a temporal neural network includes the following steps: The heating parameters of each heating zone are combined into a parameter sequence. The parameter sequence is convolved according to different time sliding windows to extract the thermal accumulation feature sequence at different time scales. The hidden layer state of the temporal neural network at the current time step is used as the query vector and the thermal accumulation feature sequence is used as the key vector. By calculating the similarity between the query vector and the key vector, an attention score distribution representing the degree of influence of each historical time step on the current rheological state is generated. The thermal cumulative feature sequence is weighted and summed using the attention score distribution to obtain the thermal response context vector. The thermal response context vector is then concatenated with the product material properties to construct an enhanced feature vector. This enhanced feature vector is then input into the recurrent unit of a temporal neural network to obtain the trend sequence of gap width changes.

[0010] Furthermore, dividing the actual trend sequence into multiple flow intervals and analyzing these flow intervals to determine abnormal heating zones and the adjustment direction of heating parameters includes the following steps: A standard process sequence library is constructed, which includes standard trend sequences of ideal rheological processes with different material properties. Based on the data form, the standard trend sequences are divided into multiple flow intervals. At the same time, the allowable fluctuation range of each sequence point in the standard trend sequence is determined. Based on the allowable fluctuation range, the standard trend sequence is converted into a standard trend range. The portion of the actual trend sequence that exceeds the standard trend range is identified as an abnormal flow interval. The flow interval and statistical characteristics of the abnormal flow interval are extracted, and the material properties, flow interval and statistical characteristics are combined into an abnormal signature. Historical adjustment records that match the abnormal signature are retrieved from a preset knowledge base. Based on the matching historical adjustment records, the adjustment direction of the heating zone and heating parameters that caused the abnormality is determined.

[0011] Furthermore, determining the adjustment direction of the heating zone and heating parameters that caused the anomaly based on the matched historical adjustment records includes the following steps: The frequency ratio caused by each heating zone and the frequency ratio of the adjustment direction are calculated from the historical adjustment records that match the abnormal signature of the current actual trend sequence. The heating zone with the highest frequency ratio and the adjustment direction are output as the matching result.

[0012] Furthermore, perturbing the heating parameters of the abnormal heating zone based on the adjustment direction includes the following steps: Based on the adjustment direction, a primary perturbation is applied to the abnormal heating zone. Simultaneously, based on the equipment thermodynamic crosstalk model, a secondary perturbation with amplitude decreasing with distance is applied to the heating zones adjacent to the abnormal heating zone to generate simulated heating parameters. The simulated heating parameters are used as the optimization target. Through iterative search, a set of simulated heating parameters is found that minimizes the difference between the predicted trend sequence output by the rheological prediction model and the actual trend sequence. The adjustment amount is determined based on the difference between the simulated heating parameters and the current heating parameters.

[0013] Furthermore, the iterative search includes the following steps: A cost function is defined to measure the difference between the predicted trend sequence and the actual trend sequence. The simulated heating parameters are used as optimization variables, and the initial simulated heating parameters are used as the search starting point. The particle swarm optimization algorithm is used to iteratively solve for the simulated heating parameters that minimize the cost function.

[0014] This invention also provides an intelligent optimization system for manufacturing processes based on multimodal data fusion. This system is used to implement the methods described above, and includes: The identification module acquires the infrared thermal imaging video stream during the production process, splits the infrared thermal imaging video stream into multiple image frames, analyzes the image frames to obtain the gap width between the product outline and the mold, and if there is a gap width in the last image frame that is greater than the critical threshold, the gap width measured in each image frame is combined with the corresponding timestamp to generate the actual trend sequence of gap width changing over time. The prediction module establishes a rheological prediction model based on a time-series neural network. The rheological prediction model predicts the trend sequence of gap width changes based on the heating parameters of each heating zone and the material properties of the product. The positioning module divides the actual trend sequence into multiple flow intervals, analyzes these flow intervals to determine abnormal heating zones and the direction of heating parameter adjustment; The analysis module perturbs the heating parameters of the abnormal heating zone based on the adjustment direction, inputs the heating parameters of each heating zone after perturbation and the material properties of the product into the rheological prediction model, so as to reverse locate the target heating zone that caused the actual trend sequence, determine its corresponding heating parameter adjustment amount, and correct the heating parameters of the target heating zone based on the adjustment amount.

[0015] The beneficial effects of this invention are as follows: This invention first acquires an infrared thermal imaging video stream during the production process and breaks it down into multiple image frames. The image frames are analyzed to obtain the gap width between the product outline and the mold. When the gap width exceeds the limit, a real-time trend sequence of the gap width over time is generated, thus achieving real-time monitoring of the dynamic rheological process of the product at high temperatures. The actual trend sequence is then divided into multiple flow intervals for analysis to identify abnormal heating zones and the direction of heating parameter adjustment. This solves the problem of traditional parameter correction lacking specificity and failing to distinguish between anomalies in different time periods leading to the same final defect morphology. Finally, based on the determined adjustment direction, the parameters of the abnormal heating zones are perturbed, and the perturbed parameters are input into a rheological prediction model for reverse localization to find the target heating zone causing the actual trend sequence and the corresponding heating parameter adjustment amount. Correction is then performed based on this adjustment amount, thereby achieving precise optimization of process parameters. This solves the problem of poor optimization results in existing technologies due to neglecting thermodynamic crosstalk and dynamic characteristics, improving the intelligence level of the manufacturing process. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of an intelligent optimization method for manufacturing processes based on multimodal data fusion, as described in this invention. Figure 2 This is a schematic diagram illustrating the principle of determining abnormal flow intervals by comparing trend sequences in this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first script may be referred to as a second script, and similarly, a second script may be referred to as a first script.

[0019] like Figure 1 As shown, a manufacturing process intelligent optimization method based on multimodal data fusion includes: The infrared thermal imaging video stream during the production process is acquired, and the infrared thermal imaging video stream is split into multiple image frames. The image frames are analyzed to obtain the gap width between the product outline and the mold.

[0020] This embodiment uses the hot bending process of curved glass as an example to describe this application in detail. However, the method of this application is also applicable to other manufacturing scenarios involving the thermorheological properties of materials, such as injection molding and carbon fiber molding. In this production scenario, the production equipment is usually a multi-station continuous or tunnel-type hot bending machine. In order to meet the requirements of complex temperature profiles, the inside of the equipment is divided into multiple independent heating zones along the conveying direction, such as a preheating zone, a softening zone, a forming zone, and an annealing zone. Each zone is equipped with an independent heating element and a temperature control unit.

[0021] The specific production process is as follows: The glass to be processed is placed above a graphite mold and passes through each heating zone sequentially via a conveyor device. Under high temperature, the glass absorbs heat and gradually softens, its viscosity decreases, and it undergoes rheological changes under its own weight or auxiliary pressure, gradually sinking and adhering to the curved contour surface of the mold. During this process, the gap width between the glass edge and the mold surface is the most critical geometric indicator for measuring the forming quality. Improper heating parameters can lead to insufficient glass softening. Due to the enclosed environment and extremely high temperature inside the furnace, traditional contact measurement is not feasible. Therefore, this application introduces infrared thermal imaging technology for non-contact, full-process monitoring, aiming to optimize the process parameters of each heating zone by analyzing the dynamic change trend of the gap width.

[0022] Returning to this step, we first acquire the infrared thermal imaging video stream, then perform frame separation on the stream. For the first frame, we use edge detection or threshold segmentation algorithms to identify the product's edge contour and the reference contour of the supporting mold. By comparing the two contours, we obtain the gap width, which directly reflects the product's forming quality at that moment. Subsequent continuous analysis of image frames captures the entire dynamic process of the product gradually softening and conforming to the mold under the influence of gravity and heat.

[0023] If the gap width in the last image frame is greater than the critical threshold, the gap width measured in each image frame is combined with the corresponding timestamp to generate an actual trend sequence of gap width changes over time.

[0024] In the final image frame, if the gap width exceeds a preset critical threshold, the product is deemed defective or poses a quality risk. At this point, the gap widths calculated for each previous frame are mapped one-to-one with their corresponding time points, generating a complete actual trend sequence. This actual trend sequence demonstrates the rate and trajectory of gap width reduction over time. This allows subsequent analysis to determine whether the product experienced bonding lag in the early, middle, or late stages of heating.

[0025] A rheological prediction model is established based on a temporal neural network. The rheological prediction model predicts the trend sequence of gap width changes based on the heating parameters of each heating zone and the material properties of the product.

[0026] A rheological prediction model is a surrogate model built on deep learning. Its function is to simulate the physical response of production equipment. By analyzing current process conditions and the inherent characteristics of the product, it outputs predicted gap width changes. The rheological prediction model can extrapolate the product forming process under different combinations of process parameters, thus eliminating the need for actual simulation and reducing production costs.

[0027] The actual trend sequence is divided into multiple flow intervals, and the flow intervals are analyzed to determine abnormal heating zones and the direction of heating parameter adjustment.

[0028] Because hot bending equipment typically comprises multiple physically continuous heating zones, heat conduction and diffusion mean that anomalies at a single point in time can be caused by the combined effects of multiple zones. Therefore, this step divides the actual trend sequence of the entire forming process into several flow intervals on the time axis, such as preheating, softening, and forming sections. By analyzing the morphological differences between the actual and ideal curves within each interval, and then combining this with the process mechanism, the heating region causing the rheological anomaly in that interval is inferred. For example, if the gap decreases too slowly in the softening section, the corresponding heating zone in the middle may be underheated. This step narrows down the search space for subsequent optimization.

[0029] The heating parameters of the abnormal heating zones are perturbed based on the adjustment direction. The heating parameters of each heating zone after perturbation and the material properties of the product are input into the rheological prediction model to reverse locate the target heating zone that caused the actual trend sequence and determine its corresponding heating parameter adjustment amount.

[0030] The heating parameters of the target heating zone are corrected based on the adjustment amount.

[0031] Based on the determined adjustment direction, the input parameters of the rheological prediction model are slightly perturbed. After each perturbation, the rheological prediction model generates a new prediction sequence. These prediction sequences are compared with the actual observed trend sequence. If the prediction sequence generated by the rheological prediction model for a specific set of heating parameters is highly similar to the actual trend sequence, then it can be determined which heating zone is experiencing an anomaly and how much the process parameters need to be adjusted. Finally, the calculated adjustment amount is sent to the equipment's PLC or temperature control system. Through closed-loop feedback adjustment, the actual operating conditions of the target heating zone are corrected, thereby eliminating the process deviations that cause gaps to exceed limits in the next production cycle, improving product fit and yield.

[0032] In this embodiment, analyzing image frames to obtain the gap width between the product outline and the mold includes the following steps: Each image frame is processed for grayscale and distortion correction to obtain a standard image. In the first standard image, the initial contour line of the product and the mold contour line are located based on the image recognition algorithm. Multiple feature points are located in the initial contour line, and the Euclidean distance between the feature points and the mold contour line is used as the initial gap width.

[0033] In this embodiment, a weighted average method is first used to convert color image frames in the infrared thermal imaging video stream into single-channel grayscale images to reduce data dimensionality. Then, a pre-calibrated camera intrinsic parameter matrix is ​​used to eliminate lens distortion, ensuring the accuracy of geometric measurements. In this application scenario, the infrared camera position is fixed, while the product moves smoothly along the production line with the mold. Therefore, the possible location of the same pixel in each standard image can be inferred based on the production line conveyor speed. Based on the above analysis, this embodiment adopts a computationally intensive contour extraction algorithm that runs only in the first standard image of the video stream, avoiding repeated global searches for each frame, reducing the waste of algorithm resources, and significantly improving the real-time processing speed of the system.

[0034] Then, select several feature points at equal intervals on the initial outline of the product determined in the first frame, and then obtain the Euclidean distance between the feature points and each edge of the mold. The smallest Euclidean distance is taken as the initial gap width of the feature point, and the mold edge corresponding to the smallest Euclidean distance is determined as the edge that the feature point is approaching.

[0035] Based on the moving speed of the production equipment and the pixel features of the feature points, the feature points are mapped to each subsequent standard image. The displacement distance of the same feature point in the vertical or horizontal direction is calculated based on adjacent standard images. The gap width of the feature point in the previous standard image is subtracted from the displacement distance. The gap width of the feature point in the current standard image is updated and obtained in a recursive manner.

[0036] Since the infrared camera is fixed while the mold moves with the conveyor belt, the theoretical displacement of the feature points can be calculated based on the production equipment's moving speed fed back by the motor encoder and the frame interval between the standard images. The feature point coordinates of the previous frame are then translated along the conveying direction to obtain the approximate predicted coordinates in the current frame. Subsequently, in order to correct for minor positional deviations caused by mechanical vibration or thermorheology, the predicted coordinates need to be corrected. The specific correction method will be described in detail later.

[0037] For each tracked feature point, its displacement distance between adjacent standard images is calculated. Specifically, based on the initial gap width, it is determined whether the feature point moves towards the vertical or horizontal mold contour line. If it moves towards the vertical mold contour line, the horizontal displacement distance is obtained. Subtracting the horizontal displacement distance from the initial gap width allows calculation of how much distance remains between the feature point and the mold contour line, which in turn allows calculation of the gap width corresponding to the current feature point.

[0038] In this embodiment, mapping feature points to each subsequent standard image includes the following steps: In the current standard image, the normal direction of the initial contour line position of the feature point is calculated, and the pixel grayscale sequence of the feature point is extracted along the normal direction. Based on the moving speed of the production equipment, the predicted coordinates of the feature point in the next standard image are estimated. With the predicted coordinates as the center, combined with the product flow inertia, a search area extending outward along the normal direction is constructed. Within the search area of ​​the next standard image, the pixel grayscale sequence of each pixel is traversed and obtained, and the similarity with the pixel grayscale sequence of the feature point in the current standard image is calculated. The pixel with the highest similarity is selected as the visual coordinate. The confidence weight is set according to the image sharpness. Based on the confidence weight, the predicted coordinates and the visual coordinates are weighted and fused to obtain the position of the feature point in the next standard image.

[0039] First, the gradient direction of the current feature point is calculated and used as the normal direction, which is the direction of the largest pixel value change rate. The gray values ​​of nine pixels along the normal direction of the feature point are extracted as a pixel gray-level sequence. For example, taking the feature point as the center, four pixels are taken forward and four pixels backward, and these are combined with the feature point's own gray value to form the pixel gray-level sequence. Next, using the known moving speed of the production equipment, the theoretical position of the feature point when it only translates with the production line is calculated, i.e., the predicted coordinates.

[0040] Because glass is in a softened rheological state at high temperatures, its edges are not static but exhibit a continuous outward expansion and downward drooping flow inertia, while maintaining a high viscosity. Furthermore, the polished graphite mold surface has an extremely low coefficient of friction, causing the glass's elongation flow under gravity and pressure to exhibit highly stable laminar flow characteristics, rather than chaotic turbulence. Based on this stable flow inertia, edge points only undergo regular displacements towards the outside of the normal. Therefore, a fan-shaped search region is constructed with the predicted coordinates as vertices, radiating outwards along the normal direction. During search matching, to ensure geometric feature consistency, the normal direction determined by feature points in the current standard image is used as a fixed sampling direction. For each candidate pixel within the fan-shaped search region, a pixel grayscale sequence crossing the edge is extracted according to the sampling direction, centered on the candidate pixel; this is defined as the candidate sequence. This directional search strategy not only conforms to the physical laws of outward flow of liquid glass, covering new positions that edge points may reach due to thermal expansion and rheology, but also effectively eliminates interference calculations from the reverse region.

[0041] Subsequently, the system searches for candidate sequences with the highest similarity to the grayscale sequences of feature point pixels within the fan-shaped region, and determines the coordinates of the pixels corresponding to these candidate sequences as visual coordinates. Specifically, a zero-mean normalized cross-correlation algorithm is used to calculate the similarity between pixel sequences. This algorithm normalizes the grayscale sequences by subtracting the mean and dividing by the standard deviation, focusing only on the waveform trend of the grayscale sequences and eliminating the influence of overall grayscale value decay caused by the decrease in glass temperature over time. To counteract potential thermal noise or motion blur in infrared imaging, the system also calculates the image sharpness score in real time. The sharpness score, for example, is the Laplacian gradient value. Higher sharpness assigns higher weight to visual coordinates, while lower sharpness assigns higher weight to predicted coordinates. The specific weighting method can be set according to actual needs and will not be described here. Through weighted fusion of these two methods, the precise location of the feature point in the next frame image can be obtained.

[0042] Specifically, the glass flow velocity is finite, and the inter-frame displacement cannot change too drastically. If the visual coordinates and the predicted point coordinates are too far apart, for example, more than 5 pixels, the visual recognition is directly judged as an error, its weight is reduced to zero, and the predicted coordinates are completely trusted.

[0043] In this embodiment, establishing a rheological prediction model based on a temporal neural network includes the following steps: The heating parameters of each heating zone are combined into a parameter sequence. The parameter sequence is convolved according to different time sliding windows to extract the thermal accumulation feature sequence at different time scales. The hidden layer state of the temporal neural network at the current time step is used as the query vector, and the thermal accumulation feature sequence is used as the key vector. By calculating the similarity between the query vector and the key vector, an attention score distribution representing the degree of influence of each historical time step on the current rheological state is generated.

[0044] As glass is a poor conductor, the infrared radiation energy generated by the heater first acts on the upper surface of the glass. It takes time for the heat to transfer from the upper surface to the central layer and the lower surface of the glass. This leads to a time misalignment between the control signal and the deformation response. For example, the deformation that occurs at the current moment may be caused by the heat applied to the surface at a certain moment in the past.

[0045] Therefore, the heating parameters of each heating zone at different times are integrated into a corresponding heating parameter sequence, which includes power output and temperature. This sequence is then processed by parallel sliding convolution using three pre-defined sets of one-dimensional convolution kernels with different time spans to generate a multi-scale thermal accumulation feature sequence. The sizes of the one-dimensional convolution kernels are, for example, a short window of 3 seconds, a medium window of 15 seconds, and a long window of 60 seconds. The short-window convolution kernel is used to capture the radiative thermal shock to the glass surface caused by sudden changes in heater power; the medium-window convolution kernel is used to obtain the temperature gradient diffusion conducted from the upper surface of the glass to the central layer; and the long-window convolution kernel represents the thermal energy absorbed and stored by the glass as a whole.

[0046] The time series model in this embodiment is an LSTM model. A temporal attention mechanism layer is constructed within the LSTM model. The temporal attention mechanism layer extracts the hidden layer state output after the previous time step as the query vector and uses the multi-scale hot-accumulated feature sequence of N time steps output by the convolution as the key vector. At the same time, the query vector and the key vector are subjected to bilinear tensor product operation using the weight matrix to calculate the projection matching degree of the two in the latent feature space. After Softmax normalization, a time dimension weight vector of length N is generated. This time dimension weight vector is the attention score distribution.

[0047] The attention score distribution is essentially the importance distribution of historical time steps. If the attention score is highest at time step t, it indicates that the input at time step t is the main cause of the current rheological state. This step uses the hidden layer state, which represents the current deformation trend, to retrieve historical data in reverse. By calculating the projection matching degree, the specific physical time required for thermal energy to penetrate the glass medium is determined, and the time step data that truly causes the current deformation is given a high weight, thereby dynamically adapting to the hysteresis changes in thermal response under different operating conditions.

[0048] The thermal cumulative feature sequence is weighted and summed using the attention score distribution to obtain the thermal response context vector. The thermal response context vector is then concatenated with the product material properties to construct an enhanced feature vector. This enhanced feature vector is then input into the recurrent unit of a temporal neural network to obtain the trend sequence of gap width changes.

[0049] Using the calculated attention score distribution as weighting coefficients, data of the same scale in the multi-scale thermal accumulation feature sequence are weighted and summed to synthesize a thermal response context vector. The thermal response context vector is equivalent to reorganizing the thermal input signals scattered along the time axis according to their magnitude of influence on the current moment. Finally, the thermal response context vector is concatenated with the current product attribute data along the feature channel dimension to construct an enhanced feature vector. This enhanced feature vector is input into the predictive decoder of a temporal neural network. The decoder iteratively uses the currently predicted state as the input for the next time step, ultimately outputting a trend sequence of the gap width.

[0050] The above process iterates continuously during model training and drives the updating of the weight matrix and LSTM model parameters by minimizing the gradient backpropagation of the prediction error.

[0051] In this embodiment, the actual trend sequence is divided into multiple flow intervals, and the analysis of these flow intervals to determine abnormal heating zones and the direction of heating parameter adjustment includes the following steps: A standard process sequence library is constructed, which includes standard trend sequences of ideal rheological processes with different material properties. Based on the data form, the standard trend sequences are divided into multiple flow intervals. At the same time, the allowable fluctuation range of each sequence point in the standard trend sequence is determined. Based on the allowable fluctuation range, the standard trend sequence is converted into a standard trend range.

[0052] First, multiple product batches confirmed as qualified by quality inspection are selected from the historical production database to obtain their corresponding actual trend sequences. Since there may be slight differences in the production cycle time between different batches, a dynamic time warping algorithm is used to align this sequence along the time axis, eliminating time phase shifts. Next, for the aligned sequence, the mean and standard deviation of the values ​​at each time sampling point are calculated. The sequence composed of the average values ​​of all time points is defined as the standard trend sequence, representing the ideal rheological process. Subsequently, morphological segmentation is performed on the standard trend sequence. The segmentation can be based on time nodes of the physical process or automatically identified by a change point detection algorithm at the dividing points where the curve shape changes significantly. In this way, the standard trend sequence is divided into multiple standard flow intervals, such as the initial preheating period, the main heating and shaping period, and the temperature stabilization period.

[0053] Based on the rule of three standard deviations, a fluctuation range is generated for each sequence point in the standard trend sequence. These fluctuation ranges are then combined chronologically to obtain the standard trend range. In other words, only values ​​in the actual trend sequence that exceed their corresponding fluctuation ranges are defined as outliers.

[0054] The portion of the actual trend sequence that exceeds the standard trend range is identified as an abnormal flow range. The flow range and statistical characteristics of the abnormal flow range are extracted, and the material properties, flow range, and statistical characteristics are combined into an abnormal signature. Historical adjustment records that match the abnormal signature are retrieved from a pre-set knowledge base. Based on the matching historical adjustment records, the heating zone that caused the abnormality and the adjustment direction of the heating parameters are determined.

[0055] In this embodiment, determining the heating zone and the direction of adjustment of heating parameters that cause the anomaly based on matching historical adjustment records includes the following steps: The frequency ratio and adjustment direction caused by each heating zone are counted in the historical adjustment records that match the abnormal signature of the current actual trend sequence. The heating zone with the highest frequency ratio and the adjustment direction are output as the matching result.

[0056] like Figure 2 As shown, the actual trend sequence is compared with the standard trend range, and the portion exceeding the standard trend range is identified as an abnormal flow interval. Then, the flow interval and morphological features of the abnormal flow interval are extracted. Morphological features include: the start time of the anomaly, duration, direction of deviation, peak value of the deviation, and steepness of the deviation curve. The flow interval and morphological features are combined to form an anomaly signature. A pre-set knowledge base stores historical fault datasets. Each historical record contains three parts of information: the flow interval and morphological features of the historical anomaly, the anomaly heating zone, and the effective parameter adjustment amount. After the system extracts the anomaly signature of the current actual trend sequence, it first performs a similarity search in the knowledge base. The Euclidean distance or cosine similarity between the anomaly signature and the historical records in the database is calculated, and multiple historical records with similarity higher than a preset threshold (e.g., 90%) are selected as a reference sample set.

[0057] Next, statistical analysis is performed on the reference sample set. For example, among 100 similar historical records, 85 cases were ultimately identified as "low temperature in the second heating zone," 10 cases as "low temperature in the third heating zone," and 5 cases were caused by other reasons. At this point, the system calculates that the prior probability of the second heating zone is 85%. Therefore, the second heating zone with a frequency ratio of 85% is identified as the central heating zone of this anomaly, and its temperature needs to be increased. Through this step, a simple algorithm can be used to quickly and initially determine the heating range where anomalies may occur and determine the direction of subsequent perturbation of heating parameters.

[0058] In this embodiment, perturbing the heating parameters of the abnormal heating zone based on the adjustment direction includes the following steps: The main perturbation is applied to the abnormal heating zone based on the adjustment direction. At the same time, based on the equipment thermodynamic crosstalk model, the secondary perturbation with amplitude decaying with distance is applied to the heating zone adjacent to the abnormal heating zone to generate simulated heating parameters. The simulated heating parameters are used as the optimization target. Through iterative search, a set of simulated heating parameters that minimizes the difference between the predicted trend sequence output by the rheological prediction model and the actual trend sequence is found. The adjustment amount is determined according to the difference between the simulated heating parameters and the current heating parameters.

[0059] In this embodiment, the iterative search includes the following steps: A cost function is defined to measure the difference between the predicted trend sequence and the actual trend sequence. The simulated heating parameters are used as optimization variables, and the initial simulated heating parameters are used as the search starting point. The particle swarm optimization algorithm is used to iteratively solve for the simulated heating parameters that minimize the cost function.

[0060] The heating zones within a heating device are not completely thermally independent. Due to their physical proximity, they exchange heat through conduction, radiation, and convection. This phenomenon is known as thermodynamic crosstalk. When a central heating zone is adjusted, its temperature change will inevitably affect the actual temperature distribution of its adjacent zones. The intensity of this effect is usually inversely proportional to the physical distance; the closer the distance, the greater the effect, and the farther the distance, the smaller the effect. If, during the simulation optimization process, the parameters of the central heating zone are changed in isolation while ignoring this crosstalk effect, the predicted trend sequence generated by the rheological prediction model will have an inherent deviation from the actual trend sequence generated by the real physical process. This is because the model does not consider the synergistic thermal effects of neighboring areas. This will lead to inaccurate heating parameter adjustments determined by the reverse location, failing to truly reflect the root cause of the defects. Therefore, to make the simulation process closer to physical reality, this embodiment applies a secondary perturbation that decays with distance to adjacent zones while applying the main perturbation.

[0061] The thermodynamic crosstalk model is pre-established through experimental calibration or finite element thermal field simulation. This model includes the temperature influence coefficients of a 1°C temperature change in one heating zone on its surrounding zones. When implementing the perturbation, firstly, based on the adjustment direction determined by the knowledge base, an initial primary perturbation, such as a 5°C temperature increase, is applied to the central heating zone identified as having the highest probability, for example, zone two. Then, the influence coefficients of zone two on all other zones are obtained, and the adjustment amount of the primary perturbation (an increase of 5°C) is multiplied by these influence coefficients to calculate the secondary perturbation amplitude that each adjacent zone should experience. For example, if the influence coefficients for zones one and three are 0.2, their respective secondary perturbation is an increase of 1°C, while the influence coefficient for zone four is 0.05, its secondary perturbation is an increase of 0.25°C. In this way, the system generates a set of simulated heating parameter combinations that include the primary perturbation and all attenuated secondary perturbations.

[0062] The simulated heating parameters are input into the rheological prediction model to obtain the predicted trend sequence. The difference between the predicted trend sequence and the actual trend sequence is calculated, and the model is iteratively adjusted based on the difference. This process is repeated until an optimal adjustment amount is found. For example, it was found that when the temperature of the second zone increases by 3.5℃, the root mean square error between the predicted trend sequence output by the model and the actual observed abnormal trend sequence reaches its minimum. At this point, the system determines that the target heating zone is the second zone, and the adjustment amount is 3.5℃.

[0063] The cost function can be a function that calculates the Euclidean distance between the predicted trend sequence and the actual trend sequence. The parameter adjustment amount of each heating zone is used as the variable that the optimization algorithm needs to find. The adjustment amount here includes not only the adjustment amount of the previously determined central heating zone, but also, in order to ensure the stability and physical feasibility of the solution process, a reasonable search boundary needs to be set for these adjustment amounts. For example, the temperature adjustment range cannot exceed the safety upper limit of the equipment, nor can it be lower than the effective lower limit of the physical process.

[0064] Starting with the previously determined heating zones and their adjustment directions, optimization begins by identifying the areas where the second heating zone's temperature is too low. For example, if the knowledge base indicates that the second heating zone's temperature is too low, the initial simulated heating parameters would be the current temperature of the second heating zone plus a small positive increment, such as 0.5°C, with the temperatures of adjacent zones adjusted accordingly based on the crosstalk model. Then, a particle swarm optimization algorithm is used iteratively to generate new simulated heating parameters. A rheological prediction model is then used to calculate a new predicted trend sequence, and the quality of this solution is evaluated based on the aforementioned cost function. This process is repeated until a preset stopping condition is met, such as reaching the maximum number of iterations or the cost function no longer showing a significant decrease over multiple iterations. When the algorithm stops, the required heating parameter adjustment amount is obtained.

[0065] This invention also provides an intelligent optimization system for manufacturing processes based on multimodal data fusion. This system is used to implement the above-mentioned method and includes: The identification module acquires the infrared thermal imaging video stream during the production process, splits the infrared thermal imaging video stream into multiple image frames, analyzes the image frames to obtain the gap width between the product outline and the mold, and if there is a part in the last image frame where the gap width is greater than the critical threshold, the gap width measured in each image frame is combined with the corresponding timestamp to generate the actual trend sequence of gap width changing over time.

[0066] The prediction module establishes a rheological prediction model based on a time-series neural network. The rheological prediction model predicts the trend sequence of gap width changes based on the heating parameters of each heating zone and the material properties of the product.

[0067] The positioning module divides the actual trend sequence into multiple flow intervals, analyzes these intervals to determine abnormal heating zones and the direction of heating parameter adjustment.

[0068] The analysis module perturbs the heating parameters of the abnormal heating zone based on the adjustment direction, inputs the heating parameters of each heating zone after perturbation and the material properties of the product into the rheological prediction model, so as to reverse locate the target heating zone that caused the actual trend sequence, determine its corresponding heating parameter adjustment amount, and correct the heating parameters of the target heating zone based on the adjustment amount.

[0069] It should be noted that, although specific algorithm models, parameter settings, and threshold selections are mentioned in the above embodiments of the present invention, such as critical thresholds for judging quality risks, zero-mean normalized cross-correlation algorithms for feature point tracking, LSTM temporal neural networks for establishing rheological prediction models, particle swarm optimization algorithms for reverse localization, and the three-standard-deviation rule for constructing standard trend ranges, this is not a limitation on the scope of protection of the present invention. Those skilled in the art should understand that the above specific implementations are merely examples. Without departing from the core idea of ​​the present invention, flexible substitutions or combinations can be made according to different production scenarios, product materials, equipment conditions, computing resources, and accuracy requirements.

[0070] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A manufacturing process intelligent optimization method based on multimodal data fusion, characterized in that, include: Acquire infrared thermal imaging video streams during the production process, break the infrared thermal imaging video streams into multiple image frames, and analyze the image frames to obtain the gap width between the product outline and the mold. If the gap width in the last image frame is greater than the critical threshold, the gap width measured in each image frame is combined with the corresponding timestamp to generate an actual trend sequence of gap width changes over time. A rheological prediction model is established based on a time-series neural network. The rheological prediction model predicts the trend sequence of gap width changes based on the heating parameters of each heating zone and the material properties of the product. The actual trend sequence is divided into multiple flow intervals, and the flow intervals are analyzed to determine abnormal heating zones and the direction of heating parameter adjustment. Based on the adjustment direction, the heating parameters of the abnormal heating zone are perturbed, and the heating parameters of each heating zone after perturbation and the material properties of the product are input into the rheological prediction model to reverse locate the target heating zone that caused the actual trend sequence and determine its corresponding heating parameter adjustment amount. The heating parameters of the target heating zone are corrected based on the adjustment amount.

2. The method according to claim 1, characterized in that, Analyzing image frames to obtain the gap width between the product outline and the mold includes the following steps: Each image frame is processed for grayscale and distortion correction to obtain a standard image. In the first standard image, the initial outline of the product and the mold outline are located based on the image recognition algorithm. Multiple feature points are located in the initial outline, and the Euclidean distance between the feature points and the mold outline is used as the initial gap width. Based on the moving speed of the production equipment and the pixel features of the feature points, the feature points are mapped to each subsequent standard image. The displacement distance of the same feature point in the vertical or horizontal direction is calculated based on adjacent standard images. The gap width of the feature point in the previous standard image is subtracted from the displacement distance. The gap width of the feature point in the current standard image is updated and obtained in a recursive manner.

3. The method according to claim 2, characterized in that, Mapping feature points to each subsequent standard image includes the following steps: In the current standard image, the normal direction of the initial contour line position of the feature point is calculated, and the pixel grayscale sequence of the feature point is extracted along the normal direction. Based on the moving speed of the production equipment, the predicted coordinates of the feature point in the next standard image are estimated. With the predicted coordinates as the center, combined with the product flow inertia, a search area extending outward along the normal direction is constructed. Within the search area of ​​the next standard image, the pixel grayscale sequence of each pixel is traversed and obtained, and the similarity with the pixel grayscale sequence of the feature point in the current standard image is calculated. The pixel with the highest similarity is selected as the visual coordinate. The confidence weight is set according to the image sharpness. Based on the confidence weight, the predicted coordinates and the visual coordinates are weighted and fused to obtain the position of the feature point in the next standard image.

4. The method according to claim 1, characterized in that, Establishing a rheological prediction model based on a temporal neural network includes the following steps: The heating parameters of each heating zone are combined into a parameter sequence. The parameter sequence is convolved according to different time sliding windows to extract the thermal accumulation feature sequence at different time scales. The hidden layer state of the temporal neural network at the current time step is used as the query vector and the thermal accumulation feature sequence is used as the key vector. By calculating the similarity between the query vector and the key vector, an attention score distribution representing the degree of influence of each historical time step on the current rheological state is generated. The thermal cumulative feature sequence is weighted and summed using the attention score distribution to obtain the thermal response context vector. The thermal response context vector is then concatenated with the product material properties to construct an enhanced feature vector. This enhanced feature vector is then input into the recurrent unit of a temporal neural network to obtain the trend sequence of gap width changes.

5. The method according to claim 1, characterized in that, Dividing the actual trend sequence into multiple flow intervals and analyzing these intervals to determine abnormal heating zones and the adjustment direction of heating parameters includes the following steps: A standard process sequence library is constructed, which includes standard trend sequences of ideal rheological processes with different material properties. Based on the data form, the standard trend sequences are divided into multiple flow intervals. At the same time, the allowable fluctuation range of each sequence point in the standard trend sequence is determined. Based on the allowable fluctuation range, the standard trend sequence is converted into a standard trend range. The portion of the actual trend sequence that exceeds the standard trend range is identified as an abnormal flow interval. The flow interval and statistical characteristics of the abnormal flow interval are extracted, and the material properties, flow interval and statistical characteristics are combined into an abnormal signature. Historical adjustment records that match the abnormal signature are retrieved from a preset knowledge base. Based on the matching historical adjustment records, the adjustment direction of the heating zone and heating parameters that caused the abnormality is determined.

6. The method according to claim 5, characterized in that, Determining the adjustment direction of the heating zone and heating parameters causing the anomaly based on matching historical adjustment records includes the following steps: The frequency ratio caused by each heating zone and the frequency ratio of the adjustment direction are calculated from the historical adjustment records that match the abnormal signature of the current actual trend sequence. The heating zone with the highest frequency ratio and the adjustment direction are output as the matching result.

7. The method according to claim 5, characterized in that, The perturbation of the heating parameters of the abnormal heating zone based on the adjustment direction includes the following steps: Based on the adjustment direction, a primary perturbation is applied to the abnormal heating zone. Simultaneously, based on the equipment thermodynamic crosstalk model, a secondary perturbation with amplitude decreasing with distance is applied to the heating zones adjacent to the abnormal heating zone to generate simulated heating parameters. The simulated heating parameters are used as the optimization target. Through iterative search, a set of simulated heating parameters is found that minimizes the difference between the predicted trend sequence output by the rheological prediction model and the actual trend sequence. The adjustment amount is determined based on the difference between the simulated heating parameters and the current heating parameters.

8. The method according to claim 7, characterized in that, Iterative search includes the following steps: A cost function is defined to measure the difference between the predicted trend sequence and the actual trend sequence. The simulated heating parameters are used as optimization variables, and the initial simulated heating parameters are used as the search starting point. The particle swarm optimization algorithm is used to iteratively solve for the simulated heating parameters that minimize the cost function.

9. A manufacturing process intelligent optimization system based on multimodal data fusion, used to implement the method as described in any one of claims 1-8, characterized in that, The system includes: The identification module acquires the infrared thermal imaging video stream during the production process, splits the infrared thermal imaging video stream into multiple image frames, analyzes the image frames to obtain the gap width between the product outline and the mold, and if there is a gap width in the last image frame that is greater than the critical threshold, the gap width measured in each image frame is combined with the corresponding timestamp to generate the actual trend sequence of gap width changing over time. The prediction module establishes a rheological prediction model based on a time-series neural network. The rheological prediction model predicts the trend sequence of gap width changes based on the heating parameters of each heating zone and the material properties of the product. The positioning module divides the actual trend sequence into multiple flow intervals, analyzes these flow intervals to determine abnormal heating zones and the direction of heating parameter adjustment; The analysis module perturbs the heating parameters of the abnormal heating zone based on the adjustment direction, inputs the heating parameters of each heating zone after perturbation and the material properties of the product into the rheological prediction model, so as to reverse locate the target heating zone that caused the actual trend sequence, determine its corresponding heating parameter adjustment amount, and correct the heating parameters of the target heating zone based on the adjustment amount.

Images