Glass processing process failure simulation method based on deep reinforcement learning

Abstract

The application belongs to the technical field of deep learning, and discloses a glass processing process failure simulation method based on deep reinforcement learning. Historical failure data and high-dimensional parameter data of a glass processing process are collected. A state space and an action space of a glass processing failure simulation environment are constructed. A reward function of a deep reinforcement learning agent is designed. The deep reinforcement learning agent is trained based on the constructed state space and action space and the reward function. In the training process, random noise and parameter drift in the glass processing process are simulated by dynamically adjusting state transition probabilities. Failure simulation of the glass processing process is performed to generate simulation results, including real-time failure path prediction and risk assessment reports. The deviation between the simulation results and the currently collected failure data is calculated, and the policy network of the deep reinforcement learning agent is updated in reverse, thereby realizing the glass processing process failure simulation based on deep reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically, to a method for simulating glass processing failures based on deep reinforcement learning. Background Technology

[0002] With the rapid development of intelligent manufacturing, glass processing, as a key link in the field of precision materials manufacturing, plays an indispensable role in industries such as construction, automobiles, electronic displays, and optical devices. Traditional glass processing involves multiple steps, including cutting, grinding, heat treatment, and surface modification. Failure simulation of these steps has become a core technology for optimizing production efficiency, reducing defect rates, and improving product quality. In recent years, failure simulation methods based on finite element analysis (FEA) and Monte Carlo simulation have been widely applied in the glass processing field. These methods simulate failure mechanisms such as stress distribution, crack propagation, and thermal shock through numerical modeling, helping engineers predict potential risks and adjust process parameters. However, existing glass processing failure simulation technologies lack adaptive learning capabilities when dealing with real-time optimization problems in high-dimensional parameter spaces, resulting in simulation results that are difficult to meet actual production needs in terms of accuracy and efficiency.

[0003] Specifically, traditional FEA and Monte Carlo methods rely on static models and preset parameters, making it impossible to automatically learn and adjust simulation strategies from historical failure data. When faced with multivariate coupling in glass processing (such as the dynamic interaction of high-dimensional parameters like temperature, stress, and material inhomogeneity), simulation models struggle to optimize parameter configurations in real time, often resulting in prediction bias rates as high as 20%-30%. For example, during continuous hot bending or laser cutting, parameter drift and environmental noise can trigger nonlinear failure paths. Existing methods require manual model calibration to adapt to new scenarios, but this manual intervention not only prolongs the simulation cycle but also introduces the risk of human error, hindering efficient iterative optimization. In large-scale production environments, this lack of adaptive learning is further amplified, resulting in poor generalization performance of simulation models. They are unable to quickly adapt to changes in new glass alloys or complex process chains, ultimately leading to delays in batch defect detection, resource waste, and production interruptions, severely restricting the intelligent transformation of the glass processing industry.

[0004] In view of this, the present invention proposes a glass processing failure simulation method based on deep reinforcement learning to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of existing technologies and achieve the above objectives, this invention provides the following technical solution: a glass processing failure simulation method based on deep reinforcement learning, comprising: Step S1: Collect historical failure data and high-dimensional parameter data of the glass processing process; Step S2: Based on historical failure data and high-dimensional parameter data, construct the state space and action space of the glass processing failure simulation environment; wherein, the state space is represented by dynamic vectors of high-dimensional parameter data, and the action space includes a discrete set of strategies for adjusting simulation parameters; Step S3: Design the reward function for the deep reinforcement learning agent; wherein the reward function is calculated based on the matching degree between the simulation prediction error and the actual failure data, and incorporates an adaptive penalty term; Step S4: Based on the constructed state space, action space, and reward function, train the deep reinforcement learning agent; during the training process, the random noise and parameter drift in the glass processing process are simulated by dynamically adjusting the state transition probability. Step S5: Using the trained deep reinforcement learning agent, perform failure simulation of the glass processing process and generate simulation results, including real-time failure path prediction and risk assessment report. Step S6: Based on the simulation results, calculate the deviation between the simulation results and the currently collected failure data, and update the policy network of the deep reinforcement learning agent in reverse.

[0006] Preferably, the historical failure data includes crack propagation records, thermal shock events, and defect distribution characteristics; High-dimensional parameter data include temperature distribution, stress field, and material inhomogeneity indices.

[0007] Preferably, the method for constructing the state space and action space of the glass processing failure simulation environment based on historical failure data and high-dimensional parameter data includes: Historical failure data is decomposed into time series to extract the spatiotemporal sequence features of failure events, and then fused with high-dimensional parameter data in a multimodal manner to generate an initial vector representation of the state space. The high-dimensional parameter data in the initial vector representation is reduced in dimension, and the nonlinear coupling components are retained to form a feature set after dimension reduction. Each feature in the feature set corresponds to a key uncertainty factor in glass processing. The obtained feature set is defined as a dynamic vector representation of the state space; The action space is defined as a set of vectors for parameter adjustment, where each action vector is a combination of temperature offset, stress correction and material property fine-tuning, and the validity boundary of each action vector is determined. The state space and action space are jointly embedded in the Markov decision process framework, and the state transition process in the framework satisfies the physical constraints of glass processing and random noise disturbances.

[0008] Preferably, the method for dimensionality reduction of the high-dimensional parameter data in the initial vector representation includes: By applying nonlinear principal component analysis, we can capture the nonlinear relationships between high-dimensional parameter data and generate a dimension-reduced embedding space. The first k principal components are retained, where the value of k is adaptively determined based on the cumulative variance contribution rate of the principal components. Domain knowledge constraints are introduced during the dimensionality reduction calculation.

[0009] Preferably, the method for designing the reward function of the deep reinforcement learning agent includes: The difference between the failure path predicted in real time by the simulation and the actual failure data is measured and used as the basic reward item. A quantitative index for the nonlinear coupling effect between high-dimensional parameter data is introduced to construct an adaptive penalty term; The basic reward term, adaptive penalty term, and evaluation of the generalization ability of the deep reinforcement learning agent are all used as optimization objectives of the reward function, and the contribution weight of each objective is balanced by weighted aggregation.

[0010] Preferably, the method for constructing the adaptive penalty term includes: Real-time monitoring of parameter drift in high-dimensional parameter data; the trace of its covariance matrix is ​​used as a comprehensive index to quantify the nonlinear coupling between parameters and the overall drift intensity. Based on comprehensive indicators, the weight coefficient of the adaptive penalty term is dynamically calculated using an exponential decay function; The adaptive penalty term with weighted coefficients is integrated into the reward function.

[0011] Preferably, the method for training the deep reinforcement learning agent includes: Initialize the policy network and value network of the deep reinforcement learning agent, where the policy network uses residual connections to process the high-dimensional parametric data dynamic vector representation of the state space; During the training iteration, an experience replay buffer is introduced to store state-action-reward tuples, and a priority sampling mechanism is used to prioritize the processing of samples with high temporal difference errors. The learning rate is dynamically adjusted during the training process, and the step size is adaptively scaled and updated based on the gradient norm of the simulation prediction error. An actor-critic training framework is adopted, in which an attention layer is embedded in the critic network to capture long-term dependencies between high-dimensional parameter data.

[0012] Preferably, the method of introducing an experience replay buffer to store state-action-reward tuples includes: The experience replay buffer is designed as a priority-based queue structure, and sampling priority is assigned according to the temporal difference error of the tuple; Establish a periodic buffer cleanup mechanism to adapt to the evolution of dynamic vector representations of high-dimensional parameter data in the state space by removing outdated historical tuples; In the priority-based sampling process, importance sampling weights are applied to correct sampling distribution bias.

[0013] Preferably, the method for performing failure simulation of the glass processing process and generating simulation results using a trained deep reinforcement learning agent includes: Real-time high-dimensional parameter data is input into a deep reinforcement learning agent to generate a dynamic vector representation of the high-dimensional parameter data. A sequence of parameter adjustment actions is generated through a policy network to simulate the evolution of failure paths in the glass processing process. Multi-step prediction based on policy networks is used to assess the probability of failure risk in future time steps and generate confidence intervals for failure prediction. The predicted simulation results are transformed into real-time failure path prediction and risk assessment reports. The real-time failure path prediction includes a failure risk heat map and a failure path evolution map, which identify high-risk areas and critical parameter points that are sensitive to failure during the processing. The risk assessment report includes a risk level assessment based on confidence intervals.

[0014] Preferably, the method for calculating the deviation between the simulation results and the currently collected failure data, and then updating the policy network of the deep reinforcement learning agent in reverse, includes: The sequence alignment error between the failure path predicted by the quantification simulation and the actual failure data is used as a deviation measure for the policy network update. The bias metric is incorporated as an additional training signal into the reward function, and the parameter weights of the policy network are updated through backpropagation. The meta-learning optimization process is introduced to quickly adjust the strategy network parameters.

[0015] The technical effects and advantages of the glass processing failure simulation method based on deep reinforcement learning in this invention are as follows: By systematically collecting historical failure data and high-dimensional parameter data, a complete dataset containing multi-dimensional information such as temperature distribution and stress field is constructed. Then, using multimodal fusion and dimensionality reduction techniques, the high-dimensional parameter data is transformed into a dynamic vector representation of state space with clear physical meaning. Simultaneously, an action space for parameter adjustment is defined and embedded into a Markov decision process framework that satisfies physical constraints. By designing a multi-objective reward function that integrates basic reward terms, adaptive penalty terms, and generalization ability assessment, parameter drift is monitored in real time, and penalty weights are dynamically adjusted, effectively balancing simulation accuracy and stability requirements. In terms of the training mechanism, an actor-critic framework combined with priority sampling and experience replay buffer optimization is adopted to ensure the efficiency and stability of the training process. Finally, the trained deep reinforcement learning agent can perform real-time failure simulations, generate visual reports including risk heatmaps and confidence intervals, and continuously optimize the policy network through a continuous learning mechanism. This solution achieves a complete technical closed loop from data acquisition and model training to simulation application, significantly improving the accuracy, adaptability, and practical value of glass processing failure simulation. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the glass processing failure simulation method based on deep reinforcement learning according to the present invention; Figure 2 This is a schematic diagram of the method for constructing the state space and action space of a glass processing failure simulation environment based on historical failure data and high-dimensional parameter data in this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 and Figure 2 In this embodiment of the invention, a glass processing failure simulation method based on deep reinforcement learning includes: Step S1: Collect historical failure data and high-dimensional parameter data of the glass processing process; Step S2: Based on the historical failure data and high-dimensional parameter data collected in step S1, construct the state space and action space of the glass processing failure simulation environment; wherein, the state space is represented by dynamic vectors of high-dimensional parameter data, and the action space includes a discrete set of strategies for adjusting simulation parameters; Step S3: Design the reward function for the deep reinforcement learning agent; wherein the reward function is calculated based on the matching degree between the simulation prediction error and the actual failure data, and incorporates an adaptive penalty term to optimize the nonlinear effects caused by high-dimensional parameter coupling; Step S4: Using a deep reinforcement learning algorithm, the deep reinforcement learning agent is trained based on the state space and action space constructed in step S2 and the reward function designed in step S3; wherein, during the training process, the random noise and parameter drift in the glass processing process are simulated by dynamically adjusting the state transition probability. Step S5: Using the deep reinforcement learning agent trained in step S4, perform failure simulation of the glass processing process and generate simulation results, including real-time failure path prediction and risk assessment report. Step S6: Based on the simulation results obtained in step S5, calculate the deviation between the simulation results and the currently collected failure data, and update the policy network of the deep reinforcement learning agent in reverse. This achieves adaptive learning and continuous optimization of the deep reinforcement learning agent.

[0019] To address the issues of complex data dimensions and unclear characteristics in glass processing failure simulation, this paper establishes standardized data acquisition specifications by clearly defining historical failure data as including crack propagation records, thermal shock events, and defect distribution characteristics, and high-dimensional parameter data as including temperature distribution, stress field, and material inhomogeneity indicators. This ensures the comprehensiveness and representativeness of the simulation's foundational data, achieving complete coverage of key influencing factors in the glass processing process and laying a reliable data foundation for subsequently constructing an accurate state space. The historical failure data includes crack propagation records, thermal shock events, and defect distribution characteristics. High-dimensional parameter data include temperature distribution, stress field, and material inhomogeneity indices.

[0020] Historical failure data can be extracted from factory logs. Crack propagation records document the formation, growth, and propagation paths of cracks in glass materials during processing. These records typically include indicators such as crack initiation point, propagation velocity, direction (e.g., radial and linear), length variation over time, and branching patterns. This data is collected because glass is fragile and easily broken; in glass processing, cracks are usually caused by mechanical stress (e.g., during cutting or grinding) or thermal gradients (e.g., during heat treatment), representing key failure modes.

[0021] Thermal shock events refer to the sudden stress caused by rapid temperature changes in glass, leading to failures such as cracking or breakage. They generally include event details such as temperature difference, exposure time, impact location (e.g., edge vs. center), and the result (e.g., complete breakage or microcracks).

[0022] Defect distribution characteristics are used to describe defects on the surface or volume of glass. In glass processing, defects are usually caused by impurities, uneven grinding, or chemical reactions, affecting overall quality and failure risk. Therefore, defect distribution characteristics generally include defect density (e.g., number per square centimeter), size distribution (e.g., average diameter), aggregation (e.g., uniformity and aggregation), and type (e.g., surface and interior).

[0023] High-dimensional parameter data refers to multivariate inputs that affect the glass processing environment. Temperature distribution refers to the spatiotemporal changes in temperature of the glass material or processing environment. Uneven heating can lead to failure, such as during annealing or bending. It is usually represented by a field (e.g., a two-dimensional / three-dimensional map showing hot spots or gradients). It generally includes average temperature, gradient, and fluctuation. The acquisition method can be to use an infrared camera to capture a temperature thermal map of the glass surface, recording the value once per second.

[0024] In glass processing, residual stress generated by cooling or applying loads may lead to crack initiation if it exceeds the material's limits. Therefore, stress fields are used to describe the distribution of mechanical stresses (such as tension, compression, and shear) inside the glass. They are usually represented by vector fields in terms of magnitude, direction, and type. The acquisition method can be to use photoelastic imaging to capture the stress concentration around the tool contact point and generate a three-dimensional stress map.

[0025] In glass processing, inhomogeneities (such as raw material mixing) can lead to weak points and uneven damage. Therefore, material inhomogeneity indicators are used to quantify changes in glass composition or structure, such as density fluctuations, refractive index changes, or impurity concentrations. These indicators may include variations in thickness, chemical composition gradients, or microstructural heterogeneity. Data can be collected through spectral or microscopic analysis to detect these changes.

[0026] To address the ineffectiveness of traditional simulation methods in handling dynamic coupling of high-dimensional parameters, this paper constructs a physically meaningful state-space dynamic vector representation by performing temporal decomposition and spatiotemporal sequence feature extraction on historical failure data and fusing it with high-dimensional parameter data in a multimodal manner. The action space is defined as a combination of temperature offset, stress correction, and material property fine-tuning. Physical constraints are embedded through a Markov decision process framework, enabling accurate description and effective control of the dynamic characteristics of the glass processing process. The method for constructing the state space and action space of the glass processing failure simulation environment based on historical failure data and high-dimensional parameter data includes: Historical failure data is decomposed temporally to extract the spatiotemporal sequence features (feature representations in the time and spatial dimensions, used to describe the dynamic evolution of failure events during processing) of failure events (specific events in historical failure data involving defects, performance degradation, or processing failures in glass products). Specifically, firstly, time windows are divided according to the processing cycle (e.g., heating-holding-cooling, corresponding to the physical process of periodic loading and release of internal thermal stress in glass), and the failure events are decomposed temporally to align with the physical processes. Within each window, the extracted features have a clear physical interpretation (e.g., specific frequency bands are extracted to correspond to periodic thermal shocks introduced by the start-stop of cooling fans or power fluctuations of heating elements; or, to capture stress peaks and their changing trends close to the fracture toughness of glass, the local maxima sequence and its gradient of the stress signal can be extracted). Simultaneously, in the spatial dimension, the glass workpiece is meshed, and the spatial features of each mesh cell (e.g., the main direction of cracks and spatial clustering of defect distribution) are extracted and associated with physical interpretations such as stress concentration areas of the workpiece. Finally, a spatiotemporal graph with embedded physical information is constructed to represent the spatiotemporal fusion features. This graph uses grid cells as nodes, whose attributes are time-series physical quantities. The connection weights between nodes are defined by process knowledge such as heat transfer efficiency. A graph neural network is used to learn from this structure, generating spatiotemporal sequence features of local states and surrounding influences. This provides high-fidelity state input for subsequent failure simulations.

[0027] The system then performs multimodal fusion with high-dimensional parameter data to generate an initial vector representation of the state space. Specifically, it uses spatiotemporal sequence features as queries and high-dimensional parameter data as keys and values. It calculates the relevance score (similarity) between the query (spatiotemporal feature) and each key (high-dimensional parameter), normalizes the score into weights, and uses these weights to represent the importance of each high-dimensional parameter to the current spatiotemporal state. The calculated attention weights are then used to perform a weighted sum of the values ​​(high-dimensional parameters) to obtain a context-aware representation. This representation is no longer a simple list of all parameters, but rather the most relevant information "filtered" out by the spatiotemporal features. Finally, this representation is concatenated with the original spatiotemporal sequence features to generate the final initial vector of the state space.

[0028] The high-dimensional parameter data in the initial vector representation is reduced in dimension, and the nonlinear coupling components are retained to form a feature set after dimension reduction. Each feature in the feature set corresponds to a key uncertainty factor in glass processing. Key uncertainty factors include, but are not limited to, thermal stress concentration factor, material toughness critical threshold gradient, and cooling rate fluctuation index. The obtained feature set is defined as the dynamic vector representation of the state space; and the dynamic vector representation of the state space is normalized to map the values ​​of each dimension to a unified numerical range to ensure the stability and convergence efficiency of deep reinforcement learning agent training.

[0029] The action space is defined as a set of vectors for parameter adjustment. Each action vector is a combination of temperature offset, stress correction and material property fine-tuning. In other words, each action vector is a multi-dimensional control command.

[0030] The Monte Carlo tree search method is used to determine the validity boundaries of each action vector, which is then used to determine the safe range of parameter adjustments that will not cause process instability. Specifically, in a simulation environment, the consequences of different operations (action vectors) are deduced like playing chess. Actions with a high success rate are considered to be within the boundaries and are safe and effective, while actions with a high failure rate are considered to be outside the boundaries and are dangerous or ineffective. This process identifies a safe and effective range of parameter adjustments.

[0031] By embedding the state space and action space together into a Markov decision process framework, a complete and trainable decision simulation environment is formed. Furthermore, the state transition process within the framework satisfies the physical constraints of glass processing and random noise perturbations. Within this framework, a state transition function is designed that not only follows the basic physical constraints of glass processing but also simulates parameter drift and uncertainty in the real production environment by introducing random noise perturbations (such as Gaussian white noise or process noise based on historical data), thus creating a simulation training environment that combines physical realism with stochastic dynamics.

[0032] To address the issues of redundant information and noise interference in high-dimensional parameter data, a nonlinear principal component analysis (PCA) method is applied to capture the nonlinear relationships between parameters, and the number of principal components is adaptively determined based on the cumulative variance contribution rate. Domain knowledge constraints are introduced during the dimensionality reduction process to ensure the retention of key nonlinear coupling components affecting failure. This achieves the goal of maximizing the preservation of key process information while reducing computational complexity, thus improving the quality and efficiency of state-space representation. The method for dimensionality reduction of high-dimensional parameter data in the initial vector representation includes: By applying nonlinear principal component analysis, we can capture the nonlinear relationships between high-dimensional parameter data and generate a dimension-reduced embedding space. The core objective is to capture the curvature relationships and higher-order interactions between parameters such as temperature, stress, and material inhomogeneity, thereby generating a low-dimensional embedding space that can preserve nonlinear coupling relationships.

[0033] The first k principal components are retained, where the value of k is adaptively determined based on the cumulative variance contribution rate of the principal components. It is usually set to the minimum k value that can explain more than 95% of the total variance of the original data, so as to ensure that the key information affecting failure is retained to the maximum extent while reducing complexity.

[0034] In the dimensionality reduction calculation process, domain knowledge constraints are introduced, including at least one of the laws of thermodynamics, the law of conservation of momentum, and the law of conservation of mass. Prior knowledge in the glass processing field is integrated into the deep reinforcement learning agent as a physical constraint. For example, a regularization term based on the heat conduction equation or stress-strain constitutive relation is added to the loss function to guide the learned embedding space to be not only data-driven but also in accordance with physical laws, avoiding the generation of physically infeasible or invalid state representations.

[0035] After dimensionality reduction is completed, the quality of dimensionality reduction is verified by the consistency index of reconstruction error and failure prediction, which is used to evaluate whether the dimensionality reduction process is effective and whether the information is sufficiently preserved.

[0036] To address the issue that a single optimization objective cannot balance simulation accuracy and stability, this paper proposes a method that uses the difference between simulation predictions and actual failure data as a basic reward term, introduces a quantification index of nonlinear coupling effects to construct an adaptive penalty term, and incorporates generalization ability evaluation as a common optimization objective. A weighted aggregation approach is used to balance the contribution weights of each objective, achieving a comprehensive performance optimization mechanism that balances system stability and model generalization ability while ensuring simulation accuracy. The method for designing the reward function of the deep reinforcement learning agent includes: The difference between the simulated failure path predicted in real time and the actual failure data is calculated and used as the base reward term R_base to evaluate the simulation accuracy. The simulated failure path is a time series data set, referring to the sequence of most likely failure events (such as crack initiation and propagation) within a future period, deduced by the trained deep reinforcement learning agent after receiving the current state (i.e., a dynamic vector composed of high-dimensional parameter data). For example, within the next 5 seconds, a crack will propagate 2 mm from position A along direction B, while a new microcrack has an 80% probability of initiating in the high-stress zone C. The actual failure data refers to the sequence of failure events actually observed and recorded during the real glass processing. Calculating the difference between the two is essentially comparing the similarity of the two time series. Because the failure path may exhibit nonlinear deformations in time and space (e.g., the predicted crack propagation rate may be slightly faster or slower than the actual rate), simple Euclidean distance is not ideal and yields poor results. Therefore, dynamic time warping distance is needed as a metric. Specifically, the failure path predicted in real time by simulation is denoted as sequence A, and the actual failure data collected is denoted as sequence B. The Euclidean distance between each point in sequence A and each point in sequence B is calculated to form a matrix. In this matrix, a path from the top left corner to the bottom right corner is found such that the sum of the distances of all points on the path is minimized. This path is the optimal alignment, and the minimum cumulative distance is the difference metric.

[0037] A quantitative index for the nonlinear coupling effect between high-dimensional parameter data is introduced to construct an adaptive penalty term; The basic reward term, adaptive penalty term, and evaluation of the generalization ability of the deep reinforcement learning agent are all used as optimization objectives of the reward function, and the contribution weight of each objective is balanced by weighted aggregation.

[0038] The evaluation of the generalization ability of a deep reinforcement learning agent refers to assessing its performance in unseen glass processing scenarios. Specifically, during training, the current deep reinforcement learning agent policy is periodically (e.g., every 1000 training steps) run in a completely independent test environment that has never been used in training. These environments simulate working conditions different from the training data (e.g., different glass geometries, new alloy material parameters, and unseen environmental noise patterns). The average performance of the deep reinforcement learning agent in these test environments (e.g., average success rate or average negative error) is calculated and used as the evaluation of the generalization ability R_gen of the deep reinforcement learning agent. The better the performance, the higher the R_gen value.

[0039] Therefore, the reward function = w1×R_base+w2×P_adaptive+w3×R_gen, where w1, w2 and w3 are the weight values ​​corresponding to the weights of the base reward, the adaptive penalty and the generalization ability evaluation. Generally, w1>w2 and w1>w3, w2<0, w3>0.

[0040] After the reward function was designed, its convergence and robustness in a glass processing environment containing random noise and parameter drift were verified through simulation experiments.

[0041] To address the simulation inaccuracy caused by parameter drift during processing, this paper uses the covariance matrix trace of high-dimensional parameter data in real time as a comprehensive index of drift intensity, and dynamically calculates the weight coefficients of the adaptive penalty term based on an exponential decay function. By integrating the weighted penalty term into the reward function, the optimization direction is automatically adjusted according to the system stability state. When significant parameter drift occurs, simulation stability is prioritized, significantly improving the system's robustness. The method for constructing the adaptive penalty term includes: Real-time monitoring of parameter drift in high-dimensional parameter data; the trace of its covariance matrix is ​​used as a comprehensive index to quantify the nonlinear coupling between parameters and the overall drift intensity. Parameter drift refers to the unexpected and unstable changes in the values ​​of high-dimensional parameters (such as temperature and stress) over time during glass processing. This parameter drift is not caused by changes in process settings, but rather by uncertainties such as equipment aging, environmental interference, and batch-to-batch material variations. For example, during continuous hot bending, the set temperature of the same heating zone remains constant, but the range and frequency of temperature fluctuations around the set value reported by the actual temperature measurement point change.

[0042] The trace of the covariance matrix is ​​used to quantify the cooperative relationship between multiple parameters. That is, it describes the pairwise relationship between all high-dimensional parameters (such as temperature 1, temperature 2, stress 1, stress 2, etc.). Positive values ​​indicate changes in the same direction, and negative values ​​indicate changes in opposite directions.

[0043] The trace of the covariance matrix is ​​the sum of all elements on the main diagonal of the matrix (i.e., the variance of each parameter itself). Its value directly reflects the strength of the overall volatility of all high-dimensional parameters. The larger the trace value, the more unstable the entire processing system, the more severe the nonlinear coupling effect between parameters, and the more serious the drift. It provides a comprehensive assessment of the overall drift intensity.

[0044] Based on the comprehensive index, the weight coefficient of the adaptive penalty term is dynamically calculated through the exponential decay function [in this design, the calculation characteristic of the function is that as the input value (i.e., the comprehensive index) increases, the output value (weight coefficient) decreases rapidly in an exponential form]. The principle is that when instability intensifies, that is, when the comprehensive index increases, the weight of the penalty term calculated by this function will increase sharply, thereby achieving strong penalty for unstable states and ensuring that a stronger penalty is applied when the parameter drift intensity is high. The adaptive penalty term with weighted coefficients is integrated into the reward function, and the basic reward term and stability requirements are balanced through constraint optimization. Overall, within a sliding time window (e.g., the last 10 seconds), all high-dimensional parameter data (e.g., temperature distribution, stress field, etc.) are continuously collected and cached. Then, the covariance matrix formed by these parameter data is calculated, and the trace of this matrix is ​​obtained. A comprehensive index I is obtained, which is substituted into a preset exponential decay function w = w_base + α × exp(β × I) to calculate the adaptive penalty term weight coefficient w at the current time. The calculated dynamic weight coefficient w is multiplied by a basic penalty term P_base to form the final adaptive penalty term P_adaptive = w × P_base. Here, w_base is the base weight, ensuring that there is always a minimum penalty. It can be initially set to 0.1 and then dynamically adjusted using trial and error. α and β are scaling factors greater than 0, used to adjust the sensitivity and output range of the function. P_base can be a quantity positively correlated with the fluctuation of the parameters, such as the mean of the rate of change of the parameters.

[0045] The effect of the adaptive penalty term was tested in a synthetic noise data environment to verify its role in improving the simulation robustness of deep reinforcement learning agents in glass processing.

[0046] After the design of steps S2 and S3 above, the deep reinforcement learning agent is regarded as an intelligent agent. The architecture of this intelligent agent is mainly composed of two key networks, namely the policy network and the value network.

[0047] Overall, for the deep reinforcement learning agent, the policy network is a function, with the state as input and the action as output. Typically, the input layer of a policy network might be a simple, unprocessed state (such as raw readings from several sensors). However, in this design, the input layer must effectively understand and process the specific physical state during glass processing. Therefore, the designed input is a "high-dimensional parameter data dynamic vector representation of the state space." This is a highly condensed and physically meaningful feature vector after multimodal fusion and nonlinear principal component analysis dimensionality reduction. This allows the first layer of the network to learn from a high-level feature rather than raw data, significantly improving learning efficiency and limiting the policy's potential. Similarly, the general output of the policy network is arbitrary, physically meaningless action encoding. In this design, all actions of the policy network's exploration and decision-making are required to be physically interpretable and within a safe range. Therefore, the output after design is a "vector set of parameter adjustments". Each action vector corresponds to a "combination of temperature offset, stress correction and material property fine-tuning", and its range is limited by "validity boundaries". For example, the output layer can use the Tanh activation function to limit the output value to [-1, +1], and then map it to a specific, safe parameter adjustment range through a scaling layer.

[0048] Value networks evaluate the quality of a state or state-action pair, and the reward function defines what constitutes "good." However, general value network learning and prediction objectives are difficult to match with the glass processing process. They struggle to answer questions such as whether the glass will crack during processing, whether the current process parameters are stable, whether the strategy will work in a different environment, and other complex and specialized questions. Therefore, value networks must learn to understand and weigh the three abstract concepts of "accuracy," "stability," and "generalization." The redesigned reward function solves these problems and perfectly meets the requirements.

[0049] In the actor-critic framework, the update direction of the policy network (actors) is guided by the value network (critic). This means that the policy network no longer blindly pursues maximizing accuracy, but instead learns to find an optimal balance between accuracy, stability, and generalization. Its decision-making logic is shaped and guided by the reward function. For example, normally, the critic would only point out high-scoring actions and suggest trying more in that direction. However, a redesigned critic will further analyze these high-scoring actions, such as, "While that action might slightly improve the prediction accuracy, it causes violent oscillations and is overall a bad action; don't update in that direction."

[0050] To address the instability and inefficiency in training deep reinforcement learning agents, this paper proposes a method that initializes a policy network and a value network, uses residual connections in the policy network to handle high-dimensional state inputs, and employs an actor-critic training framework. By combining experience replay buffer priority sampling, dynamic learning rate adjustment, and a critic network attention mechanism, efficient learning of long-term dependencies on high-dimensional parameters is achieved, ensuring the stability and convergence efficiency of the training process. The method for training the deep reinforcement learning agent includes: A deep Q-network architecture (a classic algorithm architecture in deep reinforcement learning) is used to initialize the policy network and value network of the deep reinforcement learning agent. The policy network uses residual connections to process the dynamic vector representation of high-dimensional parameter data in the state space. That is, the policy network uses residual connections to process the complex "dynamic vector representation of high-dimensional parameter data in the state space", ensuring that information can be effectively transmitted in the network. This effectively alleviates the gradient vanishing or exploding problem that occurs when the network layer is too deep, thereby making more accurate decisions. During the training iteration, an experience replay buffer is introduced to store state-action-reward tuples, and a priority sampling mechanism is used to prioritize the processing of samples with high temporal difference errors. The learning rate is dynamically adjusted during the training process, and the step size is adaptively scaled and updated based on the gradient norm of the simulation prediction error. We employ an actor-critic training framework (where the actor is the policy network responsible for executing actions, and the critic is the value network responsible for evaluating the actor's actions; the working mode is that the actor acts in the environment, the critic scores the actor's behavior based on the rewards received, and the actor adjusts their strategy based on the critic's score, that is, updates the network parameters, thus being more stable and efficient than traditional and general methods). We embed an attention layer in the critic network to capture long-term dependencies between high-dimensional parameter data (for example, when evaluating the value of the current state, we focus more on a few key temperature points while ignoring other unimportant parameters, thus making a more accurate judgment).

[0051] To address the issues of outdated data and sampling bias in the experience replay buffer, a priority-based queue structure is designed. Sampling priorities are allocated according to temporal differential errors, and a periodic cleanup mechanism is established to remove outdated tuples. By applying importance sampling weights to correct sampling distribution bias, continuous adaptation of buffer data to the current process state is achieved. This ensures the unbiasedness of gradient estimation while maintaining high-quality training samples. The method of introducing state-action-reward tuples into the experience replay buffer includes: The experience replay buffer is designed as a priority-based queue structure, and sampling priority is assigned according to the temporal difference error of the tuple; Glass processing technology evolves over time, and data collected in its early stages may not match the current state of the technology. Continuing to use this data could mislead the agent into learning incorrect knowledge. Therefore, a periodic buffer cleanup mechanism is established to remove outdated historical tuples to adapt to the evolution of the dynamic vector representation of high-dimensional parameter data in the state space. Simply put, each tuple is timestamped during generation, and if its age (the difference between the current timestamp and the tuple's age) exceeds a preset threshold, it is considered outdated. All outdated tuples are sorted from highest to lowest age, and each cleanup removes only a small portion of the top-ranked tuples (e.g., 5% of the total buffer size). This avoids drastic fluctuations in the training data distribution caused by deleting a large amount of experience at once, ensuring the stability of the training process.

[0052] While priority sampling can accelerate learning, it disrupts the original uniform distribution of randomly sampled experience, leading to excessive sampling of high-priority samples. This introduces mathematical bias, causing inaccurate updates of neural network parameters and ultimately affecting training stability and convergence. Therefore, in priority-based sampling, importance sampling weights are applied to correct sampling distribution bias and maintain training stability. Simply put, an importance sampling weight (inversely proportional to the original sampling probability of the sample) is calculated for each selected sample. When calculating the error between the predicted and target values, the error caused by each sample is multiplied by its corresponding importance sampling weight. This ensures that the expected update of network parameters remains consistent with uniform sampling, thus maintaining training stability and avoiding training divergence or oscillations caused by sampling bias.

[0053] Specifically, the main implementation steps are as follows: Step A1: Build and initialize the policy network and value network using a deep Q-network. The policy network is designed with residual connections integrated to ensure its powerful ability to handle high-dimensional state vectors.

[0054] Step A2: Run the deep reinforcement learning agent in a glass processing simulation environment. At each time step t, the deep reinforcement learning agent observes the current state s_t (i.e., the dynamic vector representation of high-dimensional parameter data). The policy network outputs an action a_t based on s_t (such as adjusting temperature or stress). The environment executes the action, transitions to the new state s_{t+1}, and feeds back a reward r_t (which is calculated by the defined reward function). This experience tuple (s_t, a_t, r_t, s_{t+1}) is stored in the experience replay buffer as a training sample.

[0055] Step A3: Calculate the temporal difference error for each sample (i.e., the difference between the commentator's predicted value and the actual target value). Samples with larger errors are considered to contain more valuable information, and therefore have higher learning value. Consequently, they are sampled with higher priority, and these high-priority samples are preferentially extracted for training.

[0056] Step A4: In each training iteration, monitor the gradient norm of the loss function (e.g., the mean squared error of the value network). If the gradient norm is large (e.g., >0.2), it indicates that the parameter space is steep and may be in an unstable region. In this case, automatically lower the learning rate and take small steps to update to prevent oscillations. If the gradient norm is small (e.g., ≤0.2), appropriately increase the learning rate to accelerate convergence. This makes the training process more efficient and able to adapt to the non-stationary characteristics of glass processing parameters.

[0057] Step A5: Using the sampled batch data, with the goal of reducing temporal difference error, update the parameters of the value network through gradient descent. Its internal attention layer will automatically learn and focus on the system parameters that are most critical to long-term reward prediction, thereby updating the commentator (value network).

[0058] The update direction of the actor (policy network) is guided by the critic, who evaluates the quality of the actor's movements and generates gradient signals. The policy network then updates its parameters in the direction that will obtain higher evaluation (i.e., higher cumulative reward).

[0059] Step A6; Repeat steps A2 to A5 until the performance of the policy network converges (i.e., the failure simulation predictions it generates meet the preset accuracy and stability requirements), at which point training is complete.

[0060] To address the issue of traditional simulation results lacking intuitiveness and practicality, this method inputs real-time high-dimensional parameter data into a trained agent to generate parameter adjustment action sequences that simulate failure path evolution. Based on a policy network, it performs multi-step prediction and assessment of failure risk probabilities and generates confidence intervals. The prediction results are then transformed into a visual report containing risk heatmaps and path evolution diagrams, achieving a real-time and intuitive display of processing risks and providing direct decision support for process optimization. The method for using a trained deep reinforcement learning agent to perform failure simulations of the glass processing process and generate simulation results includes: Real-time high-dimensional parameter data is input into a deep reinforcement learning agent to generate a dynamic vector representation of the high-dimensional parameter data. A sequence of parameter adjustment actions is generated through a policy network to simulate the evolution of failure paths in the glass processing process. Multi-step prediction is performed based on policy networks to assess the probability of failure risk in future time steps, and confidence intervals for failure prediction are generated by combining Monte Carlo methods. The predicted simulation results are transformed into real-time failure path prediction and risk assessment reports. The real-time failure path prediction includes a visualized failure risk heat map and failure path evolution map, which identifies high-risk areas and critical parameter points sensitive to failure during the processing. The risk assessment report includes a risk level assessment based on confidence intervals.

[0061] In detail, a sensor network at the glass processing site collects high-dimensional parameter data such as temperature, stress, and material thickness in real time. This data is used to generate a dynamic vector representation of the current state space, which is then input into a trained deep reinforcement learning agent. The policy network adjusts a sequence of actions based on the current input and output parameters, including specific control commands such as temperature adjustment, pressure change, and speed adjustment coefficient. These action sequences are executed in a simulation environment to simulate the state evolution of the glass workpiece over a future period (such as the next processing cycle). Through iterative processing, a complete failure path evolution trajectory is generated. The policy network performs multi-step predictions, considering the processing state changes over the next 5-10 time steps. For each future time step, the probability of failure events such as cracking or breakage is calculated using Monte Carlo simulation. The Luo method involves conducting extensive random simulations while considering process noise, statistically analyzing all simulation results to generate a distribution of failure probabilities, and calculating 95% confidence intervals based on this distribution as a risk assessment report. The glass workpiece surface is divided into grids, and each grid is colored according to the predicted failure probability: red indicates high-risk areas, and green indicates safe areas, generating a failure risk heatmap. Arrows and curves are used to display the most likely failure propagation paths, marking key inflection points and accelerated propagation areas as a failure path evolution diagram. The process parameters with the greatest impact on failure prediction are identified, and their sensitivity coefficients are quantified, enabling analysis of sensitive points for key parameters. Based on the above analysis, a complete report including risk level (high, medium, low), main risk factors, and recommended control measures is provided, denoted as the risk assessment report.

[0062] During the simulation, the deviation between the simulation prediction results and the actual failure data collected is monitored in real time. When the deviation exceeds the preset threshold, the parameter fine-tuning mechanism is triggered to maintain the prediction accuracy.

[0063] The method for calculating the deviation between the simulation results and the currently collected failure data, and then updating the policy network of the deep reinforcement learning agent in reverse, includes: The dynamic time warping distance [used to accurately compare the differences between simulation-predicted failure paths (such as predicted crack propagation processes) and actual collected failure data (such as actually observed crack propagation)] quantifies the sequence alignment error between simulation-predicted failure paths and actual collected failure data, serving as a measure of bias in policy network updates; By incorporating the bias metric as an additional training signal into the reward function and updating the parameter weights of the policy network through backpropagation using the policy gradient algorithm, the adaptability to newly acquired data is enhanced. The introduction of a meta-learning optimization process (in layman's terms, an advanced training mechanism that teaches deep reinforcement learning agents "how to learn," enabling them to quickly adjust with only a small number of samples when faced with novel glass alloys) allows deep reinforcement learning agents to rapidly adjust policy network parameters to adapt to new failure modes based on a small amount of processing data for novel glass alloys. Establish a periodic performance evaluation process and use cross-validation to test the generalization performance of the updated policy network on unseen process chains to ensure the effectiveness of the continuous optimization process.

[0064] Specifically, the implementation steps are as follows: Step B1: Align the timestamps of the simulation-predicted failure path sequence with the actual collected failure data sequence, normalize the sequences to eliminate the influence of dimensions; apply the dynamic time warping distance algorithm to find the optimal alignment path between the two sequences, calculate the cumulative distance along this optimal path as the deviation metric, the smaller this distance value, the better the prediction matches the reality, and convert it into a standardized deviation signal, set a deviation threshold, only significant deviations exceeding the threshold will trigger network updates; Step B2: Based on the original reward function, add an additional reward term based on the deviation signal. If the deviation decreases, a positive reward is given; if the deviation increases, a negative reward is given. Update the parameter weights of the policy network through backpropagation using the policy gradient algorithm. Step B3: Train on processing data of various glass alloys to learn the general mapping relationship between different material properties and failure modes; when a new glass alloy appears, extract a small number of its processing samples, perform gradient updates in a small number of steps based on the prior knowledge obtained by meta-learning, and fuse the adapted network parameters with the original knowledge to maintain the predictive ability for the original glass alloys.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0066] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0067] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A failure simulation method for glass processing based on deep reinforcement learning, characterized in that, include: Step S1: Collect historical failure data and high-dimensional parameter data of the glass processing process; Step S2: Based on historical failure data and high-dimensional parameter data, construct the state space and action space of the glass processing failure simulation environment; wherein, the state space is represented by dynamic vectors of high-dimensional parameter data, and the action space includes a discrete set of strategies for adjusting simulation parameters; Step S3: Design the reward function for the deep reinforcement learning agent; wherein the reward function is calculated based on the matching degree between the simulation prediction error and the actual failure data, and incorporates an adaptive penalty term; Step S4: Based on the constructed state space, action space, and reward function, train the deep reinforcement learning agent; during the training process, the random noise and parameter drift in the glass processing process are simulated by dynamically adjusting the state transition probability. Step S5: Using the trained deep reinforcement learning agent, perform failure simulation of the glass processing process and generate simulation results, including real-time failure path prediction and risk assessment report. Step S6: Based on the simulation results, calculate the deviation between the simulation results and the currently collected failure data, and update the policy network of the deep reinforcement learning agent in reverse.

2. The glass processing failure simulation method based on deep reinforcement learning according to claim 1, characterized in that, The historical failure data includes crack propagation records, thermal shock events, and defect distribution characteristics. High-dimensional parameter data include temperature distribution, stress field, and material inhomogeneity indices.

3. The glass processing failure simulation method based on deep reinforcement learning according to claim 2, characterized in that, The method for constructing the state space and action space of a glass processing failure simulation environment based on historical failure data and high-dimensional parameter data includes: Historical failure data is decomposed into time series to extract the spatiotemporal sequence features of failure events, and then fused with high-dimensional parameter data in a multimodal manner to generate an initial vector representation of the state space. The high-dimensional parameter data in the initial vector representation is reduced in dimension, and the nonlinear coupling components are retained to form a feature set after dimension reduction. Each feature in the feature set corresponds to a key uncertainty factor in glass processing. The obtained feature set is defined as a dynamic vector representation of the state space; The action space is defined as a set of vectors for parameter adjustment, where each action vector is a combination of temperature offset, stress correction and material property fine-tuning, and the validity boundary of each action vector is determined. The state space and action space are jointly embedded in the Markov decision process framework, and the state transition process in the framework satisfies the physical constraints of glass processing and random noise disturbances.

4. The glass processing failure simulation method based on deep reinforcement learning according to claim 3, characterized in that, The method for dimensionality reduction of high-dimensional parameter data in the initial vector representation includes: By applying nonlinear principal component analysis, we can capture the nonlinear relationships between high-dimensional parameter data and generate a dimension-reduced embedding space. The first k principal components are retained, where the value of k is adaptively determined based on the cumulative variance contribution rate of the principal components. Domain knowledge constraints are introduced during the dimensionality reduction calculation.

5. The glass processing failure simulation method based on deep reinforcement learning according to claim 4, characterized in that, The method for designing the reward function of the deep reinforcement learning agent includes: The difference between the failure path predicted in real time by the simulation and the actual failure data is measured and used as the basic reward item. A quantitative index for the nonlinear coupling effect between high-dimensional parameter data is introduced to construct an adaptive penalty term; The basic reward term, adaptive penalty term, and evaluation of the generalization ability of the deep reinforcement learning agent are all used as optimization objectives of the reward function, and the contribution weight of each objective is balanced by weighted aggregation.

6. The glass processing failure simulation method based on deep reinforcement learning according to claim 5, characterized in that, The method for constructing the adaptive penalty term includes: Real-time monitoring of parameter drift in high-dimensional parameter data; the trace of its covariance matrix is ​​used as a comprehensive index to quantify the nonlinear coupling between parameters and the overall drift intensity. Based on comprehensive indicators, the weight coefficient of the adaptive penalty term is dynamically calculated using an exponential decay function; The adaptive penalty term with weighted coefficients is integrated into the reward function.

7. The glass processing failure simulation method based on deep reinforcement learning according to claim 6, characterized in that, The method for training a deep reinforcement learning agent includes: Initialize the policy network and value network of the deep reinforcement learning agent, where the policy network uses residual connections to process the high-dimensional parametric data dynamic vector representation of the state space; During the training iteration, an experience replay buffer is introduced to store state-action-reward tuples, and a priority sampling mechanism is used to prioritize the processing of samples with high temporal difference errors. The learning rate is dynamically adjusted during the training process, and the step size is adaptively scaled and updated based on the gradient norm of the simulation prediction error. An actor-critic training framework is adopted, in which an attention layer is embedded in the critic network to capture long-term dependencies between high-dimensional parameter data.

8. The glass processing failure simulation method based on deep reinforcement learning according to claim 7, characterized in that, The method of introducing an experience replay buffer to store state-action-reward tuples includes: The experience replay buffer is designed as a priority-based queue structure, and sampling priority is assigned according to the temporal difference error of the tuple; Establish a periodic buffer cleanup mechanism to adapt to the evolution of dynamic vector representations of high-dimensional parameter data in the state space by removing outdated historical tuples; In the priority-based sampling process, importance sampling weights are applied to correct sampling distribution bias.

9. The glass processing failure simulation method based on deep reinforcement learning according to claim 8, characterized in that, The method for using a trained deep reinforcement learning agent to perform failure simulation of the glass processing process and generate simulation results includes: Real-time high-dimensional parameter data is input into a deep reinforcement learning agent to generate a dynamic vector representation of the high-dimensional parameter data. A sequence of parameter adjustment actions is generated through a policy network to simulate the evolution of failure paths in the glass processing process. Multi-step prediction based on policy networks is used to assess the probability of failure risk in future time steps and generate confidence intervals for failure prediction. The predicted simulation results are transformed into real-time failure path prediction and risk assessment reports. The real-time failure path prediction includes a failure risk heat map and a failure path evolution map, which identify high-risk areas and critical parameter points that are sensitive to failure during the processing. The risk assessment report includes a risk level assessment based on confidence intervals.

10. The glass processing failure simulation method based on deep reinforcement learning according to claim 9, characterized in that, The method for calculating the deviation between the simulation results and the currently collected failure data, and then updating the policy network of the deep reinforcement learning agent in reverse, includes: The sequence alignment error between the failure path predicted by the quantification simulation and the actual failure data is used as a deviation measure for the policy network update. The bias metric is incorporated as an additional training signal into the reward function, and the parameter weights of the policy network are updated through backpropagation. The meta-learning optimization process is introduced to quickly adjust the strategy network parameters.

Images