A coating process self-adaptive closed-loop control method and system based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's coating process adaptive closed loop control method and system, it is related to coating process intelligent control technical field, including: the slurry pump speed of acquisition coating production line, oven temperature, coating speed and coating surface density data, constructs state space vector, and with surface density deviation as reward signal;The double-agent reinforcement learning framework of layered time scale is constructed, and slurry pump speed fast response agent and oven temperature slow response agent are respectively set;Based on deep deterministic policy gradient algorithm, the double-agent is trained, and the corresponding control increment is output;Intelligent agent coordination mechanism is designed, and state sharing and reward feedback are realized;The agent after training is deployed to control system, and the adaptive closed loop control of coating process is realized.The application can effectively improve the collaborative control ability of multivariable coupling system, reduce the regulation lag, improve coating surface density control precision and stability.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for coating processes, and in particular to an adaptive closed-loop control method and system for coating processes based on reinforcement learning. Background Technology

[0002] With the rapid development of new energy battery manufacturing and high-performance coating material preparation technologies, the coating process, as a critical link, directly affects the consistency, yield, and electrochemical performance of products due to its process control precision. Traditional coating process control often relies on empirical models or feedback control methods based on mechanistic modeling, such as PID control and model predictive control (MPC). This involves adjusting key process parameters such as slurry pump speed, coating speed, and oven temperature to achieve stable control of the coating surface density. However, due to the strong nonlinearity, multivariate coupling, and significant time delay characteristics of the coating process, especially under the influence of factors such as fluctuations in slurry rheological properties, environmental disturbances, and differences in equipment dynamic response, traditional control methods often struggle to achieve high-precision and robust dynamic adjustment. Furthermore, existing control strategies typically employ a single time-scale control framework, making it difficult to account for the response differences between different control variables. For example, slurry pump speed has rapid dynamic characteristics, while oven temperature adjustment has significant inertia and delay, leading to problems such as oscillation, overshoot, or response hysteresis during the overall system adjustment process. Meanwhile, existing methods generally rely on explicit modeling and lack the ability to adaptively learn implicit dynamic features under complex working conditions, making it difficult to meet the needs of high-end manufacturing for intelligent and self-optimizing control.

[0003] CN104209246B discloses an automatic coating method and control system. It uses a double-sided penetrating beta sensor to detect the surface density of the coated electrode, and combines this with a pressure sensor and a feed pump to automatically adjust the slurry feed rate. This reduces manual intervention to some extent and alleviates the lag problem in surface density adjustment. While the method has a relatively simple structure and some engineering applicability, its control strategy essentially relies on a feedback-based single-variable adjustment mechanism. It lacks system modeling and collaborative control capabilities for multi-variable coupling relationships, and in particular, it does not consider the dynamic coupling effect between oven temperature and slurry pump speed, making it difficult to achieve high-precision control under complex operating conditions.

[0004] CN109865641B proposes a method for controlling the coating surface density of an extrusion coating machine. This method establishes a mapping relationship between cavity pressure and surface density, and adjusts the cavity pressure based on multiple parameters (including equipment parameters and slurry parameters) to achieve rapid approximation of the target surface density. While this method reduces control delay caused by measurement lag to some extent and improves adjustment efficiency, it still relies on a pre-set mapping model and parameter estimation. It lacks online learning and adaptive optimization capabilities for dynamic system changes and fails to distinguish the differences in time response of different control variables, making it difficult to effectively handle collaborative control problems across multiple time scales. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the problems of existing coating process control methods, such as insufficient multivariate coupling modeling, limited ability to handle time scale differences, and weak adaptive adjustment ability, this invention is proposed.

[0007] Therefore, the problem to be solved by this invention is how to achieve layered coordinated control of key parameters such as slurry pump speed and oven temperature under complex dynamic working conditions, and improve the real-time performance and stability of coating surface density adjustment.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide an adaptive closed-loop control method for a coating process based on reinforcement learning, comprising, Collect measurements of slurry pump speed, oven temperature, coating speed, and coating surface density from the coating production line, and construct a state space vector containing slurry pump speed, oven temperature, and coating speed. Use the deviation between the coating surface density measurement value and the target value as a reward signal. A hierarchical time-scale dual-agent reinforcement learning framework is constructed, which includes a fast-response agent that controls the slurry pump speed and a slow-response agent that controls the oven temperature. Fast-response and slow-response agents are trained using a deep deterministic policy gradient algorithm. Establish a coordination mechanism between fast-response agents and slow-response agents. The slow-response agent transmits the current oven temperature setpoint to the fast-response agent as part of the state input, and the fast-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. The trained fast-response and slow-response agents are deployed to the coating production line control system. Based on real-time coating status data, they output slurry pump speed control and oven temperature control respectively, forming a layered time-scale adaptive closed-loop control of the coating process.

[0009] Secondly, embodiments of the present invention provide an adaptive closed-loop control system for a coating process based on reinforcement learning, comprising: The data acquisition and state construction module is used to collect the slurry pump speed, oven temperature, coating speed and coating surface density measurement values ​​of the coating production line, and construct a state space vector containing the slurry pump speed, oven temperature and coating speed, and use the deviation between the coating surface density measurement value and the target value as the reward signal. The dual-agent construction module is used to construct a hierarchical time-scale dual-agent reinforcement learning framework, which includes a fast-response agent that controls the slurry pump speed and a slow-response agent that controls the oven temperature. The policy training module is used to train fast-response and slow-response agents based on the deep deterministic policy gradient algorithm. The agent coordination module is used to establish a coordination mechanism between fast-response agents and slow-response agents. The slow-response agent transmits the current oven temperature setpoint to the fast-response agent as part of the state input, and the fast-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. The control execution module is used to deploy the trained fast-response and slow-response agents to the coating production line control system. Based on real-time coating status data, it outputs slurry pump speed control and oven temperature control respectively, forming a layered time-scale adaptive closed-loop control of the coating process.

[0010] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the above-described reinforcement learning-based adaptive closed-loop control method for coating process.

[0011] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the above-described reinforcement learning-based adaptive closed-loop control method for coating processes.

[0012] Compared with existing technologies, the advantages of this invention are as follows: By collecting measurements of slurry pump speed, oven temperature, coating speed, and areal density from the coating production line, a state space vector is constructed, and areal density deviation is used as a reward signal, providing real-time and accurate environmental perception and optimization targets for subsequent agents; a hierarchical time-scale dual-agent reinforcement learning framework is constructed, independently modeling fast-response variables (slurry pump speed) and slow-response variables (oven temperature), effectively solving the coupling control problem caused by the dynamic response time differences of different variables during the coating process; subsequently, a deep deterministic policy gradient algorithm is used to train fast-response and slow-response agents respectively, and a coordination mechanism between the two is established. --The slow-response agent transmits the oven temperature setpoint to the fast-response agent as a state input, while the fast-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. This enables bidirectional information interaction and collaborative decision-making across time scales, avoiding oscillations or lags caused by time scale mismatch in traditional single controllers. The trained agents are deployed to the coating production line control system, and output slurry pump speed control and oven temperature control based on real-time state data, forming a layered time scale adaptive closed-loop control. This significantly improves the tracking accuracy and response speed of coating surface density to the target value, while reducing reliance on manual parameter tuning and the risk of process fluctuations. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 The flowchart shows an adaptive closed-loop control method for the coating process based on reinforcement learning. Figure 2 This is a structural diagram of an adaptive closed-loop control system for the coating process based on reinforcement learning. Detailed Implementation

[0014] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0015] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0016] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0017] As mentioned in the background section, existing technologies generally rely on fixed models or single feedback mechanisms, making it difficult to simultaneously consider the dynamic response characteristics of different control variables, and they lack sufficient adaptability to system nonlinearity and disturbances. To address these problems, this invention provides an adaptive closed-loop control method for the coating process based on reinforcement learning.

[0018] Reference Figure 1 , Figure 1 This is a flowchart of an adaptive closed-loop control method for a coating process based on reinforcement learning, according to an embodiment of the present invention. Figure 1 As shown, an adaptive closed-loop control method for a coating process based on reinforcement learning includes: S1: Collect the slurry pump speed, oven temperature, coating speed and coating surface density measurements of the coating production line, construct a state space vector containing slurry pump speed, oven temperature and coating speed, and use the deviation between the coating surface density measurement value and the target value as the reward signal.

[0019] Specifically, during the operation of the coating production line, slurry pump speed data is collected by a flow sensor installed in the slurry supply system, with the slurry pump speed data in revolutions per minute; oven temperature data is collected by a temperature sensor array installed inside the oven, with the oven temperature data including the temperature values ​​of each section of the oven; coating speed data is collected by the coating machine drive system, with the coating speed data in meters per minute; and coating surface density is measured by an online thickness gauge, with the coating surface density measured in grams per square meter. The slurry pump speed data, oven temperature data, coating speed data, and coating surface density measurement are synchronously uploaded to the control system at a preset sampling period.

[0020] It should be noted that the preset sampling period is set in stages according to the response characteristics of each sensor: the sampling period for slurry pump speed data and coating speed data is set to 50 milliseconds to 100 milliseconds, the sampling period for oven temperature data is set to 500 milliseconds to 1000 milliseconds, and the sampling period for coating surface density measurement is set to 200 milliseconds to 500 milliseconds; the temperature sensor array is evenly arranged in each section of the oven, and the temperature value of each section is taken as the arithmetic mean of all the measurements of the temperature sensor array in the corresponding section as the representative value of the oven temperature data of that section.

[0021] Furthermore, a data preprocessing module is established to perform outlier removal and filtering on the slurry pump speed data, oven temperature data, coating speed data, and coating surface density measurement values. Outlier removal is based on the preset reasonable range of each parameter, and sampling points exceeding the preset reasonable range are replaced by linear interpolation of adjacent normal sampling points. The filtering process uses a moving average algorithm to smooth the slurry pump speed data, oven temperature data, coating speed data, and coating surface density measurement values, respectively, to obtain filtered slurry pump speed data, filtered oven temperature data, filtered coating speed data, and filtered coating surface density measurement values. The preset reasonable range for slurry pump speed data is 20% to 100% of the rated speed, the preset reasonable range for oven temperature data is within ±10 degrees Celsius of the process set temperature, the preset reasonable range for coating speed data is 15% to 90% of the rated linear speed, and the preset reasonable range for coating surface density measurement values ​​is within ±15% of the product specification target value.

[0022] In an optional embodiment, the window length of the moving average algorithm is configured independently based on the dynamic response rate of each parameter: the moving average algorithm window length for slurry pump speed data is set to 3 to 5 sampling points, the moving average algorithm window length for oven temperature data is set to 8 to 12 sampling points, the moving average algorithm window length for coating speed data is set to 3 to 5 sampling points, and the moving average algorithm window length for coating surface density measurement is set to 5 to 8 sampling points; the above window length configuration matches the physical response time constant of each parameter to avoid excessive smoothing leading to control response lag.

[0023] Furthermore, based on the filtered slurry pump speed data, filtered oven temperature data, filtered coating speed data, and filtered coating surface density measurement value, a state space vector is constructed. This state space vector includes the filtered slurry pump speed data, filtered oven temperature data, filtered coating speed data, the filtered coating surface density measurement value at the current moment, and the filtered coating surface density measurement values ​​at historical moments. The filtered coating surface density measurement values ​​at historical moments are taken from the data of the five sampling points preceding the current moment (based on the original sampling period of the coating surface density measurement value), used to capture the dynamic changes in the coating process. The filtered coating surface density measurement value at the current moment is defined as: at the start of the current fast timescale layer sampling period, the most recent filtered coating surface density measurement value before that moment is obtained from the time-series database (i.e., the latest valid value at the start of the period), rather than the average of multiple measurements within that period, to ensure that the state space can reflect the latest changes in coating surface density in a timely manner.

[0024] Furthermore, each component in the state space vector is mapped to a dimensionless state component according to its own preset normalization interval, with the normalization interval uniformly set to zero to one. The normalization upper and lower limits corresponding to the slurry pump speed data are determined based on the rated speed range of the slurry pump, the normalization upper and lower limits corresponding to the oven temperature data are determined based on the oven process temperature window, the normalization upper and lower limits corresponding to the coating speed data are determined based on the rated linear speed range of the coating machine, and the normalization upper and lower limits corresponding to the coating surface density measurement value are determined based on the coating surface density range allowed by the product specifications. The normalized state space vector is used as the input of the environmental observation values ​​in the subsequent dual-agent reinforcement learning framework in S2.

[0025] It should be noted that the purpose of introducing the filtered coating surface density measurement value of historical time into the state space vector is to provide the temporal dynamic information of the coating process for the dual-agent reinforcement learning framework. The temporal dynamic information reflects the comprehensive influence of slurry rheological properties, temperature conduction properties and coating dynamics on the coating surface density measurement value, so that the state space vector can accurately characterize the dynamic characteristics of the actual coating production process.

[0026] Furthermore, based on product specifications, a target value for coating surface density is set, the deviation between the filtered measured value of coating surface density and the target value is calculated, and a reward signal calculation method is designed based on the deviation. The reward signal is negative, and its magnitude is proportional to the absolute value of the deviation. The smaller the absolute value of the deviation, the closer the reward signal is to zero, indicating a better current control effect.

[0027] Specifically, a deviation trend correction term is introduced into the reward signal calculation method. The deviation trend is represented by the difference between the deviation at the current time and the deviation at the previous sampling point. When the deviation trend is negative, that is, the deviation gradually decreases, a preset positive reward correction amount is added to the reward signal. The preset positive reward correction amount is proportional to the absolute value of the deviation trend, which promotes the control strategy of the dual-agent reinforcement learning framework to adjust in the direction of continuously decreasing deviation. When the deviation trend is positive, that is, the deviation gradually increases, the preset positive reward correction amount is not added.

[0028] In an optional embodiment, a control increment penalty term is added to the reward signal calculation method. The control increment penalty term is represented by the weighted sum of the square of the slurry pump speed adjustment and the square of the oven temperature adjustment. The weight coefficients are calibrated according to the sensitivity of the influence of each control quantity on the coating surface density measurement value. In the main embodiment, the reward signal is represented only by the sum of the opposite of the absolute value of the deviation and the deviation change trend correction term, with the dual optimization orientation of minimizing the absolute value of the deviation and continuously reducing the deviation.

[0029] It should be noted that the reward signal is calculated once at the end of each fast timescale layer sampling period (5 seconds): within this period, multiple coating surface density measurements are collected sequentially according to the original sampling period (200-500 milliseconds) of the coating surface density measurement. After filtering, the instantaneous deviation and deviation change trend corresponding to each sampling point are calculated to obtain the instantaneous reward signal of each sampling point within this period. The arithmetic mean of all instantaneous reward signals within this period is taken as the final reward signal corresponding to the sampling period of this fast timescale layer, which is used for subsequent state updates and agent training. This aggregation method matches the control decision period of the fast timescale layer, avoiding drastic fluctuations in the reward signal caused by surface density measurement noise.

[0030] Furthermore, a data storage mechanism is established to store slurry pump speed data, oven temperature data, coating speed data, coating surface density measurement value, coating surface density target value, and reward signal into a time-series database according to the acquisition timestamp. The slurry pump speed data, oven temperature data, coating speed data, coating surface density measurement value, coating surface density target value, and reward signal in the time-series database are used for the training and evaluation of the dual-agent reinforcement learning framework in S2.

[0031] Furthermore, the time-series database stores data in separate tables according to parameter categories. Slurry pump speed data, coating speed data, and coating surface density measurement values ​​are stored in the high-frequency data table, with the writing frequency consistent with their respective preset sampling periods. Oven temperature data is stored in the low-frequency data table, with the writing frequency consistent with the preset sampling period of the oven temperature data. The reward signal and coating surface density measurement values ​​share the same timestamp and are written to the reward data table. The time-series database supports fast queries by time range, with a query response time of no more than 10 milliseconds, to meet the access requirements of the dual-agent reinforcement learning framework for real-time training of historical data.

[0032] It should be noted that the preset sampling period for each sensor (50-100 ms for slurry pump speed data and coating speed data, 500-1000 ms for oven temperature data, and 200-500 ms for coating surface density measurement) is the original data acquisition frequency, while the sampling period for the fast timescale layer (5 seconds) and the sampling period for the slow timescale layer (60 seconds) are the execution frequency of control decisions. The design of the control decision sampling period being significantly lower than the data acquisition frequency allows for obtaining a sufficient number of measurement data points within each control decision cycle, which can be adequately filtered through a moving average algorithm to improve the robustness of control decisions.

[0033] S2: Construct a hierarchical time-scale dual-agent reinforcement learning framework, which includes a fast-response agent controlling the slurry pump speed and a slow-response agent controlling the oven temperature.

[0034] S2.1: Analysis of the dynamic influence time of slurry pump speed and oven temperature on the measured value of coating surface density in tool coating process. Slurry pump speed is identified as a fast response variable and oven temperature is identified as a slow response variable. Based on the difference in the dynamic influence time of slurry pump speed and oven temperature, a layered time scale control structure is established.

[0035] Specifically, the dynamic impact time of adjusting the slurry pump speed on the measured coating surface density is 10 to 30 seconds. The dynamic impact time is determined through a step response experiment. The specific method is as follows: under the stable operation of the coating system, a 5% step change is applied to the slurry pump speed and the oven temperature, and the response curve of the measured coating surface density is recorded. The time required for the response curve to reach 63.2% of the new steady-state value is taken as the dynamic impact time. The experiment is repeated 5 times and the average value is taken to ensure the reliability of the dynamic impact time data.

[0036] Preferably, the dynamic influence time of the oven temperature adjustment on the coating surface density measurement value is 3 to 10 minutes. Based on the difference in dynamic influence time, the control layer corresponding to the fast response variable is set as the fast time scale layer, and the control layer corresponding to the slow response variable is set as the slow time scale layer. The sampling period of the fast time scale layer is set to 5 seconds, and the sampling period of the slow time scale layer is set to 60 seconds. The ratio of the sampling period of the slow time scale layer to the sampling period of the fast time scale layer is 12, which matches the thermal inertia characteristics of the oven temperature.

[0037] S2.2: Based on a hierarchical time-scale control structure, a fast-response agent is constructed in the fast time-scale layer, and a slow-response agent is constructed in the slow time-scale layer. Both the fast-response agent and the slow-response agent adopt an Actor-Critic architecture composed of an actor network and a commentator network.

[0038] Furthermore, the state input of the fast-response agent includes the current filtered slurry pump speed, the filtered coating speed, the deviation sequence between the recent coating surface density measurement value and the coating surface density target value, and the oven temperature setting value transmitted by the slow-response agent; the action output of the fast-response agent is the slurry pump speed adjustment amount, which is superimposed on the current filtered slurry pump speed to form the slurry pump speed command value for the next fast time scale layer sampling period.

[0039] Furthermore, the state input of the slow-response agent includes the current filtered oven temperature, the deviation trend between the measured coating density and the target coating density within a longer time window, and the cumulative reward average fed back by the fast-response agent; the action output of the slow-response agent is the oven temperature adjustment amount, which is superimposed on the current oven temperature setpoint to form the oven temperature setpoint for the next slow time scale layer sampling period; the deviation trend is represented by the moving average of the deviation between the measured coating density and the target coating density within the slow time scale layer sampling period.

[0040] S2.3: Construct neural network models for fast-response agents and slow-response agents respectively. The actor networks of both fast-response agents and slow-response agents adopt a three-layer fully connected neural network structure. The number of neurons in the input layer is consistent with the dimension of their respective state inputs, and the number of neurons in the output layer is one-dimensional, corresponding to the slurry pump speed adjustment and the oven temperature adjustment respectively.

[0041] Specifically, the number of hidden layer neurons in the actor network of the fast response agent is set to 64, and the activation function is a linear rectified function. The commenter network of the fast response agent adopts the same three-layer fully connected structure as the actor network, taking the concatenated vector of the fast response agent's state input and the slurry pump speed adjustment as input, and the output is the single-state action value estimate. A residual connection structure is introduced into the actor network of the fast response agent, in which the state vector of the input layer is directly added to the output of the second hidden layer. To achieve the residual connection, when the dimension of the input layer state vector is inconsistent with the dimension of the second hidden layer output, a linear projection layer (without activation function) is added after the input layer to map the input vector to the same dimension as the output of the second hidden layer. The weight parameters of the linear projection layer are trained together with the actor network. If the two dimensions are already equal, the identity mapping is directly used for residual addition to alleviate the gradient vanishing problem of the fast response agent under high-frequency update conditions.

[0042] It should be noted that the number of hidden layer neurons in the actor network of the slow-response agent is set to 32, and the activation function is a linear rectified function. The number of hidden layer neurons in the actor network of the fast-response agent (64) is greater than that of the slow-response agent (32). The rationale is that the fast-response agent needs to handle high-frequency changing state inputs, has higher requirements for capturing short-term dynamic features, and needs stronger expressive power; while the slow-response agent mainly captures long-term trend changes, has a relatively smooth state space, and a smaller network structure is sufficient to express its policy space. This differentiated design also takes into account the balance between computational efficiency and control accuracy.

[0043] Furthermore, the commentator network of the slow-response agent adopts the same three-layer fully connected structure as the actor network of the slow-response agent. It takes the concatenated vector of the slow-response agent's state input and the oven temperature adjustment amount as input, and outputs the single-state action value estimate.

[0044] In an optional embodiment, a temporal convolutional layer is added to the actor network of the slow-response agent. The temporal convolutional layer is located between the input layer and the first hidden layer, and the kernel length is set to 5 to 10 sampling points to extract temporal features in the deviation trend. In the main embodiment, the actor network of the slow-response agent adopts a three-layer fully connected neural network structure without a temporal convolutional layer to reduce the inference computation of the slow-response agent.

[0045] S2.4: Based on the definition of state input and action output of fast-response agents and slow-response agents, establish a hierarchical interaction mechanism between fast-response agents and slow-response agents. The hierarchical interaction mechanism includes a top-down parameter transmission channel and a bottom-up feedback channel.

[0046] Furthermore, the top-down parameter transfer channel transmits the oven temperature setpoint from the slow-response agent to the state input of the fast-response agent at the beginning of each slow-timescale layer sampling period. The oven temperature setpoint remains unchanged within the slow-timescale layer sampling period, providing the fast-response agent with the current oven operating condition constraint for calculating the slurry pump speed adjustment. The bottom-up feedback channel transmits the cumulative reward average accumulated by the fast-response agent within that sampling period to the state input of the slow-response agent at the end of each slow-timescale layer sampling period. The cumulative reward average is represented by the weighted average of the reward signals corresponding to all fast-timescale layer sampling periods within that sampling period. The weight of each reward signal is set according to the time interval from the end of the slow-timescale layer sampling period (the closer to the end time, the greater the weight). For the specific weighting method, see S4.5.

[0047] In an optional simplified embodiment, an equal-weighted arithmetic mean can also be used as the cumulative reward mean, but the main embodiment uses a weighted average to more accurately reflect the recent control effect.

[0048] It should be noted that in the hierarchical interaction mechanism, the oven temperature setpoint provides the operating boundary for the slurry pump speed adjustment of the fast-response agent, and the cumulative reward average provides the optimization basis for the oven temperature adjustment of the slow-response agent. The two transmission channels do not block each other in terms of timing, and their respective triggering cycles are the sampling period of the fast time scale layer and the sampling period of the slow time scale layer.

[0049] S2.5: Configure independent experience replay buffers for fast-response agents and slow-response agents respectively.

[0050] Specifically, the experience replay buffer of the fast-response agent stores the transition samples consisting of the fast-response agent's state input, slurry pump speed adjustment, reward signal, and the fast-response agent's state input for the next fast time scale layer sampling period; the experience replay buffer of the slow-response agent stores the transition samples consisting of the slow-response agent's state input, oven temperature adjustment, cumulative reward average, and the slow-response agent's state input for the next slow time scale layer sampling period.

[0051] In an optional embodiment, the experience replay buffer capacity of the fast-response agent is set to 10,000 transfer samples, and the experience replay buffer capacity of the slow-response agent is set to 2,000 transfer samples. The experience replay buffer of the fast-response agent adopts a uniform random sampling strategy, while the experience replay buffer of the slow-response agent adopts a priority sampling strategy. The priority sampling strategy assigns sampling priority to each transfer sample based on the absolute value of the temporal difference error corresponding to each transfer sample. Transfer samples with larger absolute values ​​of temporal difference errors receive higher sampling probabilities, so that the slow-response agent prioritizes updating its strategy from transfer samples that significantly improve system performance.

[0052] S2.6: Establish a training synchronization mechanism between fast-response agents and slow-response agents to coordinate the parameter update rhythm of the two agents.

[0053] Furthermore, the fast-response agent performs a parameter update after 20 interactions with the coating production line control system, while the slow-response agent performs a parameter update after the fast-response agent completes 5 parameter updates. The ratio of the parameter update frequency of the fast-response agent to that of the slow-response agent is set to 5. This ratio is not strictly equal to the time scale difference (12:1) between the sampling periods of the fast and slow time scale layers in S2.1, but is a compromise value verified through experiments: a ratio of 5 ensures that the fast agent has enough (5) strategy iterations before each update by the slow agent to adapt to the new oven temperature setting, while avoiding excessively slow convergence of the slow agent due to an excessively large ratio. Those skilled in the art can adjust this ratio according to the actual process, but the update frequency of the fast agent should be kept higher than that of the slow agent.

[0054] It should be noted that the training synchronization mechanism ensures that after each adjustment of the oven temperature setpoint, the fast-response agent has a sufficient number of parameter update times to adapt to the coating conditions corresponding to the current oven temperature setpoint before triggering the slow-response agent to perform parameter updates. This avoids the slow-response agent performing parameter updates based on the cumulative reward average, which does not fully reflect the effect of the current oven temperature setpoint, before the fast-response agent's strategy has converged. The training synchronization mechanism is consistent with the hierarchical learning rhythm of the hierarchical time-scale control structure in S2.1.

[0055] S3: Train fast-response agents and slow-response agents according to the deep deterministic policy gradient algorithm. The fast-response agent outputs the incremental control of slurry pump speed, and the slow-response agent outputs the incremental control of oven temperature.

[0056] S3.1: Initialize the weight parameters of the Actor-Critic architecture, which consists of the actor network and the commentator network for fast-response agents and slow-response agents, respectively.

[0057] It should be noted that the weight parameters of the actor network and commentator network of the fast-response agent are initialized with a truncated normal distribution with a mean of 0 and a standard deviation of 0.1; the hidden layer weight parameters of the actor network and commentator network of the slow-response agent are initialized with a truncated normal distribution with a mean of 0 and a standard deviation of 0.05. The standard deviation is set to 0.05 to adapt to the smoother state-action value function changes in the slower time scale layer.

[0058] Furthermore, target actor networks and target commentator networks are created for fast-response agents and slow-response agents respectively. The target actor network weights of the fast-response agent are completely consistent with the actor network weights of the fast-response agent; the target commentator network weights of the fast-response agent are completely consistent with the commentator network weights of the fast-response agent; the target actor network weights of the slow-response agent are completely consistent with the actor network weights of the slow-response agent; and the target commentator network weights of the slow-response agent are completely consistent with the commentator network weights of the slow-response agent.

[0059] S3.2: Configure the training hyperparameters of the deep deterministic policy gradient algorithm for the fast-response agent and the slow-response agent respectively.

[0060] Specifically, the discount factor for the fast-response agent is set to 0.95, and the discount factor for the slow-response agent is set to 0.99. The difference in discount factors reflects the different degrees of importance that the fast-timescale layer and the slow-timescale layer place on future rewards. The learning rate for the fast-response agent is set to 0.001, and the learning rate for the slow-response agent is set to 0.0005. The learning rate of the slow-response agent is lower than that of the fast-response agent to match the slower policy update rhythm of the slow-timescale layer compared to the fast-timescale layer. The target network soft update coefficient for the fast-response agent is set to 0.01, and the target network soft update coefficient for the slow-response agent is set to 0.005.

[0061] S3.3: Design an exploration strategy for the fast-response agent by superimposing exploration noise based on the Ornstein-Uhlenbeck process into the slurry pump speed adjustment output of the fast-response agent.

[0062] Furthermore, the mean parameter of the Ornstein-Uhlenbeck process is set to 0, the volatility parameter to 0.2, and the regression velocity parameter to 0.15. The amplitude of the exploration noise gradually decreases with the increase of training steps according to an adaptive decay coefficient. The initial value of the adaptive decay coefficient is set to 0.9995, and it is dynamically adjusted according to the online control performance indicators: when the online control performance indicators show improvement in three consecutive evaluations, the adaptive decay coefficient is adjusted to the current value of 0.99; when the online control performance indicators show no improvement in three consecutive evaluations, the adaptive decay coefficient is adjusted to the current value of 1.01, but does not exceed 1.0. This adaptive mechanism ensures that the exploration behavior can be automatically adjusted according to the learning progress under different product specifications.

[0063] Furthermore, the slurry pump speed adjustment amount of the fast-response agent is limited after the exploration noise is superimposed. The limit range of the slurry pump speed adjustment amount is set to within ±5% of the current filtered slurry pump speed data rating. The limiting process prevents the slurry pump speed adjustment amount from being too large during the exploration process, which may cause abnormal fluctuations in the coating process. The limited slurry pump speed adjustment amount is written into the action field of the corresponding transfer sample in the experience playback buffer of the fast-response agent in S2.5.

[0064] S3.4: Design exploration strategies for slow-response agents.

[0065] Specifically, considering the high thermal inertia of the coating system caused by oven temperature adjustment, Gaussian noise is superimposed on the output of the slow-response agent's oven temperature adjustment. The initial value of the standard deviation of the Gaussian noise is set to 0.1, which decreases linearly to 0.02 with the number of training steps. The oven temperature adjustment of the slow-response agent is limited after the Gaussian noise is superimposed. The limiting range of the oven temperature adjustment is set to within ±2 degrees Celsius to avoid excessive adjustment of the oven temperature, which could lead to thermal instability of the coating system.

[0066] In an optional embodiment, a safety verification mechanism based on coating process constraints is introduced into the exploration strategy of the slow-response agent. The safety verification mechanism compares the result of superimposing the oven temperature adjustment amount to the current oven temperature setting value with the preset upper limit value and preset lower limit value of oven temperature in S2.1 after adding Gaussian noise to the oven temperature adjustment amount and before the limiting process. The oven temperature adjustment amount that exceeds the range is truncated to the corresponding boundary value, filtering out exploration actions that may violate the coating process constraints. In the main embodiment, the exploration strategy of the slow-response agent only uses Gaussian noise superposition and ±2 degrees Celsius limiting process, without the safety verification mechanism.

[0067] S3.5: Calculate the commenter network loss values ​​for fast-response agents and slow-response agents respectively using the deep deterministic policy gradient algorithm.

[0068] Furthermore, after every 20 interactions in the fast timescale layer, 256 transition samples are randomly sampled from the experience replay buffer of the fast response agent to form a mini-batch; the commentator network loss value is represented by the mean square error between the state-action value estimate of the current transition sample and the target state-action value estimate of the commentator network of the fast response agent; the target state-action value estimate is calculated by the target commentator network and the target actor network of the fast response agent based on the reward signal and the state space vector of the next fast timescale layer sampling period.

[0069] Furthermore, the commentator network loss value of the fast response agent is calculated using a gradient pruning operation, with the gradient pruning threshold set between 0.5 and 1.0 to prevent gradient explosion when the measured coating density value deviates abruptly from the target coating density value. The actor network of the fast response agent updates its weight parameters using a policy gradient ascent method. The policy gradient direction is the gradient direction of the state-action value estimate predicted by the commentator network of the fast response agent relative to the slurry pump speed adjustment, with the objective of maximizing the state-action value estimate predicted by the commentator network. After minimizing the commentator network loss value and performing policy gradient ascent on the actor network, a soft update is performed on the target actor network and the target commentator network of the fast response agent according to the target network soft update coefficient.

[0070] Specifically, after the fast-response agent completes 5 weight parameter updates, 128 transition samples are sampled from the slow-response agent's experience replay buffer according to a priority sampling strategy to form a mini-batch. The priority sampling strategy assigns sampling weights to each transition sample based on the absolute value of the temporal difference error corresponding to each transition sample, with transition samples having larger absolute values ​​of temporal difference errors receiving higher sampling weights. The commentator network loss value of the slow-response agent is calculated using a deep deterministic strategy gradient algorithm. The commentator network loss value of the slow-response agent is represented by the mean square error between the commentator network's estimate of the state-action value of the current transition sample and the estimate of the target state-action value. The estimate of the target state-action value is calculated by the target commentator network and the target actor network of the slow-response agent based on the cumulative reward mean and the state space vector of the next slow time scale layer sampling period.

[0071] It should be noted that the mini-batch size of the fast-response agent (256 entries) is larger than that of the slow-response agent (128 entries) because: the state space of the fast-response agent changes more frequently, requiring a larger mini-batch size to cover a sufficient range of state space distributions and improve learning stability; while the state changes of the slow-response agent are relatively gradual, and a smaller mini-batch size can provide sufficient gradient estimation accuracy; in addition, since the slow-response agent uses a priority sampling strategy, the effective information density of the samples is higher, and a smaller mini-batch size can be used compared to uniform random sampling.

[0072] Furthermore, the actor network of the slow-response agent updates its weight parameters using the policy gradient ascent method. The policy gradient direction is the gradient direction of the state-action value estimate predicted by the commentator network of the slow-response agent relative to the oven temperature adjustment. After minimizing the commentator network loss and performing policy gradient ascent on the actor network of the slow-response agent, a soft update is performed on both the target actor network and the target commentator network of the slow-response agent according to the soft update coefficient of the target network. The weight parameter update frequency of the slow-response agent is lower than that of the fast-response agent, which is consistent with the training synchronization mechanism.

[0073] S3.6: Construct a training evaluation and early stop mechanism.

[0074] Furthermore, during the training process, a validation dataset is extracted from the time-series database at a preset evaluation interval. The state space vectors from the validation dataset are input into the actor networks of fast-response agents and slow-response agents to obtain the slurry pump speed adjustment and oven temperature adjustment, respectively. Combined with the coating process simulation model, the corresponding coating surface density measurement value is calculated and compared with the coating surface density target value to obtain the online control performance index. The online control performance index includes the mean absolute value, maximum absolute value, and standard deviation of the deviation between the coating surface density measurement value and the coating surface density target value.

[0075] It should be noted that the preset evaluation interval is set to 200 fast-response agent parameter updates (equivalent to 40 slow-response agent parameter updates); the validation dataset used for each evaluation contains at least 10 complete slow-timescale sampling periods to ensure the statistical significance of the evaluation results.

[0076] Specifically, if the fast-response agent or slow-response agent fails to significantly improve the online control performance index after 5 consecutive evaluations, the learning rate of the corresponding agent is reduced to 50% of the current value before training continues. If the fast-response agent or slow-response agent fails to significantly improve the online control performance index after 10 consecutive evaluations, the training process of the corresponding agent is stopped. The criterion for determining that the online control performance index has no significant improvement is that the absolute value of the difference between the average absolute values ​​of the deviations of two adjacent evaluations is lower than the preset convergence threshold, which is set to 0.01 grams per square meter. After training, the weight parameter with the smallest average absolute value of the deviation on the validation dataset is selected as the final weight parameter for the fast-response agent and the slow-response agent, respectively, for deployment in S5.

[0077] S4: Establish a coordination mechanism between fast-response agents and slow-response agents. The slow-response agent transmits the current oven temperature setpoint to the fast-response agent as part of the state input. The fast-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. S4.1: Based on the hierarchical interaction mechanism, a hierarchical communication structure is established between slow-response agents and fast-response agents, wherein the hierarchical communication structure includes a downlink channel and an uplink channel.

[0078] Furthermore, the top-down parameter transmission channel is the downlink channel, used to transmit the oven temperature setpoint from the slow-response agent to the fast-response agent as state input; the bottom-up feedback channel is the uplink channel, used to transmit the cumulative reward average from the fast-response agent to the slow-response agent as state input; to ensure the consistency of information transmission between the downlink and uplink channels under the condition that the sampling periods of the fast timescale layer and the slow timescale layer are not synchronized, message queue buffers are set in the downlink and uplink channels respectively, with the message queue buffer length set to 10, and the information flow is managed by the first-in-first-out principle.

[0079] Furthermore, at the beginning of each slow timescale layer sampling period, the downlink message queue buffer writes a message containing the oven temperature setpoint and the corresponding timestamp. At the beginning of each fast timescale layer sampling period, the fast-response agent reads the latest message from the downlink message queue buffer and incorporates the oven temperature setpoint into the current state space vector. At the end of each slow timescale layer sampling period, the uplink message queue buffer writes a message containing the cumulative reward average and the corresponding timestamp. At the beginning of each slow timescale layer sampling period, the slow-response agent reads the latest message from the uplink message queue buffer and incorporates the cumulative reward average into the current state space vector.

[0080] S4.2: Define the information content and timing of the slow-response agent transmitting information to the fast-response agent in the downlink channel, and the feedback information content and aggregation method of the fast-response agent transmitting information to the slow-response agent in the uplink channel.

[0081] Specifically, the information transmitted through the downlink channel includes the oven temperature setpoint corresponding to the current slow timescale layer sampling period, the temperature gradient setpoints for each section of the oven, and the confidence interval of the oven temperature setpoint. The information transmission frequency of the downlink channel is consistent with the slow timescale layer sampling period, set to once every 60 seconds. Each downlink channel message is appended with a timestamp of the start time of the slow timescale layer sampling period. The fast-response agent aligns the received oven temperature setpoint with the acquisition time of its own state space vector based on the timestamp.

[0082] It should be noted that the confidence interval of the oven temperature setpoint is represented by the uncertainty of the commentator network's estimate of the state-action value of the current oven temperature adjustment for the slow-response agent. This uncertainty is calculated as follows: During the inference phase, Monte Carlo dropout is applied to the commentator network of the slow-response agent, performing 10 forward inference iterations under a fixed input state to obtain 10 state-action value estimates. The standard deviation of these estimates is calculated as the uncertainty index. A larger standard deviation indicates greater uncertainty in the commentator network's evaluation of the current state-action pair, resulting in a wider confidence interval. The width of the confidence interval is defined as twice this standard deviation, with units consistent with the measured coating surface density. A smaller uncertainty results in a narrower confidence interval, indicating higher policy confidence of the slow-response agent regarding the current oven temperature setpoint. When the fast-response agent receives an oven temperature setpoint with a wider confidence interval, it adjusts its own exploration noise amplitude according to an adaptive exploration strategy.

[0083] Furthermore, the feedback information transmitted by the uplink channel includes the cumulative average reward value within the current slow timescale layer sampling period, the average amplitude of the slurry pump speed adjustment, the adjustment frequency of the slurry pump speed adjustment, and the stability index of the coating surface density measurement value; the feedback information is summarized once at the end of each slow timescale layer sampling period and written into the message queue buffer of the uplink channel.

[0084] Preferably, the stability index of the coating surface density measurement value is represented by the standard deviation of the most recent 30 filtered coating surface density measurement values ​​within the current slow time scale layer sampling period. The smaller the standard deviation, the smaller the fluctuation of the coating process under the current oven temperature setting. The calculation method of the cumulative reward mean is the same as that in S2.4, which is represented by the arithmetic mean of the reward signals corresponding to all fast time scale layer sampling periods within the current slow time scale layer sampling period. The adjustment frequency of the slurry pump speed adjustment amount is represented by the number of times the absolute value of the slurry pump speed adjustment amount exceeds the preset adjustment amplitude threshold within the current slow time scale layer sampling period. The preset adjustment amplitude threshold is set to 1% of the rated value of the slurry pump speed data.

[0085] S4.3: Establish a processing mechanism for slow-response agents after receiving feedback information from the uplink channel.

[0086] Furthermore, at the beginning of each slow timescale layer sampling period, the slow response agent reads the latest feedback information from the message queue buffer of the uplink channel and maintains a sliding window of length 10. The sliding window sequentially stores the cumulative reward average corresponding to the most recent 10 slow timescale layer sampling periods. The slow response agent calculates the reward sliding mean, reward sliding variance, and reward change trend based on all the cumulative reward averages within the sliding window. The reward sliding mean, reward sliding variance, and reward change trend are incorporated into the state input of the slow response agent as a basis for evaluating the rationality of the current oven temperature setpoint.

[0087] Specifically, the reward change trend is calculated using a linear regression method. The mean of 10 cumulative rewards within the sliding window is used as the dependent variable, and the corresponding time series index is used as the independent variable. The resulting regression slope is the reward change trend. A positive reward change trend indicates that the control effect is continuously improving, while a negative reward change trend indicates that the control effect is continuously deteriorating. The reward sliding variance reflects the stability of the control effect of the fast-response agent under the current oven temperature setpoint. The smaller the reward sliding variance, the smaller the disturbance of the current oven temperature setpoint to the coating process.

[0088] Furthermore, the slow-response agent combines the stability index of the coating surface density measurement value with the reward sliding variance to evaluate the overall control effect of the current oven temperature setpoint. When both the stability index of the coating surface density measurement value and the reward sliding variance are lower than their respective preset thresholds, the current oven temperature setpoint is determined to be in a better state, and the slow-response agent outputs a small-amplitude oven temperature adjustment amount in the next slow time scale layer sampling period.

[0089] S4.4: Establish an integration mechanism for the fast-response intelligent agent to receive information from the downlink channel.

[0090] Furthermore, at the start of each fast timescale layer sampling period, the fast response agent reads the latest message from the downlink channel message queue buffer and incorporates the oven temperature setpoint and the difference between the oven temperature setpoint and the current filtered oven temperature data as two independent state components into the state space vector for use by the fast response agent's actor network inference. The difference between the oven temperature setpoint and the filtered oven temperature data reflects the current temperature tracking deviation of the oven, providing reference information on dynamic temperature changes for the fast response agent's slurry pump speed adjustment decision.

[0091] Specifically, the fast-response agent dynamically adjusts the volatility parameter of the Ornstein-Uhlenbeck process based on the confidence interval width input from the downlink channel. When the confidence interval width exceeds the preset confidence interval width threshold, the volatility parameter is increased by a preset exploration increment based on the current value to improve the fast-response agent's exploration capability under conditions of large uncertainty in the oven temperature setpoint. When the confidence interval width is lower than the preset confidence interval width threshold, the volatility parameter remains unchanged after decay. The preset confidence interval width threshold is set to 5% of the coating surface density target value, and the preset exploration increment is set to 0.05.

[0092] S4.5: Construct a time-weighted coordination optimization mechanism.

[0093] Furthermore, the multiple fast timescale layer sampling periods of the fast response agent within the slow timescale layer sampling period are mapped to the time bridging relationship of a single decision period of the slow response agent. At the beginning of each slow timescale layer sampling period, the reward buffer queue of the uplink channel is reset. The length of the reward buffer queue is consistent with the ratio of the slow timescale layer sampling period to the fast timescale layer sampling period, and is set to 12. The time weight of each reward signal in the reward buffer queue is set according to the time interval from the end of the current slow timescale layer sampling period. The reward signal closer to the end time corresponds to a larger time weight. The time weights are distributed in an arithmetic sequence, and the time weight corresponding to the most recent fast timescale layer sampling period is twice the time weight corresponding to the farthest fast timescale layer sampling period.

[0094] Preferably, the cumulative reward mean is represented by the weighted average of each reward signal in the reward cache queue and its corresponding time weight. This definition is consistent with the main embodiment in S2.4. The weighted average is better able to reflect the actual impact of the current oven temperature setpoint on the coating quality at the end of the slow time scale layer sampling period than the equal weighted arithmetic mean. The weighted average is written into the uplink channel message queue buffer for use by the slow response agent.

[0095] S4.6: Design an adaptive adjustment strategy for the layered interaction mechanism.

[0096] Furthermore, based on the training process stages of the training evaluation and early stop mechanism, the weights of the cumulative reward mean and the stability index of the coating surface density measurement value in the slow-response agent's state input are dynamically adjusted. In the early stage of training, the weight of the cumulative reward mean in the slow-response agent's state input is set to a higher value, and the weight of the coating surface density measurement value stability index is set to a lower value, so as to promote the slow-response agent to quickly explore the effective oven temperature setpoint range. In the later stage of training, the weight of the coating surface density measurement value stability index is gradually increased, and the weight of the cumulative reward mean is gradually decreased, guiding the slow-response agent to tend to output a stable and reliable oven temperature adjustment amount.

[0097] It should be noted that the initial training phase is defined as the stage where the number of consecutive evaluations does not exceed 5, and the later training phase is defined as the stage where the number of consecutive evaluations exceeds 5 and the online control performance index has entered the convergence range; the initial value of the cumulative reward mean weight is set to 0.7, and the final value is set to 0.3; the initial value of the coating surface density measurement stability index weight is set to 0.3, and the final value is set to 0.7; the adjustment step size of the weights of both is linearly transitioned according to the preset evaluation interval period.

[0098] In an optional embodiment, the adaptive adjustment strategy introduces an annealing-style coordination parameter adjustment scheme. When the online control performance index is lower than the preset convergence threshold for 10 consecutive evaluations, the Gaussian noise standard deviation of the slow-response agent is frozen to 0.02, and the exploration behavior of the slow-response agent is stopped to reduce unnecessary oven temperature fluctuations during production. In the main embodiment, the adaptive adjustment strategy only uses a linear transition adjustment of the cumulative reward mean weight and the coating surface density measurement stability index weight, without annealing-style freezing operation.

[0099] S4.7: Establish an exception handling process for a layered interaction mechanism.

[0100] Specifically, the anomaly handling process checks the timestamp of the latest message in the downlink message queue buffer at the beginning of each fast timescale layer sampling period. If the difference between the timestamp and the current time exceeds two slow timescale layer sampling periods, the downlink communication is determined to be interrupted. The fast-response agent continues to calculate the slurry pump speed adjustment using the most recently successfully received oven temperature setpoint and sends a downlink communication interruption warning signal to the control system. The anomaly handling process checks the timestamp of the latest message in the uplink message queue buffer at the beginning of each slow timescale layer sampling period. If the difference between the timestamp and the current time exceeds two slow timescale layer sampling periods, the uplink communication is determined to be interrupted. The slow-response agent keeps the current oven temperature setpoint unchanged and sends an uplink communication interruption warning signal to the control system.

[0101] Furthermore, the exception handling process performs data validity checks on the message content transmitted in the downlink and uplink channels. The data validity check rules include: the oven temperature setpoint must be within the range of the preset upper limit and the preset lower limit of the oven temperature; the cumulative average reward must be within the theoretical range of the reward signal; and the stability index of the coating surface density measurement must be non-negative. Message content that does not meet the data validity check rules is considered invalid. The fast response agent or the slow response agent replaces the invalid message with the corresponding field value of the previous valid message and records the number of times the invalid message occurs. When three consecutive messages are invalid, the same security response strategy as the communication interruption is triggered.

[0102] S5: Deploy the trained fast-response and slow-response agents to the coating production line control system. Based on real-time coating status data, output slurry pump speed control and oven temperature control respectively to form a layered time-scale adaptive closed-loop control of the coating process.

[0103] S5.1: Configure an industrial-grade computing server on top of the existing distributed control system in the coating production line, and install a deep learning inference engine on the industrial-grade computing server.

[0104] Furthermore, the industrial-grade computing server is equipped with 64GB of memory and an 8-core CPU for inference computation of fast-response and slow-response agents; the deep learning inference engine supports the efficient operation of the neural network model corresponding to the final weight parameters; a data acquisition interface is established to connect the industrial-grade computing server with a sensor network of flow sensors, temperature sensor arrays, speed sensors and areal density detectors, with a communication latency of no more than 5 milliseconds.

[0105] When manual operation is required, the industrial-grade computing server and the distributed control system are connected via an industrial Ethernet bus. The communication protocol of the industrial Ethernet bus adopts the OPC unified architecture standard. The data refresh cycle of the industrial Ethernet bus does not exceed 10% of the fast timescale layer sampling cycle to ensure that the fast-response agent obtains a valid state space vector at the beginning of each fast timescale layer sampling cycle.

[0106] S5.2: Export the final weight parameters of the trained fast-response agent and slow-response agent actor networks as inference-specific weight files, respectively. Perform lightweight processing on the inference-specific weight files to obtain the lightweight fast-response agent inference model and the lightweight slow-response agent inference model.

[0107] Specifically, the floating-point weight parameters of the actor network are quantized into 16-bit fixed-point representations. Sensitivity analysis is used to identify and remove neural connections in the actor network whose contributions are below a preset contribution threshold. The preset contribution threshold is determined by the following method: calculating the sensitivity of each weight parameter of the actor network to output changes (i.e., the average absolute value of the partial derivative of the output with respect to that weight), arranging all sensitivity values ​​in descending order, and setting the sensitivity value corresponding to a cumulative contribution rate of 95% as the preset contribution threshold; neural connections below this threshold are pruned and removed during the lightweighting process to minimize the computational load of the model while maintaining control accuracy; the single inference time of the lightweighted fast-response agent inference model and the lightweighted slow-response agent inference model on an industrial-grade computing server does not exceed 20% of the corresponding sampling period.

[0108] In an optional embodiment, the final weight parameters of the actor network for fast-response agents and the final weight parameters of the actor network for slow-response agents are compressed into a simpler student network using a knowledge distillation method. The number of hidden layer neurons in the student network is set to 50% of the number of hidden layer neurons in the original actor network, and the online control performance index of the student network on the validation dataset deviates from that of the original actor network by no more than 5%. In the main embodiment, the lightweight processing only uses 16-bit fixed-point quantization and neuron connection pruning, without knowledge distillation.

[0109] S5.3: Establish an online inference pipeline on an industrial-grade computing server, which includes a data preprocessing thread, a fast-response inference thread, and a slow-response inference thread.

[0110] Furthermore, the data preprocessing thread obtains the current slurry pump speed data, oven temperature data, coating speed data, and coating surface density measurement value from the data acquisition interface at a preset sampling period. It then obtains the filtered slurry pump speed data, filtered oven temperature data, filtered coating speed data, and filtered coating surface density measurement value according to the moving average algorithm. Finally, it constructs the state space vector at the current moment according to the normalization method and writes the state space vector into the shared memory buffer.

[0111] Furthermore, the fast-response inference thread operates on a fast timescale layer sampling cycle, reading the current state space vector from the shared memory buffer and the latest oven temperature setpoint from the downlink message queue buffer. It then concatenates the current state space vector with the latest oven temperature setpoint and feeds it into the lightweight fast-response agent inference model for inference, outputting the slurry pump speed adjustment. Conversely, the slow-response inference thread operates on a slow timescale layer sampling cycle, reading the current state space vector from the shared memory buffer and the latest cumulative reward average from the uplink message queue buffer. It then concatenates the current state space vector with the latest cumulative reward average and feeds it into the lightweight slow-response agent inference model for inference, outputting the oven temperature adjustment. The data preprocessing thread, fast-response inference thread, and slow-response inference thread run in parallel, exchanging data through the shared memory buffer without mutual blocking.

[0112] S5.4: Construct a control signal conversion and output mechanism to obtain the slurry pump speed control quantity and the oven temperature control quantity.

[0113] Specifically, the slurry pump speed control quantity is obtained by adding the slurry pump speed adjustment quantity output by the fast response inference thread to the current filtered slurry pump speed data; the oven temperature control quantity is obtained by adding the oven temperature adjustment quantity output by the slow response inference thread to the current oven temperature setpoint; the slurry pump speed control quantity and the oven temperature control quantity are respectively sent to the slurry pump actuator and the oven temperature control actuator via the industrial Ethernet bus at corresponding sampling periods.

[0114] Preferably, the slurry pump speed control quantity is processed by a first-order low-pass filter before being sent out, wherein the time constant of the first-order low-pass filter is set to 50% of the sampling period of the fast time scale layer to reduce the abrupt change amplitude of the slurry pump speed control quantity; the oven temperature control quantity is processed by a rate-of-change limiting process before being sent out, wherein the upper limit of the rate of change of the oven temperature control quantity is set to 2 degrees Celsius per minute to avoid thermal stress damage to the coating substrate caused by excessively rapid changes in oven temperature; the slurry pump speed control quantity after the first-order low-pass filter processing and the oven temperature control quantity after the rate-of-change limiting processing are respectively written into the downlink channel message queue buffer in S4.1 for the fast response agent to read in the next fast time scale layer sampling period.

[0115] S5.5: Establish a safety constraint and limiting module, in which the safety constraint and limiting module performs safety verification before the slurry pump speed control and oven temperature control are sent to the actuator.

[0116] Furthermore, the safety verification rules for the slurry pump speed control quantity are as follows: the slurry pump speed control quantity must be within the range of 20% to 100% of the rated speed of the slurry pump; if it exceeds the upper limit, it will be truncated to the upper limit; if it is below the lower limit, it will be truncated to the lower limit. The safety verification rules for the oven temperature control quantity are consistent with the preset upper limit and lower limit values ​​of the oven temperature in S2.1; the oven temperature control quantity exceeding the range will be truncated to the corresponding boundary value. The execution priority of the safety constraint and limiting module is higher than that of the online inference pipeline. At any time, the slurry pump speed control quantity and the oven temperature control quantity must be verified by the safety constraint and limiting module before they can be issued.

[0117] Furthermore, the safety constraint and limiting module sets safety interlock rules based on coating process knowledge. When the absolute value of the deviation between the measured coating surface density and the target coating surface density exceeds the preset interlock threshold for more than 30 seconds, the safety constraint and limiting module switches the slurry pump speed control and oven temperature control to preset conservative operating conditions and sends a safety interlock alarm signal to the operator. The preset interlock threshold can be set according to product specifications, such as ±8% of the target coating surface density. The specific value is determined by the process engineer based on the actual product quality tolerance range and is not limited to a fixed value. The preset conservative operating conditions are determined based on the combination of operating conditions with the smallest coating surface density deviation in historical coating production data. When the absolute value of the deviation between the measured coating surface density and the target coating surface density returns to below the preset interlock threshold and remains below it for more than 60 seconds, the safety interlock alarm signal is canceled, and the online inference pipeline resumes outputting the slurry pump speed control and oven temperature control.

[0118] S5.6: Design an online incremental learning mechanism to start the experience data collection module in the background without interrupting the normal operation of the online inference pipeline.

[0119] Specifically, the experience data collection module records the state space vector, slurry pump speed adjustment, oven temperature adjustment, reward signal, and cumulative reward average in the current production process using the same time-series database storage format as the data storage mechanism, forming incremental transfer samples. The incremental learning process is triggered when any of the following conditions are met: 1) The cumulative number of incremental transfer samples reaches 50% of the experience playback buffer capacity of the fast-response agent; 2) The online control performance index drops by more than 10% compared to the historical best performance and continues for 3 statistical periods; 3) The cumulative running time exceeds 8 hours after the production process parameters (such as slurry formulation and substrate type) are changed. The multi-condition triggering mechanism ensures that incremental learning can respond to changes in the production environment and product requirements in a timely manner, maintaining the adaptability of the control system.

[0120] Furthermore, the incremental learning process runs on the spare computing resources of an industrial-grade computing server. The incremental learning rate is one-tenth of the learning rate of the fast-response agent and one-tenth of the learning rate of the slow-response agent. Incremental updates are performed on the final weight parameters of the actor network of the fast-response agent and the slow-response agent, respectively. After the incremental update is completed, the incrementally updated weight parameters are verified according to the offline verification process. After the verification is passed, the incrementally updated weight parameters are loaded into the lightweight fast-response agent inference model and the lightweight slow-response agent inference model by hot replacement. The hot replacement process does not interrupt the operation of the online inference pipeline.

[0121] S5.7: Establish an anomaly detection and recovery mechanism, wherein the anomaly detection and recovery mechanism uses the root mean square value of the deviation between the measured value of the coating surface density and the target value of the coating surface density within the preset monitoring window length as the online control performance index, and simultaneously monitors the inference delay of the online inference pipeline and the downlink communication interruption warning signal and the uplink communication interruption warning signal.

[0122] Furthermore, when the online control performance index continuously exceeds the root mean square value threshold corresponding to the preset interlock threshold, or the inference delay exceeds 50% of the corresponding sampling period, or the downlink communication interruption warning signal and the uplink communication interruption warning signal are triggered simultaneously, the anomaly detection and recovery mechanism determines that the online inference pipeline is operating abnormally. It automatically switches the slurry pump speed control quantity and the oven temperature control quantity to the backup control strategy based on the preset PID controller, and records the state space vector, slurry pump speed control quantity, and oven temperature control quantity at the time of the anomaly as anomaly context information.

[0123] Specifically, the proportional coefficient, integral coefficient, and derivative coefficient of the preset PID controller are determined based on the optimal parameter tuning results under steady-state conditions in historical coating production data. When the online control performance index recovers to below the root mean square value threshold and the duration exceeds two slow time scale layer sampling cycles, the anomaly detection and recovery mechanism initiates a gradual switching process. Within 10 consecutive fast time scale layer sampling cycles, the slurry pump speed control quantity is linearly transitioned from the preset PID controller output value to the lightweight fast-response intelligent agent inference model output value. Within 2 consecutive slow time scale layer sampling cycles, the oven temperature control quantity is linearly transitioned from the preset PID controller output value to the lightweight slow-response intelligent agent inference model output value. The gradual switching process avoids abrupt changes in the slurry pump speed control quantity and the oven temperature control quantity at the switching moment.

[0124] S5.8: Establish a performance evaluation and reporting system.

[0125] Furthermore, the performance evaluation and reporting system uses a preset statistical period as its cycle, extracting the measured values ​​of coating surface density, target values ​​of coating surface density, slurry pump speed control quantities, and oven temperature control quantities from the time-series database within the corresponding statistical period, and calculating key performance indicators. Key performance indicators include the average absolute value of the deviation between the measured values ​​of coating surface density and the target values ​​of coating surface density, the maximum absolute value of the deviation, the standard deviation of the deviation, and the coating surface density pass rate. The coating surface density pass rate is defined as the proportion of sampling points whose absolute deviation is lower than the upper limit of the allowable deviation of the product specification to the total number of sampling points within the statistical period.

[0126] Furthermore, the preset statistical period is set to three levels: daily, weekly, and monthly. The performance evaluation and reporting system generates corresponding performance reports based on the key performance indicators of the three levels. The performance reports include the comparison results of the key performance indicators of the dual-agent reinforcement learning framework and the preset PID controller under the same production batch.

[0127] In an optional embodiment, a control model interpretation module is added to the performance evaluation and reporting system. This module performs sensitivity analysis on the input state components of the actor networks of fast-response and slow-response agents to identify the state components that have the greatest impact on the adjustment of slurry pump speed and oven temperature. The sensitivity analysis results are then incorporated into the performance report to provide a quantitative basis for coating process optimization. In the main embodiment, the performance evaluation and reporting system only outputs key performance indicators and comparison results, without the control model interpretation module.

[0128] In summary, this invention collects measurements of slurry pump speed, oven temperature, coating speed, and areal density from the coating production line, constructs a state space vector, and uses areal density deviation as a reward signal, providing real-time and accurate environmental perception and optimization targets for subsequent agents. It also constructs a hierarchical time-scale dual-agent reinforcement learning framework, independently modeling fast-response variables (slurry pump speed) and slow-response variables (oven temperature), effectively solving the coupling control problem caused by the dynamic response time differences of different variables during the coating process. Furthermore, it employs a deep deterministic policy gradient algorithm to train fast-response and slow-response agents separately, and establishes a coordination mechanism between them—the slow-response agent... The fast-response agent transmits the oven temperature setpoint to the slow-response agent as a state input, while the slow-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. This enables bidirectional information interaction and collaborative decision-making across time scales, avoiding oscillations or lags caused by time scale mismatch in traditional single controllers. The trained agents are deployed to the coating production line control system, and output slurry pump speed control and oven temperature control based on real-time state data, forming a layered time scale adaptive closed-loop control. This significantly improves the tracking accuracy and response speed of coating surface density to the target value, while reducing reliance on manual parameter adjustment and the risk of process fluctuations.

[0129] Based on the teachings of the above embodiments, other aspects of the present invention also propose an adaptive closed-loop control system for a coating process based on reinforcement learning, such as... Figure 2 As shown, it includes: The data acquisition and state construction module is used to collect the slurry pump speed, oven temperature, coating speed and coating surface density measurement values ​​of the coating production line, and construct a state space vector containing the slurry pump speed, oven temperature and coating speed, and use the deviation between the coating surface density measurement value and the target value as the reward signal. The dual-agent construction module is used to construct a hierarchical time-scale dual-agent reinforcement learning framework, which includes a fast-response agent that controls the slurry pump speed and a slow-response agent that controls the oven temperature. The policy training module is used to train fast-response and slow-response agents based on the deep deterministic policy gradient algorithm. The agent coordination module is used to establish a coordination mechanism between fast-response agents and slow-response agents. The slow-response agent transmits the current oven temperature setpoint to the fast-response agent as part of the state input, and the fast-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. The control execution module is used to deploy the trained fast-response and slow-response agents to the coating production line control system. Based on real-time coating status data, it outputs slurry pump speed control and oven temperature control respectively, forming a layered time-scale adaptive closed-loop control of the coating process.

[0130] This embodiment also provides a computer device applicable to the adaptive closed-loop control method for coating process based on reinforcement learning, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the adaptive closed-loop control method for coating process based on reinforcement learning as proposed in the above embodiment.

[0131] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0132] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the reinforcement learning-based adaptive closed-loop control method for the coating process as proposed in the above embodiments.

[0133] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0134] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for adaptive closed loop control of a coating process based on reinforcement learning, characterized in that, include: Collect the slurry pump speed, oven temperature, coating speed and coating surface density measurements of the coating production line, construct a state space vector containing the slurry pump speed, the oven temperature and the coating speed, and use the deviation between the coating surface density measurement value and the target value as a reward signal; A hierarchical time-scale dual-agent reinforcement learning framework is constructed, wherein the dual-agent reinforcement learning framework includes a fast-response agent that controls the slurry pump speed and a slow-response agent that controls the oven temperature. The fast-response agent and the slow-response agent are trained according to the deep deterministic policy gradient algorithm; A coordination mechanism is established between the fast-response agent and the slow-response agent. The slow-response agent transmits the current set value of the oven temperature to the fast-response agent as part of the state input. The fast-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. The trained fast-response and slow-response agents are deployed to the coating production line control system. Based on real-time coating status data, the slurry pump speed control and oven temperature control are output respectively, forming a layered time-scale adaptive closed-loop control of the coating process.

2. The reinforcement learning based coating process self-adaptive closed loop control method of claim 1, wherein, The slurry pump speed control value is obtained by adding the slurry pump speed adjustment value output by the fast response inference thread to the current filtered slurry pump speed data; the oven temperature control value is obtained by adding the oven temperature adjustment value output by the slow response inference thread to the current oven temperature setting value.

3. The adaptive closed-loop control method for coating process based on reinforcement learning as described in claim 2, characterized in that, It also includes, The fast-response inference thread reads the current state space vector from the shared memory buffer and the latest oven temperature setpoint from the downlink channel message queue buffer, using the fast time scale layer sampling period as the beat. The current state space vector and the latest oven temperature setpoint are concatenated and then sent to the lightweight fast-response agent inference model to perform inference and output the slurry pump speed adjustment amount. The slow-response inference thread reads the current state space vector from the shared memory buffer and the latest cumulative reward average from the uplink channel message queue buffer, using the slow time scale layer sampling period as the beat. The current state space vector and the latest cumulative reward average are concatenated and then sent to the lightweight slow-response agent inference model for inference, and the oven temperature adjustment amount is output.

4. The adaptive closed-loop control method for coating process based on reinforcement learning as described in claim 3, characterized in that, The final weight parameters of the trained fast-response and slow-response agent networks are exported as inference-specific weight files. These inference-specific weight files are then lightweighted to obtain lightweight fast-response agent inference models and lightweight slow-response agent inference models.

5. The adaptive closed-loop control method for coating process based on reinforcement learning as described in claim 4, characterized in that, Training fast-response and slow-response agents includes: Initialize the weight parameters of the Actor-Critic architecture, which consists of the actor network and the commentator network of the fast-response agent and the slow-response agent, respectively. Configure training hyperparameters for the deep deterministic policy gradient algorithm for the fast-response agent and the slow-response agent respectively; An exploration strategy is designed for the fast-response agent, and exploration noise based on the Ornstein-Uhlenbeck process is superimposed on the slurry pump speed adjustment output of the fast-response agent; Design an exploration strategy for the slow-response agent; The commenter network loss values ​​for the fast-response agent and the slow-response agent are calculated respectively using the deep deterministic policy gradient algorithm. Establish a training evaluation and early stop mechanism.

6. The adaptive closed-loop control method for coating process based on reinforcement learning as described in claim 5, characterized in that, The commentator network loss value is represented by the mean square error between the commentator network's estimate of the state-action value of the current transition sample and the estimate of the target state-action value; the target state-action value estimate is calculated by the target commentator network and the target actor network of the fast response agent based on the reward signal and the state space vector of the next fast timescale layer sampling period.

7. The adaptive closed-loop control method for coating process based on reinforcement learning as described in claim 6, characterized in that, The establishment of fast-response and slow-response agents includes: The dynamic influence time of slurry pump speed and oven temperature on the measured value of coating surface density was analyzed by tool coating process characteristics. The slurry pump speed was identified as a fast response variable and the oven temperature was identified as a slow response variable. Based on the difference in the dynamic influence time of the slurry pump speed and the oven temperature, a layered time scale control structure was established. Based on the hierarchical timescale control structure, a fast-response agent is constructed in the fast timescale layer, and a slow-response agent is constructed in the slow timescale layer. Both the fast-response agent and the slow-response agent adopt the Actor-Critic architecture composed of an actor network and a commentator network. Construct neural network models for the fast-response agent and the slow-response agent respectively; Configure independent experience replay buffers for the fast-response agent and the slow-response agent respectively; A training synchronization mechanism is established between the fast-response agent and the slow-response agent to coordinate the parameter update rhythm of the two agents.

8. An adaptive closed-loop control system for a coating process based on reinforcement learning, based on the adaptive closed-loop control method for a coating process based on reinforcement learning as described in any one of claims 1 to 7, characterized in that, include: The data acquisition and state construction module is used to collect the slurry pump speed, oven temperature, coating speed and coating surface density measurement values ​​of the coating production line, and construct a state space vector containing the slurry pump speed, the oven temperature and the coating speed, and use the deviation between the coating surface density measurement value and the target value as a reward signal. A dual-agent construction module is used to construct a hierarchical time-scale dual-agent reinforcement learning framework, wherein the dual-agent reinforcement learning framework includes a fast-response agent that controls the slurry pump speed and a slow-response agent that controls the oven temperature. The policy training module is used to train the fast-response agent and the slow-response agent according to the deep deterministic policy gradient algorithm. The agent coordination module is used to establish a coordination mechanism between the fast-response agent and the slow-response agent. The slow-response agent transmits the current set value of the oven temperature to the fast-response agent as part of the state input. The fast-response agent feeds back the accumulated reward information to the slow-response agent for strategy optimization. The control execution module is used to deploy the trained fast-response agent and the slow-response agent to the coating production line control system. Based on real-time coating status data, it outputs slurry pump speed control quantity and oven temperature control quantity respectively, forming a layered time-scale adaptive closed-loop control of the coating process.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the adaptive closed-loop control method for coating process based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the adaptive closed-loop control method for coating process based on reinforcement learning as described in any one of claims 1 to 7.

Images