Polyacrylamide production process monitoring data analysis system based on reinforcement learning

Abstract

The invention relates to the technical field of industrial process control and data analysis, in particular to a polyacrylamide production process monitoring data analysis system based on reinforcement learning, which comprises a data acquisition step: acquiring process data at a first frequency, and receiving quality data at a second frequency; a strategy generation step: outputting an action by using a neural network, and calculating an instant agency reward by using an agency model; an asynchronous experience playback step: constructing a nonlinear storage structure, and mapping lagging quality data back to historical process data fragments to complete a causal closed loop; a model updating step: calculating a delay correction reward, and updating network parameters based on the mixed reward; according to the invention, decoupling of real-time decision and delay calibration is realized through a dual-time scale architecture, and the response capability and high-precision control of a test vacuum period are ensured.

Description

Technical Field

[0001] This invention relates to the field of industrial process control and data analysis technology, specifically to a reinforcement learning-based monitoring and data analysis system for polyacrylamide production processes. Background Technology

[0002] Production process monitoring data analysis refers to the real-time estimation and control of the final quality indicators of production products based on sensor data streams collected from the industrial site. Current production process monitoring methods mainly include three types: simulation method based on physicochemical mechanism equations, process control method based on statistics, and data-driven method based on deep neural networks. However, when conducting production monitoring and analysis based on existing technologies, the monitoring model lacks real-time feedback signals while waiting for test results due to the high frequency of process sensing data and the significant time lag in quality testing data during chemical production. Relying solely on mechanistic models or data models makes it difficult to balance the response to instantaneous physical states with the calibration of cumulative errors, and there are also difficulties in fusion of heterogeneous data, all of which reduce the accuracy of predicting and controlling the quality of the final product. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a reinforcement learning-based monitoring and data analysis system for polyacrylamide production processes. Specifically, the technical solution of this invention includes: The data acquisition module is configured to receive process sensing data streams at a first frequency and quality detection data streams at a second frequency. The strategy generation module is configured to generate real-time monitoring strategies based on process sensing data streams using a first neural network, and to calculate instant agent rewards based on process sensing data streams using a pre-set agent model. The asynchronous experience playback module is configured to build an asynchronous experience playback buffer, mapping the quality inspection data stream back to the corresponding historical process sensor data fragments; The model update module is configured to calculate the delay correction reward using the mapped quality inspection data stream, and update the parameters of the first neural network based on the weighted combination of the immediate agent reward and the delay correction reward; wherein, the second frequency is lower than the first frequency; the process sensing data stream represents the real-time physical state of the production equipment, and the quality inspection data stream represents the final physicochemical indicators of the production products.

[0004] Preferably, the strategy generation module includes: a feature extraction unit configured to extract local spatial features of the process sensing data stream using a one-dimensional convolutional neural network and extract time-dependent features of the process sensing data stream using a long short-term memory network to generate a real-time state vector; and a proxy reward calculation unit configured to input the real-time state vector into a preset proxy model and output a virtual evaluation value representing the current production state conversion rate as an immediate proxy reward; wherein, the preset proxy model is a calculation model pre-constructed based on the physicochemical mechanism equations of the production process.

[0005] Preferably, the asynchronous experience playback module includes: a timestamp alignment unit, configured to record the generation timestamp of each frame of data and store it in the asynchronous experience playback buffer when the process sensing data stream is received; a backtracking mapping unit, configured to retrieve historical process sensing data segments that match the generation timestamp in the asynchronous experience playback buffer according to a preset lag time parameter when the quality detection data stream is received; and a causal association unit, configured to use the quality truth value in the quality detection data stream as a label and bind it to the retrieved historical process sensing data segments to form a complete experience sample containing delay feedback.

[0006] Preferably, the model update module includes: a deviation calculation unit, configured to calculate the numerical deviation between the true quality value in the quality detection data stream and the virtual evaluation value output by the surrogate model at historical moments; and a backpropagation unit, configured to use the numerical deviation as a delay correction reward, utilizing... - The algorithm backpropagates the delayed correction reward to the historical decision steps of the first neural network; the parameter correction unit is configured to increase the weight of the delayed correction reward in the weighted combination when the numerical deviation exceeds a preset safety threshold, so as to dominate the gradient update direction of the first neural network.

[0007] Preferably, the system further includes: a counterfactual reasoning module, configured to generate virtual adverse working condition data using a generative adversarial network based on the current process sensing data stream; and a risk warning module, configured to input the virtual adverse working condition data into a first neural network, and trigger a warning signal if the predicted state output by the first neural network exceeds the safe range; wherein, the virtual adverse working condition data is synthetic data with parameter drift or sudden noise artificially introduced.

[0008] Preferably, the system also includes a slow timescale calibrator configured to utilize The encoder processes the quality inspection data stream and extracts long-term quality trend features; the strategy generation module is also configured to concatenate the long-term quality trend features with the process sensing data stream as the input of the first neural network.

[0009] Preferably, the production process is a polymerization reaction process of polyacrylamide; the process sensing data stream includes: temperature, pressure, stirring motor current and initiator flow rate of the reactor; the quality detection data stream includes: molecular weight distribution data and degree of hydrolysis data of the polyacrylamide product; the hysteresis time parameter is 4 hours to 6 hours.

[0010] Preferably, the system includes: an edge computing node deployed on the production site, configured to run a data acquisition module and a strategy generation module, and perform millisecond-level real-time inference; and a cloud server configured to run an asynchronous experience playback module and a model update module, which, in response to receiving new quality inspection data streams, performs event-triggered model training and parameter distribution, or performs periodic training synchronized with the quality inspection frequency.

[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention effectively solves the problem of fusion of heterogeneous data in chemical production, namely high-frequency, low-quality process data and high-quality, lagging quality data, by constructing a dual-timescale data processing architecture. Through a strategy generation module, a pre-set agent model is used to calculate real-time agent rewards, filling the feedback vacuum during the waiting period for test data. This achieves the effect of maintaining normal system operation based on physical laws even in the absence of real labels. An asynchronous experience playback module constructs a non-linear time-sequential storage structure, using timestamp alignment and backtracking mapping mechanisms to accurately map delayed quality data back to corresponding historical process sensor data fragments, realizing a closed-loop connection of the causal chain. Compared to traditional methods, this application decouples real-time decision-making from delay calibration, ensuring millisecond-level real-time response capabilities while utilizing highly reliable test data to achieve precise control of long-delay processes. 2. This invention significantly improves the physical consistency and cold-start capability of policy generation through the deep fusion of deep learning feature extraction and physical mechanism equations; by combining the feature extraction unit with... - and The network can capture the local spatial coupling characteristics and time-dependent characteristics between sensors; in particular, it employs an explicit Euler method combined with a unit delay operator. By decoupling and discretizing the kinetic equations, the algebraic loop dependency between concentration and rate is completely resolved at the mathematical model level. Furthermore, a concentration iteration calculation based on variable volume mass conservation is introduced, which accurately reflects the dilution effect caused by the feeding operation. This design enables the system to have basic monitoring capabilities during the cold start phase without any test data, avoiding the blind exploration of the pure data-driven model in the early training stage and ensuring the convergence speed and safety of the model under physical constraints. 3. This invention effectively eliminates the cumulative bias of the physical model by establishing a hindsight model update mechanism based on dynamic weighted combination; it uses a bias calculation unit to map the cumulative reaction rate to a quality estimate using a linear regression model and calculates the calibration deviation between the estimate and the actual test value; and it sets a parameter correction unit based on historical statistics. A safety threshold is set; when the deviation exceeds this threshold, the weight of the delay correction reward is automatically increased. This mechanism forces the model update direction to align with real test data; it utilizes sparse but high-confidence test data to dominate gradient updates, effectively correcting physical model failures caused by equipment scaling or sensor drift, thus ensuring real-time performance while leveraging... - The algorithm backpropagates precise quality feedback to historical decision steps, enabling high-precision prediction and control of the final product quality. 4. This invention enhances the robustness and engineering applicability of the system by introducing a counterfactual reasoning module and a cloud-edge collaborative architecture; it also enhances the system's robustness and engineering applicability by constructing a system with an auxiliary regression head. - The network, combined with a gradient penalty mechanism, can generate virtual harsh working condition data that strictly follows physical drift instructions. Then, the risk warning module can detect the safety boundary of the strategy under parameter perturbation, effectively defending against potential black swan risks. At the same time, by deploying edge computing nodes to perform millisecond-level real-time inference and cloud servers to perform high-load asynchronous backtracking training, the computing power requirements and real-time requirements are balanced. The edge ensures production safety under network disconnection, while the cloud uses a massive historical experience database to achieve continuous model evolution, thus realizing an organic combination of lightweight deployment and intelligent upgrade. Attached Figure Description The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1 This is a structural diagram of the system of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0013] Example 1: Please see Figure 1 A reinforcement learning-based monitoring and data analysis system for polyacrylamide production process, comprising: The data acquisition module is configured to receive process sensing data streams at a first frequency and quality detection data streams at a second frequency. The strategy generation module is configured to generate real-time monitoring strategies based on process sensing data streams using a first neural network, and to calculate instant agent rewards based on process sensing data streams using a pre-set agent model. The asynchronous experience playback module is configured to build an asynchronous experience playback buffer, mapping the quality inspection data stream back to the corresponding historical process sensor data fragments; The model update module is configured to calculate the delay correction reward using the mapped quality inspection data stream, and update the parameters of the first neural network based on the weighted combination of the immediate agent reward and the delay correction reward; wherein, the second frequency is lower than the first frequency; the process sensing data stream represents the real-time physical state of the production equipment, and the quality inspection data stream represents the final physicochemical indicators of the production products.

[0014] This embodiment discloses a reinforcement learning-based polyacrylamide production process monitoring data analysis system. This system aims to solve the problem of heterogeneous data fusion in chemical production, where process data is high-frequency but low-quality, and quality data is high-quality but extremely lagging. The system executes a data acquisition step, where the data acquisition module connects to field instruments via a communication link, including but not limited to a wired bus or industrial wireless network, at a first frequency. That is, 1 By adjusting the sampling rate, signals from the temperature sensor, pressure transmitter, and current transformer of the stirring motor in the reactor are acquired to construct a set representing the real-time physical state of the production equipment. Meanwhile, this module through The laboratory information management system interface, at a second frequency That is, receiving a set of data characterizing the final physicochemical properties of the production products every 4 to 6 hours. ; The strategy generation module utilizes the first neural network as the system's fast, intuitive brain, at any given time... Based on only Output control actions; during this period, in order to solve To address the problem of the algorithm lacking immediate feedback, the system runs a proxy model based on reaction kinetic equations in parallel to calculate the immediate proxy reward in real time. ; The asynchronous experience replay module constructs a non-linear, time-sequential storage structure, so that when the system is at time... Received quality data At that time, this module is responsible for retrieving the timestamp from the buffer. Historical process data fragments, and This serves as a label attached to the fragment, completing the closed loop of the causal chain; The model update module is configured to calculate the latency correction reward using the mapped quality inspection data stream, and based on a weighted combination of the immediate agent reward and the latency correction reward. Update the parameters of the first neural network; where the weights... The weighting coefficient for the delayed correction reward, which is adaptively adjusted based on quality prediction deviation, is initially set to 0.2 during the cold start phase; an hindsight-based update mechanism is adopted, which not only utilizes... Online fine-tuning is performed, and correction terms are calculated using real-world quality data that arrives late. The previous decisions were significantly re-weighted. This embodiment achieves decoupling of real-time decision-making and delay calibration by constructing a data processing architecture with dual time scales. The immediate agent reward ensures the system's real-time response capability, enabling it to work normally according to physical laws even during the vacuum period of waiting for test data in chemical production processes with large lag feedback characteristics. Meanwhile, the delay correction reward uses highly reliable test data to eliminate the accumulated deviation of the physical model, thereby ensuring real-time performance while using an asynchronous backtracking mechanism to achieve high-precision prediction and control of the final product quality.

[0015] Example 2: The strategy generation module includes: The feature extraction unit is configured to extract local spatial features of the process sensing data stream using a one-dimensional convolutional neural network and extract time-dependent features of the process sensing data stream using a long short-term memory network to generate a real-time state vector. The agent reward calculation unit is configured to input the real-time state vector into a preset agent model and output a virtual evaluation value representing the current production state conversion rate as an immediate agent reward; wherein, the preset agent model is a calculation model pre-constructed based on the physicochemical mechanism equations of the production process.

[0016] This embodiment features a structured design for the strategy generation module, specifying the detailed hyperparameter settings of the deep learning model to ensure the reproducibility of the technical solution. The feature extraction unit receives the process sensor data stream, which is then processed by a convolutional layer consisting of three layers. - The aim is to extract the instantaneous coupling characteristics, i.e., local spatial characteristics, between different sensors such as temperature and pressure; The specific network structure is as follows: the first layer has a convolutional kernel size of 7, 32 filters, a stride of 1, and an activation function of [function name missing]. The second convolutional layer has a kernel size of 5, 64 filters, and an activation function of . The third convolutional layer has a kernel size of 3, 128 filters, and an activation function of . Batch normalization is added between layers to accelerate convergence; the output feature map is fed into two layers. Networks are used to capture long-term dependencies over time, such as the rate of temperature rise in a reactor. The number of hidden units in the layer is set to 256, and the ratio between layers is set to 0.2. To prevent overfitting, the final output real-time state vector St is compressed to a 64-dimensional latent space representation via a fully connected layer, which condenses all the dynamic information of the production process at the current moment. Based on this, the first neural network constructs a strategy head, configured as a fully connected layer structure. The output layer contains two neurons, corresponding to the opening setpoint of the monomer feed regulating valve and the stroke frequency setpoint of the initiator metering pump, respectively. The activation function maps the output to the normalized action space [-1, 1], which is then converted into actual control commands by the denormalization module and issued to the target. The system; simultaneously, the agent reward calculation unit inputs the real-time state vector into a pre-set agent model, which is constructed based on the Arrhenius equation and the law of conservation of mass, and is used to fill the reward vacuum during the waiting period for test results in reinforcement learning; in order to completely resolve the algebraic ring dependency between concentration and rate at the mathematical model level, this embodiment explicitly adopts the explicit Euler method combined with the unit delay operator. The dynamic equations are decoupled and discretized; this instant agent reward The parameter optimization of the first neural network is designed to characterize the instantaneous conversion rate and serves as a feedback signal. Its calculation formula is as follows: in, The pre-exponential factor is a proportionality coefficient characterizing the relationship between the chemical reaction rate constant and temperature. It is derived from data fitting of historical production batches, and its unit is set to 1. To match the overall order of a 1.5-order reaction; Activation energy of polymerization reaction, unit: ; Let be the ideal gas constant, and take the value of . ; The real-time absolute temperature inside the reactor is derived from the real-time data collected by the temperature sensor, and the unit is... ; Real-time estimated monomer concentration, in units of ; Real-time estimated initiator concentration, in units of ; and The reaction order is set as follows in this embodiment: and This is used to reflect the contribution of each component concentration to the reaction rate; through the above physicochemical mechanism equation, the system can provide reinforcement learning with immediate feedback with clear physical meaning, solving the reward sparsity problem in the environment of large lag in chemical reactions; To accurately reflect the volume changes and dilution effects caused by the feeding operation and to break the algebraic loop, the system introduces a delay operator. Its definition is Construct an explicit iterative formula to update the reaction volume in real time: in, and These are the real-time collected monomer feed flow rate and initiator pump injection flow rate, respectively, with units uniformly converted to... ; Calculate the concentration: Expanding into numerical calculation form, it is as follows: in, The feed volumetric flow rate is a process variable that is collected in real time by a Coriolis mass flow meter deployed on the individual feed line; although this variable is not listed in the basic sensor list of Example 7, it is synchronously collected into the data stream as a necessary calculation input. The standard molar concentration for a single raw material storage tank is set as follows: To address the loop definition issue in the early stages of iteration, the system is set to... Instant agent rewards The initial value is 0, and the initial monomer concentration is... The concentration of the base liquid added to the reactor; Real-time estimated initiator concentration, in units of Also based on the delay operator Perform an explicit update: in, For real-time acquisition of initiator pump flow rate, The standard concentration for the raw material storage tank is set as follows: , Let be the thermal decomposition rate constant of the initiator, and take a value of . ; The reaction order, derived from reaction kinetic experiments, is dimensionless; in the specific settings of this embodiment, these two parameters are set as fixed constants, for example, for the polyacrylamide system, taking... and ; Based on the above delay operator The explicit discretization process in this embodiment clearly defines the use of the rate from the previous time step at the mathematical model level. Update current concentration Then utilize the current concentration Calculate the current rate The acyclic computation path ensures the determinism and stability of the numerical simulation code; This embodiment combines the data feature extraction capabilities of deep learning with the interpretability of physical mechanisms. The strategy generation module can both perceive subtle fluctuations in data and follow basic chemical reaction laws. In particular, by introducing concentration iteration calculation based on variable volume mass conservation, the shortcomings of the traditional constant volume integral method in reflecting the feeding dilution effect are solved, making the physical perception capability more accurate. This enables the system to have basic monitoring capabilities in the cold start phase, i.e., when there is no test data, avoiding the blind exploration of the pure data-driven model in the early stage of training, and ensuring the convergence speed and safety of the model under physical constraints.

[0017] Example 3: The asynchronous experience replay module includes: The timestamp alignment unit is configured to record the generation timestamp of each frame of data and store it in the asynchronous experience playback buffer when the process sensing data stream is received. The backtracking mapping unit is configured to, upon receiving a quality inspection data stream, retrieve historical process sensor data segments that match the generated timestamp from the asynchronous experience playback buffer according to a preset lag time parameter. The causal association unit is configured to use the true quality value in the quality inspection data stream as a label and bind it to the retrieved historical process sensor data fragments to form a complete empirical sample including delayed feedback.

[0018] This embodiment details the internal logic of the asynchronous experience playback module; the timestamp alignment unit maintains a circular buffer. Upon receiving the process sensing data stream, record the generation timestamp of each frame of data. The data is stored in this buffer, where each sample tuple contains the state, action, agent reward, and timestamp; when the backtracking mapping unit receives the quality inspection data stream, it reads the sampling time attached to the record. ; in response The system pushes a quality record. The unit is based on a preset lag time parameter. In the buffer Search time window All state sequences within ,in, To allow for time jitter windows, considering the transmission latency of industrial Ethernet in the production environment and To address the clock synchronization error with the server, this embodiment will jitter the window. Explicitly set at 3000 , that is, 3 To ensure correct matching of corresponding data frames even under network fluctuations; the causal correlation unit performs a delayed binding operation to transfer the true quality value from the quality detection data stream. As tags, they are bound to retrieved historical process sensor data fragments to generate enhanced samples; This embodiment overcomes the asynchronous nature of sampling and testing in time through an explicit timestamp alignment mechanism. This module reconnects the operational causes and quality results that were originally broken on the timeline, enabling the reinforcement learning algorithm to learn based on real causal relationships rather than based on erroneous temporal associations, thereby establishing an accurate credit allocation path in long-lag process control.

[0019] Example 4: The model update module includes: The deviation calculation unit is configured to calculate the numerical deviation between the true quality value in the quality inspection data stream and the virtual evaluation value output by the surrogate model at historical moments; The backpropagation unit is configured to use numerical deviation as a delay correction reward, utilizing... - The algorithm backpropagates the delayed reward to the historical decision steps of the first neural network; The parameter correction unit is configured to increase the weight of the delay correction reward in the weighted combination when the numerical deviation exceeds a preset safety threshold, so as to dominate the gradient update direction of the first neural network.

[0020] This embodiment details the parameter correction mechanism and specific algorithm implementation process of the model update module, particularly the specific mathematical methods used to solve the physical model bias problem by constructing a dynamic weighted reward function; the bias calculation unit defines a calibration bias. Used to calculate the true quality value in the quality inspection data stream. The virtual evaluation value output by the proxy model at historical moments The numerical difference between them; among which, Instant agent rewards within historical time windows Perform discrete-time integration and via polynomial mapping The obtained cumulative conversion estimate; in order to eliminate the uncertainty of the black-box description of polynomial mapping, this embodiment explicitly... For a first-order linear regression model, its expression is: in, With the true value of quality The physical unit for all of them is clearly defined as weight-average molecular weight. ; This is the molecular weight growth factor corresponding to the unit conversion rate, and its unit is set to... To ensure that the dimensions of the formulas are consistent; The base molecular weight at which the reaction begins, in units of These two parameters were obtained by least-squares fitting of historical production batches; among them, The length of the integration time window is determined by the lag time parameter. Divide by sampling period Confirmed, in this embodiment Take 4 hours, or 14400 seconds, at a sampling frequency of 1. ,Right now seconds, step; For linear mapping models Regarding the physical rationale, this embodiment points out that although the full-cycle kinetics of polymer polymerization is nonlinear, in the mid-reaction phase monitored by this system, i.e., the region of significant gelation effect, the chain growth rate and monomer consumption rate exhibit a high degree of coupling. Statistical analysis based on historical data shows that within this specific process operating domain, the cumulative reaction rate and molecular weight growth show a significant linear positive correlation. The coefficient is greater than 0.9; therefore, a first-order linear model is used as the reward mapping function, with parameters... Essentially, it characterizes the average chain length growth modulus corresponding to the conversion of a unit monomer under this process condition. This satisfies the efficiency requirements of real-time calculation while remaining within the allowable error range in engineering. <5%, providing effective physical gradient guidance; Backpropagation unit calculates delay correction reward To address the issue that simple numerical deviations cannot directly guide decision-making at each step, this embodiment employs a proportional calibration method to distribute macroscopic quality deviations across microscopic time steps. Specifically, considering the risk of numerical instability where the denominator may approach zero, a minimum smoothing term is introduced. Delayed correction reward Defined as: The formula uses a scaling factor between the actual and predicted values ​​to scale the instantaneous rate calculated by the physical model, thereby forcibly calibrating its cumulative total while preserving the reaction trend inferred by the physical model, i.e., the time profile. Based on this, a hybrid reward function The calculation formula is as follows, using a dynamic weighted combination: in, The confidence coefficient for the delay weight; this formula ensures that when When the value approaches 1, the system relies entirely on data calibrated from real test data. Learn to revise the physical model Amplitude error; the backpropagation unit utilizes The algorithm will combine the rewards Backpropagation is performed to the historical decision steps of the first neural network, where... The computation is backtracked to historical moments via an asynchronous experience replay buffer. Execution; given that the first neural network employs a strategy head and a value head... - In order to ensure that the control strategy and value assessment are optimized in sync, the backpropagation unit calculates the timing difference error. And perform dual parameter updates: for the value head, utilize the qualification trace Minimize prediction error; for the policy head, utilize The policy gradient ascent is used as the advantage function estimate for updating; In this algorithm, the discount factor Set to 0.99 to focus on long-term returns, trace decay parameter Set to 0.95 to balance bias and variance, the joint update formula performed by the parameter correction unit is as follows: in, The learning rate is set to a value of [value to be filled in]. , The strategy loss coefficient has a value of [value]. This ensures that the algorithm can converge correctly based on the delayed-corrected hybrid reward and generate an effective monitoring strategy. The parameter correction unit performs adaptive weight adjustment; a preset safety threshold is used. Set as ,in, The system predicts deviations based on the quality of the most recent 100 batches in the historical database. The rolling standard deviation, calculated statistically, is used to characterize the baseline noise level of the system under current operating conditions; when the system has accumulated less than 100 batches of historical data, This is set as a preset empirical constant; in this embodiment, it is taken as the upper limit of the allowable standard deviation of the product's molecular weight, i.e., 5000. To ensure safety during the cold start phase; when numerical deviations When the physical model fails, the deviation amplitude increases linearly. : in, Adjust the slope This mechanism achieves a logical closed loop: when the deviation is small, the system mainly utilizes dense physical rewards. Fast convergence; when the deviation exceeds the safety threshold, Rapidly increasing, forcing reward signals Towards By converging, the sparse but high-confidence test data can be used to guide the gradient update direction of the first neural network, effectively correcting the cumulative error caused by equipment scaling or sensor drift. In this embodiment, Setting it to 0.2 aims to ensure that certain physical constraints are retained in the initial stage of the system, while adjusting the slope... Setting it to 2.0 is to quickly increase the weight of the real quality feedback when the deviation triggers the safety threshold, thereby enabling agile correction of the physical model drift.

[0021] Example 5: The system also includes: The counterfactual reasoning module is configured to generate virtual harsh operating condition data based on the current process sensor data stream using a generative adversarial network. The risk warning module is configured to input virtual harsh working condition data into the first neural network. If the predicted state output by the first neural network exceeds the safe range, a warning signal is triggered. The virtual harsh working condition data is synthetic data with parameter drift or sudden noise artificially introduced.

[0022] This embodiment details the internal structure of the counterfactual reasoning module, particularly the generative adversarial network. How to ensure that the generated virtual data strictly follows the artificially introduced drift instructions; the system constructs an auxiliary classification generative adversarial network. - Architecture; Generator Receive real status Random noise and drift control vector ,For example The virtual harsh operating condition data is output, corresponding to the temperature drift and current drop, respectively. ; To solve the traditional The generated data cannot be guaranteed to contain specific physical characteristics, which is the problem of black-box generation. This embodiment addresses this issue in the discriminator. An auxiliary regression head was added in parallel. Discriminator Not only outputs the probability of true or false judgment It also outputs a predicted value for the amount of hidden drift in the input sample. The training process employs a multi-task loss function: in, For standard Combating losses; in order to overcome The gradient vanishing or exploding problem during training, and satisfying 1- To address the continuity constraint, this embodiment introduces a gradient penalty mechanism, adding the following formula to the loss function: Among them, the penalty coefficient Set to 10. This is a random interpolation between real samples and generated samples; The weighting factor for the regression loss is set to 10; the above weighting coefficients... and The value of is determined based on the principle of dimensional balance, aiming to keep the adversarial loss, gradient penalty term, and regression loss consistent in magnitude, ensuring that the generator can balance the authenticity of the data and the accuracy of the drift instructions during the optimization process; this loss function forces the generator to... The input drift command must be Accurately encoded into the generated virtual data Otherwise, the regression head of the discriminator cannot be fooled. This ensures that when the system requests to generate a temperature drift of +2°C, the generated feature vectors do indeed correspond to the temperature rise state in the physical manifold, rather than being randomly generated noise. The risk warning module will verify this through physical constraints. Input the first neural network; in this embodiment, the first neural network is designed as a complete backbone multi-head architecture, which includes the policy head and value head of embodiment 4 to execute - Building upon the control mechanism, an additional, independent state prediction head is integrated in parallel. The policy head outputs action vectors, the value head executes comment functions, and the state prediction head is specifically used to map and output predicted values ​​of key quality indicators under the current operating condition, such as the polydispersion index. The width used to characterize the molecular weight distribution; The state prediction head is specifically configured as a fully connected neural network structure: the input layer is connected to the 64-dimensional hidden space of the shared encoder, and the first hidden layer contains 32 neurons. Activation function; the output layer contains one neuron, using... Activation function to output continuous Predicted value; interlayer ratio of 0.1 is provided. Layers are added to enhance generalization ability; if the network's state prediction head predicts in Product quality indicators under certain conditions, such as molecular weight distribution width This will exceed the technological limit, that is If the value is greater than 4.5, an early warning is triggered; this early warning mechanism, based on counterfactual reasoning, is essentially probing the current strategy. Robust boundaries within the parameter perturbation neighborhood effectively defend against potential black swan equipment failure risks.

[0023] Example 6: The system also includes a slow timescale calibrator, configured to utilize The encoder processes the quality inspection data stream and extracts long-term quality trend features. The strategy generation module is also configured to concatenate long-term quality trend features with process sensing data streams as input to the first neural network. This embodiment supplements the key time-scale calibrator implementation details; considering the non-uniform sampling characteristics of the quality inspection data stream, with sampling intervals fluctuating between 4 and 6 hours, traditional position encoding based on fixed indices is no longer applicable; this embodiment adopts a sinusoidal position encoding mechanism based on continuous time differences; the system calculates the... The second sampling and the first Time difference between sampling And calculate the cumulative absolute time. ; Before performing position encoding, for the dimensions of the quality inspection data stream, such as 2D, and... For model dimensions, such as 64 dimensions, there is a mismatch issue. The system performs a feature embedding operation: utilizing a learnable linear projection layer. and bias The original quality data Mapped to high-dimensional feature vectors ;use Generate time-coded vectors , its first and The formula for calculating dimension is: in, for The input embedding dimension; this encoded vector is directly embedded with the quality indicator features after linear projection. Adding, that is As The encoder input; this processing method explicitly maps non-uniform time intervals to a high-dimensional feature space, enabling the self-attention mechanism to correctly understand the crucial temporal information of whether the most recent test was 4 hours ago or 6 hours ago, thereby extracting long-term quality trend features. The appropriate weight decay characteristics are assigned to different historical samples at the same time; finally, the policy generation module will... With real-time status By stitching together, multimodal feature fusion is achieved under non-uniform time scales.

[0024] Example 7: The production process involves the polymerization reaction of polyacrylamide. The process sensing data stream includes: reactor temperature, pressure, agitator motor current, and initiator flow rate; The quality testing data stream includes: molecular weight distribution data and degree of hydrolysis data of polyacrylamide products; the hysteresis time parameter is 4 to 6 hours.

[0025] This embodiment specifically defines the specific parameters of the polyacrylamide (PAM) polymerization reaction process; the system is configured with a process sensor data stream acquisition channel, including using a PT100 thermal resistor to acquire the reactor temperature in the range of 0-100℃, monitoring the absolute pressure inside the reactor to confirm the nitrogen protection status, acquiring the stirring motor current to reflect the viscosity change of the system, and monitoring the initiator flow rate of the redox initiation system; In addition, to fully support the reaction kinetics calculations of Example 2, the system is also equipped with a Coriolis mass flow meter to collect monomer feed flow rate in real time. This ensures the accuracy of monomer concentration estimation; simultaneously, the system is configured with a data stream receiving channel for quality testing, including data from gel permeation chromatography. The measured molecular weight distribution data, and the characterization The degree of hydrolysis data includes the proportion of amide groups converted to carboxyl groups on the molecular chain; additionally, the system sets the hysteresis time parameter to 4 to 6 hours, based on... The production process includes sampling, dissolving, and filtering. The physical time limit required to complete chromatographic analysis; This embodiment addresses... By modeling specific variables of the production process, the nonlinear mapping relationship between the gel effect (i.e., the self-accelerating exothermic phenomenon) and the final molecular weight is accurately captured. This targeted parameter setting solves the long-standing problem in this field that it is difficult to adjust the reaction through mid-process testing once it has started, ensuring the applicability and high accuracy of the model in specific chemical scenarios.

[0026] Example 8: The system includes: Edge computing nodes are deployed in the production site and configured to run data acquisition and strategy generation modules, performing millisecond-level real-time inference. The cloud server is configured to run the asynchronous experience replay module and the model update module. In response to receiving new quality inspection data streams, it performs event-triggered model training and parameter distribution, or performs periodic training synchronized with the quality inspection frequency.

[0027] This embodiment describes the system's cloud-edge collaborative architecture; edge computing nodes are deployed on the production site. Next to the cabinet, an embedded design is used. The module loads only the pre-trained first neural network and agent model, and is configured to run the data acquisition and policy generation modules. Its purpose is to ensure that even in the event of a network outage, it can still achieve speeds of less than 50. The delayed output monitoring strategy; at the same time, the cloud server is configured to run a high-load asynchronous experience playback module and model update module, which uses massive storage to receive compressed historical data uploaded by the edge terminal; after receiving the test report from the LIMS system, the cloud server performs backpropagation training and sends the updated model parameter package to the edge node in the next synchronization cycle; This embodiment balances computing power requirements with real-time requirements through a cloud-edge collaborative architecture. High-frequency inference tasks are completed locally, avoiding the impact of network latency on production safety, while high-energy-consuming training and backtracking tasks are completed in the cloud, utilizing the cloud's unlimited storage to build a long-term historical experience database, thereby achieving an organic combination of lightweight deployment and continuous intelligent evolution.

[0028] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data analysis system for monitoring polyacrylamide production process based on reinforcement learning, characterized in that, The system includes: The data acquisition module is configured to receive process sensing data streams at a first frequency and quality detection data streams at a second frequency. The strategy generation module is configured to generate real-time monitoring strategies based on process sensing data streams using a first neural network, and to calculate instant agent rewards based on process sensing data streams using a pre-set agent model. The asynchronous experience playback module is configured to build an asynchronous experience playback buffer, mapping the quality inspection data stream back to the corresponding historical process sensor data fragments; The model update module is configured to calculate the delay correction reward using the mapped quality inspection data stream, and update the parameters of the first neural network based on the weighted combination of the immediate agent reward and the delay correction reward; wherein, the second frequency is lower than the first frequency; the process sensing data stream represents the real-time physical state of the production equipment, and the quality inspection data stream represents the final physicochemical indicators of the production products.

2. The reinforcement learning-based polyacrylamide production process monitoring data analysis system according to claim 1, characterized in that, The strategy generation module includes: a feature extraction unit, configured to extract local spatial features of the process sensing data stream using a one-dimensional convolutional neural network and extract time-dependent features of the process sensing data stream using a long short-term memory network to generate a real-time state vector; and a proxy reward calculation unit, configured to input the real-time state vector into a preset proxy model and output a virtual evaluation value representing the current production state conversion rate as an immediate proxy reward; wherein, the preset proxy model is a calculation model pre-constructed based on the physicochemical mechanism equations of the production process.

3. The reinforcement learning-based polyacrylamide production process monitoring data analysis system according to claim 1, characterized in that, The asynchronous experience playback module includes: a timestamp alignment unit, configured to record the generation timestamp of each frame of data and store it in the asynchronous experience playback buffer when the process sensing data stream is received; a backtracking mapping unit, configured to retrieve historical process sensing data segments that match the generation timestamp in the asynchronous experience playback buffer according to a preset lag time parameter when the quality detection data stream is received; and a causal association unit, configured to use the quality truth value in the quality detection data stream as a label and bind it to the retrieved historical process sensing data segments to form a complete experience sample containing delay feedback.

4. The reinforcement learning-based polyacrylamide production process monitoring data analysis system according to claim 3, characterized in that, The model update module includes: a deviation calculation unit, configured to calculate the numerical deviation between the true quality value in the quality inspection data stream and the virtual evaluation value output by the surrogate model at historical moments; and a backpropagation unit, configured to use the numerical deviation as a delay correction reward. - The algorithm backpropagates the delayed correction reward to the historical decision steps of the first neural network; the parameter correction unit is configured to increase the weight of the delayed correction reward in the weighted combination when the numerical deviation exceeds a preset safety threshold, so as to dominate the gradient update direction of the first neural network.

5. The reinforcement learning-based polyacrylamide production process monitoring data analysis system according to claim 1, characterized in that, The system also includes: a counterfactual reasoning module, configured to generate virtual adverse working condition data using a generative adversarial network based on the current process sensor data stream; and a risk warning module, configured to input the virtual adverse working condition data into a first neural network, and trigger a warning signal if the predicted state output by the first neural network exceeds the safe range; wherein, the virtual adverse working condition data is synthetic data artificially introduced with parameter drift or sudden noise.

6. The reinforcement learning-based polyacrylamide production process monitoring data analysis system according to claim 1, characterized in that, The system also includes a slow timescale calibrator, configured to utilize The encoder processes the quality inspection data stream and extracts long-term quality trend features. The strategy generation module is also configured to concatenate long-cycle quality trend features with process sensing data streams as input to the first neural network.

7. The reinforcement learning-based polyacrylamide production process monitoring data analysis system according to any one of claims 1-6, characterized in that, The production process involves the polymerization reaction of polyacrylamide. The process sensing data stream includes: temperature, pressure, stirring motor current, and initiator flow rate of the reactor; the quality detection data stream includes: molecular weight distribution data and degree of hydrolysis data of the polyacrylamide product; the hysteresis time parameter is 4 to 6 hours.

8. The reinforcement learning-based polyacrylamide production process monitoring data analysis system according to any one of claims 1-6, characterized in that, The system includes: edge computing nodes, deployed on the production site, configured to run data acquisition and strategy generation modules, and perform millisecond-level real-time inference; and cloud servers, configured to run asynchronous experience playback and model update modules, which, in response to receiving new quality inspection data streams, perform event-triggered model training and parameter distribution, or perform periodic training synchronized with the quality inspection frequency.

Images