A method, system, and device for training a decision model based on a disturbance-resistant environment model.

Abstract

This invention provides a decision model training method based on a disturbance-resistant environment model, comprising: pre-training a disturbance-resistant environment model; constructing an initial decision model for outputting control actions; using the trained disturbance-resistant environment model as a virtual interactive environment, iteratively training the initial decision model in the virtual interactive environment through offline reinforcement learning, and updating the model parameters of the initial decision model to obtain an updated decision model; and obtaining the trained decision model when the iterative training reaches a preset training termination condition. This invention achieves state data prediction for the next moment under disturbance resistance by training a disturbance-resistant environment model, and utilizes the disturbance-resistant environment model as a virtual environment for offline reinforcement learning iterative training of the initial decision model. This avoids the training risks of real systems while improving the adaptability of the decision model, effectively solving the technical problems of policy models being susceptible to disturbances, having low training reliability, and poor decision-making performance in wastewater treatment control.

Description

Technical Field

[0001] This invention relates to the field of wastewater treatment technology, and in particular to a method, system, device, and storage medium for training a decision model based on an anti-disturbance environmental model. Background Technology

[0002] In urban wastewater treatment processes, the biological aeration stage is a core unit for ensuring effluent quality meets standards and controlling operational energy consumption. The aeration process is characterized by strong nonlinearity, large lag, and multiple disturbances. Factors such as influent water concentration, influent flow rate, sludge concentration, and water temperature all have a significant impact on dissolved oxygen demand, while disturbances such as sensor measurement noise and instantaneous load impacts further exacerbate the uncertainty of system operation.

[0003] Traditional wastewater treatment aeration control often employs fixed-parameter proportional-integral-differential (PID) control or manual, experience-based adjustment, which struggles to adapt to dynamically changing operating conditions. This can lead to problems such as excessive fluctuations in dissolved oxygen concentration, effluent quality exceeding standards, or excessive energy consumption due to over-aeration. In recent years, reinforcement learning-based intelligent decision-making models have been increasingly applied to aeration control, enabling them to optimize control actions through interactive learning with the environment.

[0004] However, existing technologies have significant drawbacks: Directly training strategy models in real wastewater treatment environments carries risks of unstable control and water quality exceeding standards, which can severely impact the safe operation of the entire wastewater treatment process. Conventional environmental models simply fit system state transition relationships based on historical data, without explicitly modeling and mitigating disturbances. This makes it impossible to recreate complex, real-world disturbance environments in virtual training, resulting in weak generalization ability of the trained decision models and insufficient adaptability of the output control actions when encountering sudden changes in water quality and flow. Furthermore, existing reinforcement learning training processes lack multi-objective constraint balance, making it difficult to simultaneously achieve effluent quality compliance, reduced aeration energy consumption, and stable control actions, easily leading to frequent action jumps and increased equipment wear.

[0005] Therefore, how to construct a virtual environment model with disturbance resistance capabilities, train the decision-making model safely, stably, and efficiently under offline conditions, and enable the decision-making model to output accurate, stable, and low-energy control actions under complex disturbances is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This invention trains an anti-disturbance environment model using acquired historical operational data to achieve anti-disturbance state prediction. The anti-disturbance environment model is used as a virtual environment to perform offline reinforcement learning iterative training on the initial decision model, which improves the adaptability of the decision model while avoiding the training risks of real systems. By continuously updating the decision model parameters until the preset termination conditions are met, the trained decision model is output, effectively solving the technical problems of susceptibility to disturbances, low training reliability, and poor decision-making effect in wastewater treatment control.

[0007] The purpose of this invention is to provide a method for training a decision model based on a disturbance-resistant environment model; The technical solution provided by this invention is as follows: A method for training a decision model based on a disturbance-resistant environment model, comprising: Acquire historical operational data of the wastewater treatment process, including status data and corresponding control actions; Based on the historical operating data, an initial disturbance immunity environment model is trained to obtain a trained disturbance immunity environment model; wherein, the trained disturbance immunity environment model is used to obtain the predicted state data for the next moment of disturbance immunity based on the current state data and the current control action. Construct an initial decision model for outputting the control action at the current moment based on the state data at the current moment and the control action at the previous moment; Using the trained disturbance-resistant environment model as a virtual interactive environment, the initial decision model is iteratively trained in the virtual interactive environment through offline reinforcement learning based on the historical operating data, and the model parameters of the initial decision model are updated to obtain the updated decision model; when the iterative training reaches the preset training termination condition, the trained decision model is obtained.

[0008] Preferably, the initial disturbance-resistant environment model includes: a forward diffusion unit, a cross-attention network, a temporal convolutional network, and a prediction unit; The forward diffusion unit is used to add random perturbations to the original state data input into the initial disturbance-resistant environment model during the training phase of the initial disturbance-resistant environment model to obtain state features with perturbations; the random perturbations are used to simulate at least one of water quality fluctuations, flow rate changes, and sensor errors. The cross-attention network includes an embedding layer, a cross-attention encoder, and a cross-attention decoder; The embedding layer is used to generate action embedding features and state embedding features based on the input control action and the original state data; The cross-attention encoder is used to perform cross-attention encoding with the state embedding feature as the query and the joint embedding feature formed by the state embedding feature and the action embedding feature as the key and value, to obtain the encoded feature. The cross-attention decoder is used to perform cross-attention decoding with the encoded features as the query and the perturbated state embedding features as the key and value, to obtain the decoded features. A temporal convolutional network is used to capture local temporal features in the encoded features and the perturbed state features to obtain local features; The prediction unit is used to obtain the predicted state data for the next time step based on the local features and the decoded features.

[0009] Preferably, the step of training an initial disturbance-resistant environment model based on the historical operational data to obtain a trained disturbance-resistant environment model includes: Extract the current state data, the control action corresponding to the current state data, and the actual state data for the next moment from the historical operation data; The forward diffusion unit adds a random perturbation to the current state data to obtain perturbed state data; The current state data, the current control action, and the perturbed state data are input into the embedding unit to obtain the current state embedding feature, the current action embedding feature, and the perturbed current state embedding feature. The current state embedding feature, the current action embedding feature, and the perturbated current state embedding feature are input into the cross-attention network to obtain the encoded feature and the decoded feature; The encoded features and the perturbed state features are input into the temporal convolutional network to obtain the local features; The local features and the decoded features are input into the prediction unit to obtain the prediction state data for the next time step; With the goal of minimizing the error between the predicted state data and the actual state data at the next time step, the parameters of the initial disturbance-resistant environment model are iteratively updated using the gradient descent algorithm until the error converges to a preset threshold, thus obtaining a trained disturbance-resistant environment model.

[0010] Preferably, constructing the initial decision model includes: constructing an initial decision model comprising a policy network and a value network; the policy network is used to output the control action at the current moment based on the input state data at the current moment and the control action at the previous moment; the value network is used to obtain a predicted future return based on the state data at the current moment. The initial decision model is obtained by initializing the parameters of the policy network and the value network.

[0011] Preferably, the step of using the trained disturbance-resistant environment model as a virtual interactive environment, and iteratively training the initial decision model in the virtual interactive environment based on the historical operating data through offline reinforcement learning to update the model parameters of the initial decision model, includes: Extract an initial state data and the corresponding control action from the previous moment from the historical operation data, and use them as the current state data and the control action from the previous moment. Using the policy network, the disturbance-resistant environment model, and a preset reward function, multi-step virtual interaction is performed on the current state data and the control action from the previous moment to obtain a virtual interaction trajectory; wherein, performing multi-step virtual interaction to obtain the virtual interaction trajectory includes: The current state data and the control action from the previous moment are input into the policy network to obtain the current control action; the current state data and the current control action are input into the disturbance-resistant environment model to obtain the predicted state data for the next moment; the instant reward for the current control action is calculated based on the preset reward function; wherein, the current control action, the predicted state data for the next moment, and the instant reward constitute a time step in the virtual interaction trajectory; Continue to perform virtual interaction on the control action at the current moment and the predicted state data at the next moment output by the policy network and the anti-disturbance environment model until the preset number of virtual interaction steps is reached to obtain the virtual interaction trajectory; The actual future reward for each time step is calculated based on the immediate reward at each time step in the virtual interaction trajectory. The predicted future returns for each time step are calculated based on the value network, and the advantage function is determined based on the actual future returns and the predicted future returns. The loss function of the policy network is constructed based on the advantage function, and the loss function of the value network is constructed based on the deviation between the predicted future return and the actual future return. Based on the loss function of the policy network and the loss function of the value network, the parameters of the policy network and the value network are updated through backpropagation algorithm to complete one iteration of training.

[0012] Preferably, the preset reward function is a multi-objective reward function, which includes at least: a water quality compliance reward, an energy consumption penalty, and a control action smoothness penalty. The water quality compliance reward items are positively correlated with the degree of compliance of the effluent water quality indicators; The energy consumption penalty term is positively correlated with the energy consumption value of the control action at the current moment; The control action smoothing penalty term is positively correlated with the change amplitude of the control action at adjacent time points; The immediate reward is obtained by summing the water quality compliance reward, the energy consumption penalty, and the control action smoothness penalty. The reward function aims to maximize the immediate reward so that the decision model outputs a control action that satisfies the water quality compliance constraint, minimizes the energy consumption of the control action, and ensures smooth control action changes.

[0013] Preferably, the preset training termination condition includes: the number of iterations reaches a preset threshold.

[0014] The second objective of this invention is to provide a decision model training system based on a disturbance-resistant environment model; The technical solution provided by this invention is as follows: A decision model training system based on a disturbance-resistant environment model includes: The acquisition module is used to acquire historical operating data of the wastewater treatment process, including status data and corresponding control actions. The environment model pre-training module is used to train an initial disturbance-resistant environment model based on the historical operating data to obtain a trained disturbance-resistant environment model; wherein, the trained disturbance-resistant environment model is used to obtain the predicted state data for the next moment of disturbance based on the current state data and the current control action; An initial strategy model construction module is used to construct an initial decision model for outputting the control action at the current moment based on the current state data and the control action at the previous moment; The iterative training module is used to use the trained disturbance-resistant environment model as a virtual interactive environment, and based on the historical running data, to iteratively train the initial decision model in the virtual interactive environment through offline reinforcement learning, and update the model parameters of the initial decision model to obtain an updated decision model; when the iterative training reaches the preset training termination condition, the trained decision model is obtained.

[0015] The third objective of this invention is to provide a computer device; The technical solution provided by this invention is as follows: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the decision model training methods described above.

[0016] A fourth objective of this invention is to provide a computer-readable storage medium; The technical solution provided by this invention is as follows: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the decision model training methods described above.

[0017] This invention provides a decision model training method based on a disturbance-resistant environment model, comprising: pre-training a disturbance-resistant environment model; constructing an initial decision model for outputting control actions; using the trained disturbance-resistant environment model as a virtual interactive environment, iteratively training the initial decision model in the virtual interactive environment through offline reinforcement learning, and updating the model parameters of the initial decision model to obtain an updated decision model; and obtaining the trained decision model when the iterative training reaches a preset training termination condition. This invention achieves state data prediction for the next moment under disturbance resistance by training a disturbance-resistant environment model, and utilizes the disturbance-resistant environment model as a virtual environment for offline reinforcement learning iterative training of the initial decision model. This avoids the training risks of real systems while improving the adaptability of the decision model, effectively solving the technical problems of policy models being susceptible to disturbances, having low training reliability, and poor decision-making performance in wastewater treatment control.

[0018] The present invention also provides a decision model training system based on an anti-disturbance environment model. Since this system and the decision model training method solve the same technical problem and belong to the same technical concept, they should have the same beneficial effects, and will not be described in detail here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a decision model training method based on an anti-disturbance environment model in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a decision model training system based on an anti-disturbance environment model in an embodiment of the present invention; Figure 3 This is an internal structural diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] like Figure 1 As shown, this embodiment of the invention provides a method for training a decision model based on a disturbance-resistant environment model, including: S1. Obtain historical operational data of the wastewater treatment process, including status data and corresponding control actions; In this embodiment, the status data includes, but is not limited to, parameters that characterize the system's operating status, such as influent flow rate, influent chemical oxygen demand (COD), ammonia nitrogen concentration, total nitrogen concentration, dissolved oxygen concentration in the aeration tank, mixed liquor sludge concentration, water temperature, and pH value. Control actions include executable control commands such as speed setting commands for the aeration blower, frequency control commands for the frequency converter, or opening commands for the air supply valve. The status data in the historical operating data is time-series status data, and the control actions are the control actions that correspond to the status data in time.

[0023] In practical applications, step S1 also includes preprocessing the acquired historical running data, including outlier removal, missing value interpolation and filling, and normalization, in order to improve the stability of subsequent model training.

[0024] S2. Based on historical operating data, train an initial disturbance immunity environment model to obtain a trained disturbance immunity environment model; wherein, the trained disturbance immunity environment model is used to obtain the predicted state data for the next moment of disturbance immunity based on the current state data and the current control action. In this embodiment, a neural network model capable of simulating the state evolution of a wastewater treatment system, namely a disturbance-resistant environment model, is constructed and trained. Supervised training of the initial disturbance-resistant environment model is performed based on historical operating data, enabling the model to learn the dynamic changes of the system under different operating states and control actions. Simultaneously, the environment model acquires the ability to suppress the effects of disturbances, thus outputting stable and accurate predicted state data for the next moment even when the input exhibits fluctuations or disturbances. This gives the environment model the ability to resist disturbances and provide stable predictions, accurately simulating system state changes under complex operating conditions. It provides a reliable and robust virtual interactive environment for subsequent decision-making models, improving the stability of subsequent training.

[0025] As one implementation, the initial disturbance-resistant environment model employs a differentiable neural network structure, such as a denoised diffusion probability model (DDPM), a convolutional network (CNN), a recurrent neural network (RNN), or a variant thereof.

[0026] Preferably, the initial disturbance-resistant environment model is a denoised diffusion probability model; During the training phase of the initial robust environment model, random perturbations are added to the original state data input to the initial robust environment model through a denoising diffusion probability model to obtain perturbed state data, so that the initial robust environment model can learn the mapping relationship of recovering the original state data from the perturbed state data during training. During the inference phase of the trained disturbance-resistant environment model, the denoising diffusion probability model outputs the predicted state data for the next moment based on the input current state data and the current control action.

[0027] Preferably, the initial disturbance-resistant environment model includes: a forward diffusion unit, a cross-attention network, a temporal convolutional network, and a prediction unit; The forward diffusion unit is used to add random perturbations to the original state data input into the initial disturbance-resistant environment model during the training phase of the initial disturbance-resistant environment model, so as to obtain perturbed state data; the random perturbations are used to simulate at least one of water quality fluctuations, flow rate changes and sensor errors; Cross-attention networks consist of an embedding layer, a cross-attention encoder, and a cross-attention decoder; The embedding layer is used to generate action embedding features and state embedding features based on the input control actions and raw state data. The cross-attention encoder is used to perform cross-attention encoding with state embedding features as queries and joint embedding features consisting of state embedding features and action embedding features as keys and values ​​to obtain encoded features. The cross-attention decoder is used to perform cross-attention decoding with encoded features as queries and perturbated state embedding features as keys and values ​​to obtain decoded features. A temporal convolutional network is used to capture local temporal features in the encoded features and the perturbed state features to obtain local features; The prediction unit is used to obtain the predicted state data for the next time step based on local features and decoded features.

[0028] In this embodiment, during the environmental model training phase, a forward diffusion unit simulates various disturbances present in real-world scenarios, making the model training data more closely resemble actual working conditions. The forward diffusion unit receives the original state data during the training phase and gradually adds random disturbances according to a predefined noise schedule (such as linear or cosine schedule), generating disturbed state data. The added random disturbances simulate typical sources of uncertainty in wastewater treatment processes, such as water quality fluctuations, changes in influent flow rate, and sensor measurement noise, requiring the model to learn state transition patterns under disturbed input conditions.

[0029] In the de-perturbation modeling stage of this invention, the cross-attention network and the temporal convolutional network do not employ a conventional homogeneous input structure. Instead, they achieve collaborative modeling of perturbated state data and the global context through differentiated information sources and fusion methods. Specifically, the cross-attention encoder uses the pure state embedding features as the query and the joint embedding features (concatenated from the state embedding features and action embedding features) as the key and value for cross-attention encoding. The physical significance of this design is that the encoder, through an attention mechanism, retrieves the most relevant historical state in the state space and simultaneously fuses action information, thereby extracting the global background features of the system under perturbation conditions, i.e., the encoded features. The cross-attention decoder uses the encoded features output by the encoder as the query and the perturbated state embedding features as the key and value for cross-attention decoding. The core function of the decoder is to retrieve relevant information from the pure global dynamic background provided by the encoder, using the current perturbated state as an index, thereby achieving conditional feature recovery and outputting decoded features. This design effectively avoids injecting perturbation information into all components of the attention simultaneously, preventing feature contamination.

[0030] Temporal Convolutional Networks (TCNs) work in parallel or cascaded with Cross-Attention Networks to capture local temporal features in encoded features and perturbed state features. Through causal convolution and dilated convolution structures, TCNs can keenly extract short-term dynamic information such as high-frequency perturbations and local mutations, making up for the shortcomings of Cross-Attention Networks in modeling local details.

[0031] Through this structural design of "dual-path modeling (cross-attention network) and dual-stage fusion (cross-attention network + temporal convolutional network)," the environment model is simultaneously subject to long-term temporal consistency constraints and short-term dynamic continuity constraints. This avoids the loss of local details caused by relying solely on the cross-attention network and overcomes the problem that relying solely on convolutional structures makes it difficult to model long-distance dependencies.

[0032] The prediction unit fuses the global dependency features output by the cross-attention network and the local dynamic features output by the temporal convolutional network again, and maps them back to the dimension of the original state space through one or more fully connected layers (such as multilayer perceptron MLP), and finally outputs the predicted state data for the next time step. This enables the environment model to output stable and reliable state prediction results even when there are perturbations in the input, and significantly improves the environment model's ability to resist perturbations and its prediction accuracy.

[0033] In practical applications, disturbance-resistant environment models include: Data Segmentation Unit: First, the input multivariate time series state (length L) and control actions are segmented into multiple patches of length P. This transforms long sequences into structured blocks, reduces sequence length, and improves the model's ability to express local dynamic patterns. Patch-level modeling effectively suppresses the cumulative diffusion of prediction errors over time, improves the numerical stability of the environmental model in multi-step and long-term rolling prediction processes, and provides more reliable environmental feedback for subsequent intelligent decision-making based on the environmental model's prediction results.

[0034] Forward diffusion unit: This module receives state embedding features. According to a predefined noise schedule (such as linear or cosine schedule), Gaussian noise is gradually added to it in S steps to obtain perturbed state data. This process simulates various perturbations that exist in real systems.

[0035] Embedding layer: Then, a linear embedding layer maps each patch to a high-dimensional feature space, resulting in state embedding features. Action embedding features .

[0036] Transformer encoder: Transforms the segmented action sequence Also perform embedding to obtain action embedding features. The Transformer encoder uses a self-attention mechanism to compute actions. The long-range dependency between the output and historical states. It represents the global dynamic background of the system in a high-dimensional space.

[0037] Transformer Decoder: The decoder's query comes from the output of the Transformer encoder. The Key and Value are derived from the perturbed state embedding features output by the forward diffusion unit. This design allows the environmental model to retrieve relevant information from clean, unperturbed historical state data using the current initial state as an index during inference, thereby achieving precise conditional deperturbing.

[0038] Temporal Convolutional Networks (TCNs): TCNs use... and The element-wise summation is used as the input. Through causal convolution and dilated convolution, TCN can effectively capture local, high-frequency temporal features, such as rapid state fluctuations caused by a single perturbation. The output of TCN... It addresses the shortcomings of Transformer in modeling local details.

[0039] Prediction Unit: Finally, the output of the Transformer decoder is... With TCN output The data is then fused (e.g., added or spliced), and then mapped back to the dimensions of the original state space through a multilayer perceptron (MLP), ultimately outputting the predicted state for the next time step.

[0040] Preferably, based on historical operational data, an initial disturbance-resistant environment model is trained to obtain a trained disturbance-resistant environment model, including: Extract the current state data, the corresponding control action at the current moment, and the actual state data for the next moment from historical operational data; By adding random perturbations to the current state data using a forward diffusion unit, we obtain perturbed state data. The current state data, the current control action, and the perturbed state data are input into the embedding unit to obtain the current state embedding feature, the current action embedding feature, and the perturbed current state embedding feature. The current state embedding feature, the current action embedding feature, and the perturbed current state embedding feature are input into the cross-attention network to obtain the encoded features and the decoded features. The encoded features and the perturbated state features are input into a temporal convolutional network to obtain local features; The local features and decoded features are input into the prediction unit to obtain the prediction state data for the next time step. The training objective is to minimize the error between the predicted state data and the actual state data at the next time step. The parameters of the initial robust environment model are iteratively updated using the gradient descent algorithm until the error converges to a preset threshold, thus obtaining the trained robust environment model.

[0041] In this embodiment, the environmental model is trained under supervision using undisturbed sample data. During the training process, random perturbations are introduced through forward diffusion units, enabling the model to learn the mapping relationship between the perturbations and the real state. The model parameters are continuously optimized with the goal of minimizing the prediction error. Through forward diffusion, the environmental model can gradually acquire the ability to suppress interference and make accurate predictions, ensuring that the environmental model after training can stably simulate the dynamic changes of the sewage treatment system, and providing a high-fidelity virtual interactive environment for subsequent decision model training.

[0042] Specifically, firstly, the current state data is processed through a forward diffusion unit. Add simulated perturbations to obtain perturbed current state embedding features. Then, and current action embedding features Feature extraction is performed using an input cross-attention network and a temporal convolutional network. The extracted decoded features and local features are then input into the prediction unit to obtain the prediction state. Finally, with the goal of minimizing the prediction error, the gradient descent algorithm is used to iteratively update all parameters of the environment model until the environment model converges, resulting in a well-trained, robust environment model.

[0043] As one implementation method, samples are extracted in batches from the offline dataset constructed in step S1, and each sample contains a triple: the current state data. Current control actions And the actual state data of the next moment. .Will( , The data is input into the current initial disturbance-resistant environment model, and after forward propagation, the model outputs the predicted state data for the next time step. Define a loss function to quantify the prediction error, such as mean squared error loss. Then, use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters, and use a gradient descent optimizer to update the model parameters to gradually reduce the prediction error. Repeat the above process, iterating through the offline dataset until the loss function value converges to a preset threshold or no longer decreases significantly. At this point, the environmental model has learned the complex mapping relationship from the current state data and the current control action to the state data of the next time step.

[0044] Loss function for the disturbance-resistant environment model:

[0045] in, This is the training loss function for the environment model, used to measure the error between the model's predicted state data and the actual state. The expectation operator represents averaging over all samples in the offline dataset; The L2 norm squared is used to measure the magnitude of the prediction error. This is the state transition function for the disturbance-resistant environment model, used to predict the state data at the next moment based on the current state data and the current control action; The state data for the predicted next moment; This represents the actual state data for the next moment.

[0046] S3. Construct an initial decision model for outputting the control action at the current moment based on the current state data and the control action at the previous moment; In this embodiment, the initial decision model includes, but is not limited to, the policy network-value network model, the deep policy gradient model, the deep Q-decision model, the proximal policy optimization model, and the deterministic policy gradient model.

[0047] Preferably, constructing the initial decision model includes: constructing an initial decision model comprising a policy network and a value network; the policy network is used to output the control action at the current moment based on the input current state data and the control action at the previous moment; the value network is used to obtain the predicted future return based on the current state data. The policy network and value network are initialized with parameters to obtain the initial decision model.

[0048] In this embodiment, a decision model based on the policy network-value network (Actor-Critic) architecture is constructed, and the parameters of the policy network and value network are randomly initialized to obtain an initial decision model.

[0049] The policy network (Actor) is responsible for making decisions, and its input is the current state data. and the corresponding control action of the previous moment It should be noted that the current state data... It executes the corresponding control action from the previous moment in the system state. The output is the control action at the current moment. (For continuous control tasks, parameters of the output action probability distribution are typically used, such as the mean and standard deviation of a Gaussian distribution, and then the control action is obtained through random sampling.) The goal of the policy network is to learn an optimal mapping function that allows taking its output action in the current state to obtain the highest possible future reward. In the early stages of training, the output of the policy network is random; as training progresses, its parameters are continuously optimized, and the output control action gradually tends towards the optimal control action. The input corresponds to the control action from the previous time step. It is used to determine the current parameters of sewage treatment equipment such as blowers at the current moment, and to generate the current control action based on the current parameters.

[0050] Value Network (Critic): This network performs evaluations, and its input is the current state data. However, the output is a scalar value, which is the predicted future return. This value represents the return from the current state under the current policy. Initially, the system estimates the future rewards. The value network does not directly determine actions, but rather provides a robust baseline for updating the policy network. By comparing the actual future rewards with the value network's predicted future rewards, it can determine whether an action is "good" or "bad," thereby guiding the policy network's update direction, effectively reducing variance during training, and improving learning stability.

[0051] The Actor-Critic architecture offers the following advantages: high training stability, as the baseline provided by the value network significantly reduces the variance of policy gradient estimation, making the training process smoother and more stable, and avoiding training collapse caused by sparse reward signals or excessive noise; high sample efficiency, as the policy network can learn from the advantage function and update using time difference errors, which has higher sample utilization efficiency than policy gradient methods that rely solely on Monte Carlo rewards; and suitability for continuous control, as this architecture is well-suited for continuous motion space problems such as the speed regulation of aeration blowers in wastewater treatment, and can output smooth and precise control quantities rather than discrete gear selections.

[0052] S4. Using the trained anti-disturbance environment model as a virtual interactive environment, based on historical operating data, the initial decision model is iteratively trained in the virtual interactive environment through offline reinforcement learning, and the model parameters of the initial decision model are updated to obtain the updated decision model; when the iterative training reaches the preset training termination condition, the trained decision model is obtained.

[0053] It should be noted that reinforcement learning can include Proximal Policy Optimization (PPO), Value Iteration, Q-learning, etc. Preferably, the decision model uses the PPO algorithm.

[0054] Preferably, the trained disturbance-resistant environment model is used as a virtual interactive environment. Based on historical operating data, the initial decision model is iteratively trained in the virtual interactive environment using offline reinforcement learning to update the model parameters of the initial decision model, including: Extract an initial state data and the corresponding control action from the previous moment from the historical operation data, and use them as the current state data and the control action from the previous moment. Using the policy network, the disturbance-resistant environment model, and a preset reward function, multi-step virtual interaction is performed on the current state data and the control action from the previous moment to obtain a virtual interaction trajectory; wherein, performing multi-step virtual interaction to obtain the virtual interaction trajectory includes: The current state data and the control action from the previous moment are input into the policy network to obtain the current control action; the current state data and the current control action are input into the disturbance-resistant environment model to obtain the predicted state data for the next moment; the instant reward for the current control action is calculated based on the preset reward function; wherein, the current control action, the predicted state data for the next moment, and the instant reward constitute a time step in the virtual interaction trajectory; Continue to perform virtual interaction on the control action at the current moment and the predicted state data at the next moment output by the policy network and the anti-disturbance environment model until the preset number of virtual interaction steps is reached to obtain the virtual interaction trajectory; The actual future reward for each time step is calculated based on the immediate reward at each time step in the virtual interaction trajectory. The predicted future returns for each time step are calculated based on the value network, and the advantage function is determined based on the actual future returns and the predicted future returns. The loss function of the policy network is constructed based on the advantage function, and the loss function of the value network is constructed based on the deviation between the predicted future return and the actual future return. Based on the loss function of the policy network and the loss function of the value network, the parameters of the policy network and the value network are updated through backpropagation algorithm to complete one iteration of training.

[0055] In this embodiment, the specific steps for iteratively training the Actor-Critic decision model using the trained robust environment model are as follows: (1) State and action initialization: Randomly select a trajectory from the offline historical dataset, and use the first historical state data and the corresponding historical control action in the trajectory as the state data at the current moment. and control actions (As a starting point for training).

[0056] (2) Policy network decision-making: using the current state data and control actions The input is fed into the policy network (Actor). The policy network then processes the data based on the current state. and control actions Calculate and output the distribution of control actions at the current moment, and randomly sample the actions. This action represents the optimal control command that the current strategy model considers to be in the current state.

[0057] (3) Environmental model simulation: using the current state data The control action output by the policy network at the current moment Both inputs are fed into the pre-trained robust environment model. The robust environment model performs one forward propagation and outputs the predicted state data for the next time step. This step simulates the result of executing the control action output by the policy network in a virtual world. It should be noted that during the process of the trained disturbance-resistant environment model serving as a virtual interactive environment, i.e., the reasoning stage of predicting the state data at the next moment, the forward diffusion unit does not participate in the reasoning. It only outputs the predicted state data at the next moment based on the input state data at the current moment and the control action at the current moment through the cross-attention network, the temporal convolutional network, and the prediction unit.

[0058] (4) Reward calculation: The predicted state at the next moment. and the actions performed The input is fed into a preset reward function to calculate an immediate scalar reward, which is a prediction of future returns (value estimation). The value of the reward function directly reflects the quality of this decision.

[0059] To facilitate subsequent gradient optimization, the reward maximization problem is transformed into a loss minimization problem. The reward function includes: Single-step loss function:

[0060] Multi-step cumulative loss function:

[0061] in, The loss function; This is the discount factor, with a value range of (0,1). This represents the predicted state at the current moment and the immediate reward value under the action at the current moment.

[0062] (5) Calculation of advantage function and target return: Repeat steps (1)-(4) above to generate a line of length. Virtual interaction trajectory Based on the value estimate of each state in the value network computation trajectory, the predicted future returns are obtained. Then, based on the temporal difference method, the dominance function for each time step is calculated. (Measures of performance relative to average) and actual future returns Where the actual future return is the preset target return. Advantage function ,in, For state Predicting future returns, This is the discount factor.

[0063] (6) Constructing the loss function includes: The core of constructing the policy loss function is to maximize the expected value of the advantage function. The expression for the loss function is as follows: ,in, For expectation operator, This is the logarithm of the probability distribution output by the policy network. The physical meaning of the policy loss function is to increase the probability of "good" actions (positive advantage) and decrease the probability of "bad" actions (negative advantage).

[0064] Value network loss function: Construct a loss function, typically the mean squared error between the value network's predicted value and the actual target return. ,in, This represents the actual future return (which can be calculated using multi-step cumulative reward values ​​or time difference methods). Its physical meaning is to make the predictions of the value network increasingly accurate.

[0065] Backpropagation and parameter update: The total loss is obtained by weighted summation of the policy network loss and the value network loss. The expression for the total loss function is as follows: ,in, These are the weighting coefficients.

[0066] Then, the gradient of the total loss with respect to the parameters of the policy network and the value network is calculated using the backpropagation algorithm, and the parameters of these two networks are updated using an optimizer. The key point is that the parameters of the environment model are frozen during this process and do not participate in the update.

[0067] Backpropagation path based on environment model: When the perturbation-resistant environment model is a differentiable model, the gradient can be backpropagated using the chain rule:

[0068] in, This represents the gradient of the total loss function with respect to the policy model parameters. The gradient of the total loss function with respect to the immediate reward value; The gradient of the environment model with respect to the control action; To reward the gradient of the state data; This represents the gradient of the policy network.

[0069] The gradient is propagated back layer by layer through "reward → state → action → policy parameters".

[0070] Preferably, the preset reward function is a multi-objective reward function, which includes at least: a water quality compliance reward, an energy consumption penalty, and a control action smoothness penalty. The incentives for meeting water quality standards are positively correlated with the degree to which the effluent water quality indicators meet the standards. The energy consumption penalty is positively correlated with the energy consumption value of the control action at the current moment; The smoothing penalty term for the controlled action is positively correlated with the magnitude of change in the controlled action at adjacent time points; The immediate reward is obtained by summing the water quality compliance reward, energy consumption penalty, and control action smoothness penalty. The reward function aims to maximize the immediate reward so that the decision model outputs control actions that meet the water quality compliance constraint, minimize control action energy consumption, and ensure smooth control action changes.

[0071] In this embodiment, by combining multiple objectives such as water quality, energy consumption, and stability, the decision-making model can be guided to take into account the effluent water quality, operating energy consumption, and equipment operating stability during the optimization process. With the goal of maximizing immediate rewards, the strategy model can automatically learn safe, economical, and stable control strategies, avoiding problems such as water quality exceeding standards, excessive energy consumption, or drastic fluctuations in operation caused by single-objective optimization, thereby improving the overall practicality of the control strategy.

[0072] Preferably, the preset training termination condition includes: the number of iterations reaches a preset threshold.

[0073] It should be noted that the number of iterations in this context refers to the number of "one training step," which specifically means the number of iterations the decision model performs to update parameters based on a batch of data during the offline reinforcement learning optimization process.

[0074] Specifically, each training iteration includes: sampling an initial state and its corresponding historical action from historical data; outputting an action from the policy network based on the current decision model; inputting this action and the current state into a pre-trained robust environment model to obtain the predicted next state and calculate the immediate reward; repeating the above process to generate a virtual interaction trajectory of fixed length; calculating the policy network loss and value network loss based on this trajectory, and performing one backpropagation and parameter update. The complete process from data sampling, trajectory generation to parameter update is counted as one training step (or one iteration). The system maintains a training step counter, which increments by one each time the above process is completed. When the accumulated value of the counter reaches a pre-set threshold, training automatically terminates, and the latest decision model is saved as the final trained decision model.

[0075] Compared to existing technologies, this invention constructs a virtual interactive environment using an anti-disturbance environment model to achieve offline optimization training of the decision-making model, avoiding the risks of training in real systems and improving the model's anti-disturbance capability and control reliability. Through a collaborative architecture of policy networks and value networks and reasonable parameter initialization, it lays a stable foundation for reinforcement learning optimization and accelerates convergence. Through virtual environment trajectory generation, advantage function calculation, and backpropagation updates, it achieves efficient offline iteration of the decision-making model. By inputting the current state and the current control action output by the policy network into the environment model to obtain the predicted state, and calculating the immediate reward based on the reward function, it constructs the advantage function and dual-network loss for backpropagation, forming a closed loop of "decision-deduction-evaluation-optimization". A multi-objective reward function is designed, including water quality compliance rewards, energy consumption penalties, and action smoothing penalties, unifying conflicting industrial control objectives into optimization indicators and guiding the model to output the optimal aeration control strategy that satisfies water quality constraints, minimizes energy consumption, and ensures stable control.

[0076] like Figure 2 As shown, this embodiment of the invention provides a decision model training system based on a disturbance-resistant environment model, comprising: A decision model training system based on a disturbance-resistant environment model includes: The acquisition module is used to acquire historical operating data of the wastewater treatment process. The historical operating data includes status data and corresponding control actions. The environment model pre-training module is used to train an initial disturbance-resistant environment model based on historical operating data to obtain a trained disturbance-resistant environment model. The trained disturbance-resistant environment model is used to obtain the predicted state data for the next moment of disturbance based on the current state data and the current control action. The initial strategy model building module is used to build an initial decision model that outputs the control action at the current moment based on the current state data and the control action at the previous moment; The iterative training module is used to use the trained disturbance-resistant environment model as a virtual interactive environment. Based on historical operating data, it iteratively trains the initial decision model in the virtual interactive environment through offline reinforcement learning, and updates the model parameters of the initial decision model to obtain the updated decision model. When the iterative training reaches the preset training termination condition, the trained decision model is obtained.

[0077] Specific limitations regarding the decision model training system based on a disturbance-resistant environment model can be found in the limitations of the decision model training method based on a disturbance-resistant environment model described above, and will not be repeated here. Each module in the aforementioned decision model training system based on a disturbance-resistant environment model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0078] like Figure 3 As shown, in one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: Acquire historical operational data of the wastewater treatment process, including status data and corresponding control actions; Based on historical operating data, an initial disturbance immunity environment model is trained to obtain a trained disturbance immunity environment model. The trained disturbance immunity environment model is used to obtain the predicted state data for the next moment of disturbance immunity based on the current state data and the current control action. Construct an initial decision model to output the control action at the current moment based on the current state data and the control action at the previous moment; The trained disturbance-resistant environment model is used as a virtual interactive environment. Based on historical operating data, the initial decision model is iteratively trained in the virtual interactive environment through offline reinforcement learning, and the model parameters of the initial decision model are updated to obtain the updated decision model. When the iterative training reaches the preset training termination condition, the trained decision model is obtained.

[0079] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0080] In one embodiment, a computer-readable storage medium is provided, storing a computer program that, when executed by a processor, performs the following steps: Acquire historical operational data of the wastewater treatment process, including status data and corresponding control actions; Based on historical operating data, an initial disturbance immunity environment model is trained to obtain a trained disturbance immunity environment model. The trained disturbance immunity environment model is used to obtain the predicted state data for the next moment of disturbance immunity based on the current state data and the current control action. Construct an initial decision model to output the control action at the current moment based on the current state data and the control action at the previous moment; The trained disturbance-resistant environment model is used as a virtual interactive environment. Based on historical operating data, the initial decision model is iteratively trained in the virtual interactive environment through offline reinforcement learning, and the model parameters of the initial decision model are updated to obtain the updated decision model. When the iterative training reaches the preset training termination condition, the trained decision model is obtained.

[0081] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0082] Furthermore, in the various embodiments of the present invention, each functional module can be fully integrated into a processor, or each module can be a separate device, or two or more modules can be integrated into a device; each functional module in the various embodiments of the present invention can be implemented in hardware or in the form of hardware plus software functional units.

[0083] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, they perform the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0084] It should be understood that the use of terms such as "system," "apparatus," "unit," and / or "module" in this application is only applicable to distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0085] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "a," and / or "the" are not specifically singular and may include the plural. Generally, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0086] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0087] If a flowchart is used in this application, it is used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0088] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a decision model based on a disturbance-resistant environment model, characterized in that, include: Historical operational data of the wastewater treatment process is acquired, including status data and corresponding control actions. The status data includes one or more of the following: influent flow rate, influent chemical oxygen demand, ammonia nitrogen concentration, total nitrogen concentration, dissolved oxygen concentration in the aeration tank, mixed liquor sludge concentration, water temperature, and pH value. The control actions include one or more of the following: speed setting command for the aeration blower, frequency control command for the frequency converter, and opening command for the air supply valve. Based on the historical operating data, an initial disturbance immunity environment model is trained to obtain a trained disturbance immunity environment model; wherein, the trained disturbance immunity environment model is used to obtain the predicted state data for the next moment of disturbance immunity based on the current state data and the current control action. The initial disturbance-resistant environment model includes: a forward diffusion unit, a cross-attention network, a temporal convolutional network, and a prediction unit; The forward diffusion unit is used to add random perturbations to the original state data input into the initial disturbance-resistant environment model during the training phase of the initial disturbance-resistant environment model to obtain state features with perturbations; the random perturbations are used to simulate at least one of water quality fluctuations, flow rate changes, and sensor errors. The cross-attention network includes an embedding layer, a cross-attention encoder, and a cross-attention decoder; The embedding layer is used to generate action embedding features and state embedding features based on the input control action and the original state data; The cross-attention encoder is used to perform cross-attention encoding with the state embedding feature as the query and the joint embedding feature formed by the state embedding feature and the action embedding feature as the key and value, to obtain the encoded feature. The cross-attention decoder is used to perform cross-attention decoding with the encoded features as the query and the perturbed state features as the key and value, to obtain the decoded features; A temporal convolutional network is used to capture local temporal features in the encoded features and the perturbed state features to obtain local features; A prediction unit is used to obtain the predicted state data for the next time step based on the local features and the decoded features. Construct an initial decision model for outputting the control action at the current moment based on the state data at the current moment and the control action at the previous moment; Using the trained disturbance-resistant environment model as a virtual interactive environment, the initial decision model is iteratively trained in the virtual interactive environment through offline reinforcement learning based on the historical operating data, and the model parameters of the initial decision model are updated to obtain the updated decision model; when the iterative training reaches the preset training termination condition, the trained decision model is obtained.

2. The decision model training method according to claim 1, characterized in that, The process of training an initial disturbance-resistant environment model based on the historical operational data to obtain a trained disturbance-resistant environment model includes: Extract the current state data, the control action corresponding to the current state data, and the actual state data for the next moment from the historical operation data; The forward diffusion unit adds a random perturbation to the current state data to obtain perturbed state data; The current state data, the current control action, and the perturbed state data are input into the embedding unit to obtain the current state embedding feature, the current action embedding feature, and the perturbed current state embedding feature. The current state embedding feature, the current action embedding feature, and the perturbated current state embedding feature are input into the cross-attention network to obtain the encoded feature and the decoded feature; The encoded features and the perturbed state features are input into the temporal convolutional network to obtain the local features; The local features and the decoded features are input into the prediction unit to obtain the prediction state data for the next time step; With the goal of minimizing the error between the predicted state data and the actual state data at the next time step, the parameters of the initial disturbance-resistant environment model are iteratively updated using the gradient descent algorithm until the error converges to a preset threshold, thus obtaining a trained disturbance-resistant environment model.

3. The decision model training method according to claim 2, characterized in that, The construction of the initial decision model includes: constructing an initial decision model comprising a policy network and a value network; the policy network is used to output the control action at the current moment based on the input current moment state data and the control action at the previous moment; the value network is used to obtain the predicted future return based on the current moment state data; The initial decision model is obtained by initializing the parameters of the policy network and the value network.

4. The decision model training method according to claim 3, characterized in that, The step of using the trained disturbance-resistant environment model as a virtual interactive environment, and iteratively training the initial decision model in the virtual interactive environment based on the historical operating data through offline reinforcement learning to update the model parameters of the initial decision model, includes: Extract an initial state data and the corresponding control action from the previous moment from the historical operation data, and use them as the current state data and the control action from the previous moment. Using the policy network, the disturbance-resistant environment model, and a preset reward function, multi-step virtual interaction is performed on the current state data and the control action from the previous moment to obtain a virtual interaction trajectory; wherein, performing multi-step virtual interaction to obtain the virtual interaction trajectory includes: The current state data and the control action from the previous moment are input into the policy network to obtain the current control action; the current state data and the current control action are input into the disturbance-resistant environment model to obtain the predicted state data for the next moment; the instant reward for the current control action is calculated based on the preset reward function; wherein, the current control action, the predicted state data for the next moment, and the instant reward constitute a time step in the virtual interaction trajectory; Continue to perform virtual interaction on the control action at the current moment and the predicted state data at the next moment output by the policy network and the anti-disturbance environment model until the preset number of virtual interaction steps is reached to obtain the virtual interaction trajectory; The actual future reward for each time step is calculated based on the immediate reward at each time step in the virtual interaction trajectory. The predicted future returns for each time step are calculated based on the value network, and the advantage function is determined based on the actual future returns and the predicted future returns. The loss function of the policy network is constructed based on the advantage function, and the loss function of the value network is constructed based on the deviation between the predicted future return and the actual future return. Based on the loss function of the policy network and the loss function of the value network, the parameters of the policy network and the value network are updated through backpropagation algorithm to complete one iteration of training.

5. The decision model training method according to claim 4, characterized in that, The preset reward function is a multi-objective reward function, which includes at least: a water quality compliance reward, an energy consumption penalty, and a control action smoothness penalty. The water quality compliance reward items are positively correlated with the degree of compliance of the effluent water quality indicators; The energy consumption penalty term is positively correlated with the energy consumption value of the control action at the current moment; The control action smoothing penalty term is positively correlated with the change amplitude of the control action at adjacent time points; The immediate reward is obtained by summing the water quality compliance reward, the energy consumption penalty, and the control action smoothness penalty. The reward function aims to maximize the immediate reward so that the decision model outputs a control action that satisfies the water quality compliance constraint, minimizes the energy consumption of the control action, and ensures smooth control action changes.

6. The decision model training method according to claim 1, characterized in that, The preset training termination condition includes: the number of iterations reaches a preset threshold.

7. A decision model training system based on a disturbance-resistant environment model, characterized in that, For implementing the method as described in any one of claims 1-6, comprising: The acquisition module is used to acquire historical operating data of the wastewater treatment process, including status data and corresponding control actions. The environment model pre-training module is used to train an initial disturbance-resistant environment model based on the historical operating data to obtain a trained disturbance-resistant environment model; wherein, the trained disturbance-resistant environment model is used to obtain the predicted state data for the next moment of disturbance based on the current state data and the current control action; An initial strategy model construction module is used to construct an initial decision model for outputting the control action at the current moment based on the current state data and the control action at the previous moment; The iterative training module is used to use the trained disturbance-resistant environment model as a virtual interactive environment, and based on the historical running data, to iteratively train the initial decision model in the virtual interactive environment through offline reinforcement learning, and update the model parameters of the initial decision model to obtain an updated decision model; when the iterative training reaches the preset training termination condition, the trained decision model is obtained.

8. A computer device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the decision model training method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the decision model training method as described in any one of claims 1-6.

Images