An aeration equipment automatic collaborative scheduling method and system based on reinforcement learning

Abstract

The application relates to an aeration equipment automatic collaborative scheduling method and system based on reinforcement learning. The method comprises the following steps: acquiring aeration tank heterogeneous measurement data, processing the aeration tank heterogeneous measurement data through a three-layer clock alignment mechanism to obtain multi-source observation data with a global unified timestamp; inputting the observation data into a finite volume grid physical model, and correcting the observation data through ensemble Kalman filtering assimilation and Lagrange reverse tracking to obtain an aeration tank space-time state field; performing feature compression and fusion on the state field to obtain a high-dimensional feature vector, and mapping the high-dimensional feature vector to generate an action set; inputting the action set into a gray box digital twin model to deduce an expected return; selecting an optimal action and converting the optimal action into an instruction to input an aeration equipment control system. The method can solve the problems of multi-source data space-time asynchronization and state representation physical distortion, improve the strategy generalization ability and training efficiency, reduce the trial and error cost, realize the collaborative scheduling of the aeration equipment, and balance the water quality standard and energy saving and consumption reduction.

Description

Technical Field

[0001] This invention belongs to the field of wastewater treatment, and in particular relates to an automated collaborative scheduling method and system for aeration equipment based on reinforcement learning. Background Technology

[0002] With the increasing urgency of wastewater treatment plants seeking synergistic optimization of energy conservation, emission reduction, and water quality compliance, intelligent control technology for aeration processes based on reinforcement learning has attracted attention due to its ability to autonomously explore optimal strategies without requiring precise mechanistic models. This type of technology typically relies on sensor data to construct state vectors and trains a policy network through interaction with the environment to output control commands such as fan frequency and valve opening.

[0003] In traditional methods, sensor data synchronization is often achieved using network time protocols or simple shared clock lines. Data fusion is often achieved by using Kalman filtering to process each physical quantity independently and then concatenating them into a state vector. Policy training relies on purely data-driven environment models or direct online trial and error.

[0004] However, the above approach has two prominent problems. First, clock skew and transmission delay jitter from heterogeneous sensors lead to spatiotemporal asynchrony of multi-source data, and the fusion process ignores the hydrodynamic causal relationships between physical quantities such as dissolved oxygen and flow velocity, resulting in physically distorted state representations that cannot accurately reflect rapid dynamic processes such as bubble transport and diffusion. Second, the generalization ability of purely data-driven surrogate models deteriorates sharply when encountering inflow shocks or sudden changes in water temperature not covered in the training, while high-precision mechanistic models are computationally too time-consuming to meet the needs of online training, resulting in low efficiency of policy samples and high trial-and-error costs in real systems. Summary of the Invention

[0005] Therefore, it is necessary to provide an automated collaborative scheduling method and system for aeration equipment based on reinforcement learning, which can generate a spatiotemporally synchronized and physically consistent aeration tank state field and utilize physical prior knowledge to improve the efficiency of strategy training and generalization ability.

[0006] Firstly, this application provides an automated collaborative scheduling method for aeration equipment based on reinforcement learning, comprising:

[0007] S1. Obtain heterogeneous measurement data from the aeration tank to obtain multi-source observation data with a globally unified timestamp;

[0008] S2. Input the multi-source observation data into the finite volume grid physical model, perform ensemble Kalman filtering assimilation and Lagrange backtracking spatiotemporal correction on the multi-source observation data, and obtain the spatiotemporal state field of the aeration tank.

[0009] S3. Perform spatial feature compression and physical feature fusion on the spatiotemporal state field of the aeration tank to obtain a high-dimensional state feature vector, and perform action feature mapping on the high-dimensional state feature vector to obtain an action set.

[0010] S4. Input the action set into the pre-built gray box digital twin model, perform differentiable dynamics forward extrapolation on the candidate actions in the action set to obtain the future state prediction sequence, and calculate the expected reward corresponding to the candidate action based on the future state prediction sequence.

[0011] S5. Select the candidate action with the highest expected return from the action set as the optimal action, and convert the optimal action into a coordinated action command and input it into the aeration equipment control system.

[0012] In one embodiment, the gray-box digital twin model is constructed through the following steps:

[0013] S11. Based on the convection-diffusion mass transfer equation and microbial oxygen consumption kinetics, the calculation relationship of the nominal predicted value is obtained;

[0014] S12. Based on multi-source observation data, obtain sensor observations, dissolved oxygen concentration sequences, flow velocity sequences, actuator action sequences, and water temperature sequences to construct a training sample set;

[0015] S13. Construct an initial model of a deep neural network. Input the training sample set into the initial model of the deep neural network, perform nonlinear transformation, feature fusion and data fitting to obtain the oxygen mass transfer coefficient, microbial oxygen consumption rate and additive source term.

[0016] S14. Based on the calculation relationship of the nominal predicted value, the oxygen mass transfer coefficient, the microbial oxygen consumption rate and the additive source term are corrected and predicted to obtain the corrected dissolved oxygen predicted value.

[0017] S15. Fit the loss function to the input data of the corrected dissolved oxygen prediction value, calculate the physical regularization loss, and train the initial model of the deep neural network based on the physical regularization loss, the training sample set and the corrected dissolved oxygen prediction value to obtain the pre-trained deep neural network.

[0018] S16. Combine the finite volume grid physical model and the pre-trained deep neural network to obtain the gray box digital twin model; wherein, the gray box digital twin model uses the finite volume grid physical model as the nominal dynamic skeleton.

[0019] Based on the above embodiments, a loss function is fitted to the input data of the corrected dissolved oxygen prediction value to calculate the physical regularization loss. Then, based on the physical regularization loss, the training sample set, and the corrected dissolved oxygen prediction value, the initial model of the deep neural network is trained to obtain a pre-trained deep neural network, including:

[0020] S21. Calculate the squared mean of the difference between the sensor observations and the corrected dissolved oxygen predictions in the training sample set at the corresponding grid positions to obtain the observation fitting loss.

[0021] S22. Input the corrected dissolved oxygen prediction value into the data fitting loss function, and calculate the physical regularization loss; whereby the expression of the data fitting loss function is:

[0022]

[0023] in, For physical regularization loss, To correct the predicted dissolved oxygen value, for Dissolved oxygen concentration observation at time [time]. The time interval calculated between two consecutive dissolved oxygen concentration observations. For flow velocity vectors, The turbulent diffusion coefficient is... The oxygen mass transfer coefficient is... This represents the saturated dissolved oxygen concentration. For microbial oxygen consumption rate, This indicates summing over all grid cells;

[0024] S23. Add the observation fitting loss and the physical regularization loss to obtain the total loss;

[0025] S24. Update the parameters of the initial deep neural network model with the total loss using the backpropagation algorithm. Repeat S21 to S23 until the total loss converges to obtain the pre-trained deep neural network.

[0026] In one embodiment, multi-source observation data is input into a finite volume grid physical model. The multi-source observation data is then assimilated using ensemble Kalman filtering and subjected to Lagrange inverse tracking spatiotemporal correction to obtain the spatiotemporal state field of the aeration tank, including:

[0027] S31. Spatial discretization of the aeration tank and establishment of a prediction model based on fluid transport and biochemical reaction mechanisms;

[0028] S32. Run the set of Kalman filter prediction steps with a fixed control beat, take the spatiotemporal state field of the aeration tank output in the previous beat as the current posterior state field, and take the cooperative action command generated in the previous beat as the control input. Input the current posterior state field and the control input into the prediction model to deduce and obtain the prediction state set.

[0029] S33. Extract each dissolved oxygen observation value and the corresponding observation time and location from the multi-source observation data, and extract the flow velocity information at the observation location corresponding to the observation time from the forecast state set. Combine each dissolved oxygen observation value and the flow velocity information corresponding to each dissolved oxygen observation value to obtain the set of observation values ​​to be corrected.

[0030] S34. Perform Lagrange backward tracking spatiotemporal correction on the set of observations to be corrected to obtain spatiotemporally corrected observations;

[0031] S35. Calculate the ensemble Kalman gain matrix based on the residuals between the spatiotemporally corrected observations and the predicted state set;

[0032] S36. The predicted state set is weighted and corrected by the set of Kalman gain matrices to obtain the spatiotemporal state field of the aeration tank.

[0033] In one embodiment of the present invention, the spatiotemporal state field of the aeration tank is subjected to spatial feature compression and physical feature fusion to obtain a high-dimensional state feature vector. Then, action feature mapping is performed on the high-dimensional state feature vector to obtain an action set, including:

[0034] S41. Input the spatiotemporal state field of the aeration tank into the encoder of the collaborative scheduling strategy network to compress the spatial dimension of the spatiotemporal state field of the aeration tank and obtain the local feature map of each spatial region on different physical quantity channels.

[0035] S42. Calculate the correlation weights between different physical quantities through the cross-attention mechanism between different physical quantity channels, and use the correlation weights to perform weighted fusion of local feature maps to obtain the global physical context vector.

[0036] S43. Perform global average pooling on the local feature map to obtain the global feature vector. Concatenate the global feature vector and the global physical context vector, and map them to a fixed-dimensional vector through a fully connected network to obtain a high-dimensional state feature vector.

[0037] S44. Input the high-dimensional state feature vector into the decoder of the policy network to obtain the action distribution mean vector and covariance matrix, and randomly sample the multidimensional Gaussian distribution defined by the action distribution mean vector and covariance matrix to obtain the action set.

[0038] Secondly, this application also provides an automated collaborative scheduling system for aeration equipment based on reinforcement learning, used to implement the method described in the first aspect, including:

[0039] The time unification module is used to acquire heterogeneous measurement data from the aeration tank and obtain multi-source observation data with a globally unified timestamp.

[0040] The fusion correction module is used to input multi-source observation data into a finite volume grid physical model, perform ensemble Kalman filtering assimilation and Lagrange backtracking spatiotemporal correction on the multi-source observation data, and obtain the spatiotemporal state field of the aeration tank.

[0041] The feature processing module is used to compress spatial features and fuse physical features in the spatiotemporal state field of the aeration tank to obtain a high-dimensional state feature vector, and to perform action feature mapping on the high-dimensional state feature vector to obtain an action set.

[0042] The simulation and deduction module is used to input the action set into a pre-built gray box digital twin model, perform differentiable dynamic forward deduction on the candidate actions in the action set, and obtain the expected reward corresponding to the candidate actions; wherein, the gray box digital twin model uses a finite volume mesh physical model as the nominal dynamic skeleton;

[0043] The decision execution module is used to select the candidate action with the highest expected return from the candidate action set as the optimal action, and convert the optimal action into a coordinated action command and input it into the aeration equipment control system.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods in the first aspect of this application.

[0045] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods in the first aspect of this application.

[0046] The aforementioned reinforcement learning-based automated collaborative scheduling method and system for aeration equipment, through a three-layer clock alignment mechanism integrating precise time protocol synchronization, clock skew Kalman filter estimation, and sensor response delay compensation, synchronizes heterogeneous measurement data, solving the problem of spatiotemporal asynchrony of multi-source data caused by clock skew and transmission delay jitter of heterogeneous sensors. By inputting multi-source observation data into a finite volume grid physical model for ensemble Kalman filter assimilation and Lagrange backward tracking spatiotemporal correction, a physically consistent spatiotemporal state field of the aeration tank is generated, overcoming the deficiency of traditional fusion methods that ignore the causal relationship between physical quantities in fluid dynamics. By constructing a gray-box digital twin model with a finite-volume grid physical model as the nominal dynamic skeleton and a pre-trained deep neural network to provide unmodeled dynamic compensation, physical prior knowledge is embedded into the environmental model. This overcomes the problem of the degradation of generalization ability of pure data-driven proxy models under unseen operating conditions, while avoiding the drawback of excessive computation time of high-precision mechanism models. By performing differentiable dynamic forward extrapolation and selecting the optimal action in the gray-box digital twin model, the coordinated scheduling of fan frequency and the opening degree of each air inlet valve is realized, which improves the sample efficiency of strategy training, reduces the trial and error cost of real system, and balances the achievement of effluent water quality standards with energy saving and consumption reduction in aeration. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic diagram of an implementation environment provided for one embodiment of the present invention;

[0049] Figure 2 This is a flowchart of an automated collaborative scheduling method for aeration equipment based on reinforcement learning, according to one embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of an automated collaborative scheduling system for aeration equipment based on reinforcement learning, according to one embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] The automated collaborative scheduling method for aeration equipment based on reinforcement learning provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the collaborative scheduling terminal 100 communicates with the sensor 101 via a network. The collaborative scheduling terminal 100 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The sensor 101 can be, but is not limited to, a dissolved oxygen sensor, a flow rate sensor, and a pressure sensor.

[0053] In one exemplary embodiment, such as Figure 2 As shown, an automated collaborative scheduling method for aeration equipment based on reinforcement learning is provided, which is then applied to... Figure 1 Taking the collaborative scheduling terminal 100 as an example, the method includes the following steps:

[0054] S1. Obtain heterogeneous measurement data from the aeration tank to obtain multi-source observation data with a globally unified timestamp.

[0055] Specifically, heterogeneous measurement data can refer to various physical quantity data collected by sensors of different types and locations deployed in different areas of the aeration tank. The sampling characteristics and clock references of these sensors differ, leading to spatiotemporal asynchrony issues in the original data. A three-layer clock alignment mechanism can be used to address this spatiotemporal asynchrony. This mechanism includes three levels: precise time protocol synchronization, clock deviation Kalman filter estimation, and sensor response delay compensation. This mechanism can correct measurement data from different sources and with different time series to a unified time reference, resulting in multi-source observation data with a globally unified timestamp. Kalman filtering can be an algorithm for dynamic state estimation, using state equations and observation equations, combined with process noise and observation noise, to achieve accurate estimation of clock deviation. Delay compensation can employ linear interpolation to correct the timing deviation of the original measurement data based on the sensor response delay time, ensuring data timing consistency.

[0056] For example, the collaborative scheduling terminal 100 can acquire heterogeneous measurement data of the aeration tank through sensors 101 deployed in different areas of the aeration tank; the collaborative scheduling terminal 100 can adopt a three-layer clock alignment mechanism that integrates precise time protocol synchronization, clock deviation Kalman filter estimation, and sensor response delay compensation to synchronize the heterogeneous measurement data: the collaborative scheduling terminal 100 can calculate the network transmission delay and complete the master-slave clock coarse synchronization by sending synchronization messages and delay request response messages to the sensors 101; the collaborative scheduling terminal 100 can dynamically estimate the small clock deviation that still exists after coarse synchronization by calculating the state vector and observation values ​​using the Kalman filter model; wherein, the expression for calculating the state vector can be:

[0057]

[0058] in, for The state vector at time t, The state transition matrix is ​​used to describe the transition from state to state. Time's up The relationship of time transition, For process noise; the calculation expression for the observations can be:

[0059]

[0060] in, Let k be the observation value at time k. The observation matrix is ​​used to map the state vector to the observation space. To address random errors present during observation, the collaborative scheduling terminal 100 can perform a filtering process in two steps: prediction and update, to correct clock deviations. The collaborative scheduling terminal 100 can also use linear interpolation for delay compensation to obtain compensated measurement values. The expression for calculating the compensated measurement values ​​can be:

[0061]

[0062] in, for Measurement values ​​after time compensation for The original measurement value at time, for The original measurement value at time, For sensor response delay time, The sampling period of the sensor is denoted by 100. The collaborative scheduling terminal 100 can use the state vector and the observed values ​​to correct the compensated measured values ​​and obtain multi-source observation data.

[0063] S2. Input the multi-source observation data into the finite volume grid physical model, perform ensemble Kalman filtering assimilation and Lagrange backtracking spatiotemporal correction on the multi-source observation data, and obtain the spatiotemporal state field of the aeration tank.

[0064] Specifically, the finite volume grid physical model can be a mechanistic model constructed based on the actual geometric characteristics of the aeration tank. It uses the finite volume method to divide the aeration tank into several grid computational units, and simulates the physicochemical processes within the aeration tank by solving the governing equations related to fluid dynamics, bubble transport, and dissolved oxygen mass transfer. Ensemble Kalman filtering assimilation is a data assimilation method that generates probability distributions of approximate states from multiple sample sets, fuses multi-source observation data with model predictions, corrects model prediction biases, and improves state estimation accuracy. Lagrange backward tracking spatiotemporal correction is based on the Lagrange particle tracking principle. By tracking the bubble motion trajectory, it backward calculates the spatial position of bubbles at different time points, thereby correcting the grid state values ​​and compensating for local spatiotemporal deviations. The core governing equations of the finite volume grid physical model encompass the continuity equation, momentum equation, bubble transport equation, and dissolved oxygen mass transfer equation, used to accurately simulate various physicochemical processes within the aeration tank.

[0065] For example, the collaborative scheduling terminal 100 can input multi-source observation data into a pre-constructed finite volume grid physical model, and simulate the transfer and consumption process of dissolved oxygen in the aeration tank by solving the dissolved oxygen mass transfer equation to obtain dynamic data of dissolved oxygen concentration; wherein, the calculation expression of the dynamic data of dissolved oxygen concentration is:

[0066]

[0067] in, Dissolved oxygen concentration, For time, Let V be the velocity vector of the mixture. The dissolved oxygen diffusion coefficient, The volumetric mass transfer coefficient is . This represents the saturated dissolved oxygen concentration. For microbial oxygen consumption rate, The rate of change of dissolved oxygen concentration over time. The rate of change of dissolved oxygen concentration with space. This is the dissolved oxygen mass transfer consumption item; For dynamic dissolved oxygen concentration data, the collaborative scheduling terminal can use the dynamic dissolved oxygen concentration data as the initial predicted value of the finite volume grid physical model, incorporate it into the ensemble Kalman filter assimilation process, and calculate the residual and Kalman gain between the predicted value and the actual observed value by combining multi-source observation data to correct the initial predicted value and obtain dissolved oxygen concentration distribution data. Based on the dissolved oxygen concentration data, the collaborative scheduling terminal 100 can correct the state deviation of the local grid by using the Lagrange inverse tracking spatiotemporal correction method to obtain the spatiotemporal state field of the aeration tank, including the dissolved oxygen concentration distribution and other related physical quantities.

[0068] S3. Perform spatial feature compression and physical feature fusion on the spatiotemporal state field of the aeration tank to obtain a high-dimensional state feature vector, and perform action feature mapping on the high-dimensional state feature vector to obtain an action set.

[0069] Specifically, spatial feature compression is the core processing step of reducing the dimensionality of the spatiotemporal state field of the aeration tank and removing redundant information. Spatial feature compression can use a lightweight convolutional neural network to extract key spatial features in the spatiotemporal state field through operations such as convolution, pooling, and full connection. This reduces the amount of data processing while retaining information, completing feature extraction and dimensionality compression to obtain a low-dimensional spatial feature vector. Physical feature fusion can be the link between spatial feature compression and action feature mapping. Physical feature fusion can combine the compressed spatial features with the fluid dynamic causal relationships between various physical quantities in the aeration tank. By constructing an attention mechanism model, differentiated weights are assigned to each dimension of the spatial feature vector, thereby improving the accuracy and effectiveness of state feature representation. Action feature mapping can be the process of transforming high-dimensional state feature vectors into a set of executable candidate actions for aeration equipment. This is achieved through an action mapping network, which maps abstract high-dimensional feature vectors to specific combinations of aeration equipment control parameters, generating candidate actions covering different control scenarios.

[0070] For example, the collaborative scheduling terminal 100 can input the three-dimensional data of the spatiotemporal state field into a lightweight convolutional neural network, and sequentially perform convolution operations to extract local spatial features, pooling operations to reduce feature dimensionality, and fully connected operations to integrate global features, thereby completing spatial feature compression and outputting a low-dimensional spatial feature vector. The collaborative scheduling terminal 100 constructs an attention mechanism model to analyze the hydrodynamic causal relationships between various physical quantities in the aeration tank, and calculates the importance scores of each dimension of the spatial feature vector through the attention network to obtain weight coefficients; wherein, the expression for calculating the weight coefficients can be:

[0071]

[0072] in, For the first Weight coefficients of dimensional features For the first Importance score of dimensional features The sum of the importance scores for all dimensions is the exponential sum. The collaborative scheduling terminal 100 can perform weighted fusion of low-dimensional spatial feature vectors according to the weight coefficients to obtain high-dimensional state feature vectors. The collaborative scheduling terminal 100 performs action feature mapping on the high-dimensional state feature vectors, calls the pre-built action mapping network, inputs the high-dimensional state feature vectors into the action mapping network, and after passing through multiple fully connected operations, the output values ​​are transformed into specific control parameters through the mapping function to obtain the fan frequency and valve opening. Among them, the expression for calculating the fan frequency can be:

[0073]

[0074] In the formula, For the frequency of the fan, The output value of the motion mapping network; the expression for calculating the valve opening can be: In the formula, For valve opening; the collaborative scheduling terminal 100 can match the calculated fan frequency and valve opening of each group to obtain candidate actions; the collaborative scheduling terminal 100 can adopt a random disturbance strategy to apply small disturbances to the fan frequency and valve opening of each group of candidate actions to generate multiple different combinations of control parameters, each combination corresponding to an independent candidate action, and all these candidate actions are summarized to obtain an action set.

[0075] S4. Input the action set into the pre-built gray box digital twin model, perform differentiable dynamics forward extrapolation on the candidate actions in the action set to obtain the future state prediction sequence, and calculate the expected reward corresponding to the candidate action based on the future state prediction sequence.

[0076] Specifically, the gray-box digital twin model can be a hybrid model combining mechanistic and data-driven models. The mechanistic model can be constructed and simplified based on the physicochemical equations of the aeration process to improve computational efficiency. The data-driven model can use long short-term memory networks to compensate for the simplification errors of the mechanistic model and achieve a balance between accuracy and efficiency. Differentiable dynamics forward extrapolation can be based on automatic differentiation technology to extrapolate the changes in the aeration tank state of candidate actions over a period of time through the gray-box digital twin model and calculate the expected returns corresponding to the candidate actions. The expected return function can be used to evaluate the merits of candidate actions, taking into account the two core objectives of energy saving and water quality compliance. The fan energy consumption is calculated through specific equations to quantify the energy cost of the action.

[0077] For example, the collaborative scheduling terminal 100 can input the action set into a pre-constructed gray-box digital twin model, and perform differentiable dynamic forward extrapolation on each candidate action in the action set through the gray-box digital twin model. This extrapolates the changes in the aeration tank state of the candidate action within a preset future time period, calculates parameters such as dissolved oxygen concentration, energy consumption, and water quality indicators at each time step, and combines them to obtain a future state prediction sequence. The expression for calculating energy consumption can be:

[0078]

[0079] in, For energy consumption, Energy consumption coefficient The frequency of the wind turbine; the collaborative scheduling terminal 100 can calculate the expected return for each candidate action based on the future state prediction sequence and according to the expected return function; wherein, the calculation expression of the expected return function can be:

[0080]

[0081] in, For expected returns, This is the weighting factor for water quality returns. In return for water quality For energy consumption return, The weighting coefficients for energy consumption rewards are used; the collaborative scheduling terminal 100 can record the expected reward values ​​of all candidate actions, obtain a reward list, provide a basis for optimal action selection, and calculate the derivative of the reward function with respect to the action parameters for subsequent optimization of reinforcement learning strategies.

[0082] S5. Select the candidate action with the highest expected return from the candidate action set as the optimal action, and convert the optimal action into a coordinated action command and input it into the aeration equipment control system.

[0083] Specifically, the optimal action can be the candidate action with the highest expected return in the action set. If there are multiple candidate actions with the same expected return, the action with the lowest energy consumption is selected. At the same time, it must meet the action safety threshold requirements to avoid extreme action parameters. The coordinated action instruction can be a digital signal that the optimal action can be converted into a recognition of the aeration equipment control system and follow a preset communication protocol. The fault tolerance mechanism can be an auxiliary mechanism to ensure the stable operation of the aeration system. When a group of aeration units fails, the unit is automatically adjusted to a safe state, and the optimal action of the remaining units is recalculated to avoid water quality exceeding the standard or energy consumption surging.

[0084] For example, the collaborative scheduling terminal 100 can traverse all candidate actions in the action set, extract the expected return value of each candidate action, and select the candidate action with the highest expected return as the optimal action. If there are multiple candidate actions with the same expected return, the action with the lowest total energy consumption is selected. The collaborative scheduling terminal 100 can check whether the optimal action meets the preset action safety threshold. If it exceeds the threshold, it will automatically adjust to the nearest safety threshold and record the abnormal adjustment information. The collaborative scheduling terminal 100 can convert the optimal action into a collaborative action instruction. According to the preset communication protocol, it will convert the fan frequency and valve opening into digital signals that the equipment can recognize, obtain the action instructions corresponding to each aeration unit, and transmit them to the aeration equipment control system through the bus. The collaborative scheduling terminal can collect the operating status of each aeration unit and the multi-source observation data of the aeration tank in real time, and feed them back to the gray box digital twin model and reinforcement learning strategy network for subsequent strategy iteration optimization to form closed-loop control. When a fault is detected in a certain group of aeration units, the fault tolerance mechanism is automatically activated, the unit is adjusted to a safe state, and the optimal action of the remaining units is recalculated to ensure the stable operation of the aeration system.

[0085] The aforementioned reinforcement learning-based automated collaborative scheduling method for aeration equipment utilizes a three-layer clock alignment mechanism—integrating precise time protocol synchronization, clock skew Kalman filter estimation, and sensor response delay compensation—to synchronize heterogeneous measurement data. This addresses the spatiotemporal asynchrony problem caused by clock skew and transmission delay jitter in heterogeneous sensors. By inputting multi-source observation data into a finite volume grid physical model for ensemble Kalman filter assimilation and Lagrange backward tracking spatiotemporal correction, a physically consistent spatiotemporal state field for the aeration tank is generated, overcoming the shortcomings of traditional fusion methods that neglect the causal relationships between physical quantities in fluid dynamics. A gray-box digital twin model was constructed, using a finite-volume grid physical model as the nominal dynamic skeleton and a pre-trained deep neural network to provide unmodeled dynamic compensation. This model embeds prior physical knowledge into the environmental model, overcoming the problem of declining generalization ability of pure data-driven proxy models under unseen operating conditions, while avoiding the drawback of excessive computation time for high-precision mechanism models. By performing differentiable dynamic forward extrapolation and selecting the optimal action in the gray-box digital twin model, the coordinated scheduling of fan frequency and the opening degree of each air inlet valve was achieved, improving the sample efficiency of strategy training, reducing the trial and error cost of real systems, and balancing the achievement of effluent water quality standards with energy saving and consumption reduction in aeration.

[0086] In one embodiment of the present invention, the gray-box digital twin model is constructed through the following steps:

[0087] S11. Based on the convection-diffusion mass transfer equation and microbial oxygen consumption kinetics, the calculation relationship of the nominal predicted value is obtained.

[0088] Specifically, the convection-diffusion mass transfer equation can be an equation describing the transfer and spatial diffusion of dissolved oxygen and other substances in the aeration tank, and can be used to reflect the migration characteristics of substances in fluids; microbial oxygen consumption kinetics can be the theoretical basis for revealing the laws of dissolved oxygen consumption during microbial metabolism, and clarify the relationship between microbial oxygen consumption and related influencing factors; the nominal predicted value calculation relationship can be derived by combining the convection-diffusion mass transfer equation and microbial oxygen consumption kinetics, and is used as the basic relationship for preliminary calculation of the predicted dissolved oxygen value.

[0089] For example, the collaborative scheduling terminal can theoretically derive the calculation relationship of the nominal predicted value based on the convection-diffusion mass transfer equation and microbial oxygen consumption kinetics; wherein, the calculation expression of the convection-diffusion mass transfer equation can be:

[0090]

[0091] in, The rate of change of dissolved oxygen concentration. For Hamiltonian operators, The diffusion coefficient is... For source and sink items, This represents the dissolved oxygen concentration.

[0092] S12. Based on multi-source observation data, obtain sensor observations, dissolved oxygen concentration sequences, flow velocity sequences, actuator action sequences, and water temperature sequences to construct a training sample set.

[0093] Specifically, the dissolved oxygen concentration sequence can reflect the changes in dissolved oxygen concentration at different time points in the aeration tank; the flow velocity sequence can reflect the flow velocity of the mixed liquor in the aeration tank at different time points; the actuator action sequence can reflect the operating status and action parameters of the aeration equipment actuator at different time points; the water temperature sequence can reflect the temperature changes of the water in the aeration tank at different time points; and the training sample set can be a dataset generated by organizing and integrating the above dissolved oxygen concentration sequence, flow velocity sequence, actuator action sequence, and water temperature sequence.

[0094] For example, the collaborative scheduling terminal extracts dissolved oxygen concentration sequence, flow velocity sequence, actuator action sequence and water temperature sequence based on the acquired multi-source observation data. It performs simple processing such as denoising and alignment on each type of sequence, removes invalid data, and integrates the processed time series data to construct a training sample set for training deep neural networks.

[0095] S13. Construct an initial model of a deep neural network. Input the training sample set into the initial model of the deep neural network, perform nonlinear transformation, feature fusion and data fitting to obtain the oxygen mass transfer coefficient, microbial oxygen consumption rate and additive source term.

[0096] Specifically, the initial model of a deep neural network can be a model with a basic network structure but without training. The initial model of a deep neural network can consist of an input layer, a hidden layer, and an output layer. The initial model of a deep neural network can be used to fit three key parameters: the output oxygen mass transfer coefficient, the microbial oxygen consumption rate, and the additive source term. Forward propagation calculation is its core operation process.

[0097] For example, the collaborative scheduling terminal can construct an initial deep neural network model consisting of an input layer, a hidden layer, and an output layer: the number of nodes in the input layer matches the dimension of various time-series data in the training sample set; the hidden layer is set with multiple layers of neurons and selects appropriate activation functions to achieve feature extraction, transformation, and fitting of the input sample data; the output layer is set with three nodes, corresponding to the three target output parameters: oxygen mass transfer coefficient, microbial oxygen consumption rate, and additive source term; the collaborative scheduling terminal can input the constructed training sample set into the initial deep neural network model in batches, start the forward propagation calculation of the initial deep neural network model, and after receiving the sample data, the input layer passes it to the hidden layer layer by layer. The data is nonlinearly transformed and feature fused through activation functions, and then the processed feature data is passed to the output layer. Through the linear operation of the output layer, the sample data is fitted to obtain the oxygen mass transfer coefficient, microbial oxygen consumption rate, and additive source term. The collaborative scheduling terminal can perform preliminary range verification of the oxygen mass transfer coefficient, microbial oxygen consumption rate, and additive source term to ensure that the parameters conform to the actual operating conditions of the aeration tank.

[0098] S14. Based on the calculation relationship of nominal predicted values, the oxygen mass transfer coefficient, microbial oxygen consumption rate and additive source term are corrected and predicted to obtain the corrected dissolved oxygen predicted value.

[0099] Specifically, the corrected prediction can be a process of adjusting the initial dissolved oxygen prediction value by substituting key parameters into the calculation relationship based on the nominal prediction value; the corrected dissolved oxygen prediction value can be the dissolved oxygen prediction result obtained after parameter correction, which is closer to the actual working conditions than the nominal prediction value.

[0100] For example, the collaborative scheduling terminal can calculate the corrected dissolved oxygen prediction value based on the nominal prediction value calculation relationship, using the oxygen mass transfer coefficient, microbial oxygen consumption rate, and additive source term; wherein, the calculation expression for the corrected dissolved oxygen prediction value can be:

[0101]

[0102] in, To correct the predicted dissolved oxygen value, This is the initial nominal dissolved oxygen prediction value. The oxygen mass transfer coefficient is... This represents the saturated dissolved oxygen concentration. For microbial oxygen consumption rate, It is an additive source term.

[0103] S15. Fit the loss function to the input data of the corrected dissolved oxygen prediction value, calculate the physical regularization loss, and train the initial model of the deep neural network based on the physical regularization loss, the training sample set and the corrected dissolved oxygen prediction value to obtain the pre-trained deep neural network.

[0104] Specifically, the data fitting loss function can be a function used to calculate the deviation between the model's predicted value and the actual value; the pre-trained deep neural network can be a mature network model that has been trained, has optimized model parameters, and can accurately output key parameters.

[0105] For example, the collaborative scheduling terminal can fit a loss function to the input data of the corrected dissolved oxygen prediction value, and obtain the physical regularization loss by calculation. The collaborative scheduling terminal can combine the training sample set and the corrected dissolved oxygen prediction value to iteratively train the initial model of the deep neural network, continuously adjust the model parameters until the model converges, and obtain the pre-trained deep neural network.

[0106] S16. Combine the finite volume grid physical model and the pre-trained deep neural network to obtain the gray box digital twin model; wherein, the gray box digital twin model uses the finite volume grid physical model as the nominal dynamic skeleton.

[0107] For example, the collaborative scheduling terminal can use a finite volume grid physical model as a nominal dynamic skeleton and combine a pre-trained deep neural network with the skeleton to integrate the advantages of both, make up for the shortcomings of a single model, complete the construction of a gray box digital twin model, and ensure that the model can accurately predict the state changes of the aeration tank.

[0108] This application provides an automated collaborative scheduling method for aeration equipment based on reinforcement learning. By combining the convection-diffusion mass transfer equation and microbial oxygen consumption kinetics to derive the nominal prediction relationship, a training sample set is constructed based on multi-source observation data. Key parameters are output through deep neural network training, dissolved oxygen prediction values ​​are corrected, and the model is optimized. Finally, a gray-box digital twin model is constructed by combining a finite volume grid physical model with a pre-trained deep neural network. This effectively balances the accuracy of model mechanism and prediction precision, makes up for the shortcomings of a single model, and achieves accurate prediction of dissolved oxygen in the aeration tank, providing reliable model support for subsequent collaborative scheduling of aeration equipment.

[0109] Based on the above embodiments, a loss function is fitted to the input data of the corrected dissolved oxygen prediction value to calculate the physical regularization loss. Then, based on the physical regularization loss, the training sample set, and the corrected dissolved oxygen prediction value, the initial model of the deep neural network is trained to obtain a pre-trained deep neural network, including:

[0110] S21. Calculate the squared mean of the difference between the sensor observations and the corrected dissolved oxygen predictions in the training sample set at the corresponding grid positions to obtain the observation fitting loss.

[0111] Specifically, the observation fitting loss can be used as an indicator to measure the degree of deviation between the corrected dissolved oxygen prediction and the actual sensor observation.

[0112] For example, the collaborative scheduling terminal can extract the corrected dissolved oxygen prediction value and the dissolved oxygen sensor observation value in the training sample set, and match the corrected dissolved oxygen prediction value and the dissolved oxygen sensor observation value according to the grid calculation unit of the aeration tank to ensure that the sensor observation value at each grid position can correspond to the corrected dissolved oxygen prediction value at the same position; the collaborative scheduling terminal can calculate the difference between the sensor observation value and the corrected dissolved oxygen prediction value at each corresponding grid position; the collaborative scheduling terminal can perform a square operation on the difference to amplify the deviation and avoid the cancellation of positive and negative deviations, and take the arithmetic mean of the squared differences of all grid positions to obtain the observation fitting loss.

[0113] S22. Fit the loss function to the input data of the corrected dissolved oxygen prediction value and calculate the physical regularization loss.

[0114] Specifically, the expression for the data fitting loss function is:

[0115]

[0116] in, For physical regularization loss, To correct the predicted dissolved oxygen value, for Dissolved oxygen concentration observation at time [time]. The time interval calculated between two consecutive dissolved oxygen concentration observations. For flow velocity vectors, The turbulent diffusion coefficient is... The oxygen mass transfer coefficient is... This represents the saturated dissolved oxygen concentration. For microbial oxygen consumption rate, This indicates that the summation is performed on all grid cells.

[0117] Specifically, physical regularization loss can be the loss value calculated by fitting the loss function to the data. Physical regularization loss can be used to constrain the training process of the initial model of the deep neural network, ensuring that the corrected dissolved oxygen prediction value output by the model conforms to the physical laws of dissolved oxygen mass transfer and consumption in the aeration tank, and avoiding the model from producing prediction results that do not conform to the actual working conditions.

[0118] S23. Add the observation fitting loss and the physical regularization loss to obtain the total loss.

[0119] For example, the collaborative scheduling terminal can sum the observation fitting loss and the physical regularization loss to obtain the total loss. The collaborative scheduling terminal can judge the current training effect of the initial deep neural network model based on the magnitude of the total loss. The smaller the total loss, the higher the prediction accuracy of the initial deep neural network model and the more it conforms to physical laws, thus indicating the optimization direction for subsequent model parameter updates.

[0120] S24. Update the parameters of the initial deep neural network model with the total loss using the backpropagation algorithm. Repeat S21 to S23 until the total loss converges to obtain the pre-trained deep neural network.

[0121] Specifically, the backpropagation algorithm can be used to update the parameters of a deep neural network model. By propagating the gradient of the total loss in reverse, the parameters of neurons in each layer of the deep neural network model are adjusted to optimize the deep neural network model. The convergence of the total loss can refer to the total loss value stabilizing and no longer decreasing significantly, at which point the model parameters reach the optimal state. The pre-trained deep neural network can be a mature model that has undergone parameter optimization and can accurately output key parameters.

[0122] For example, the collaborative scheduling terminal can use the backpropagation algorithm to calculate the gradient of the total loss with respect to the parameters of each layer of the initial deep neural network model, with the total loss as the optimization objective. The collaborative scheduling terminal can gradually adjust the neuron parameters of the input layer, hidden layer and output layer of the initial deep neural network model according to the gradient direction to reduce the total loss. After the adjustment is completed, the collaborative scheduling terminal can repeat steps S21 to S23 to recalculate the observation fitting loss, physical regularization loss and total loss, and continuously iterate to adjust the parameters of the initial deep neural network model until the total loss tends to stabilize and no longer decreases significantly, that is, the total loss converges. At this time, training is stopped and the pre-trained deep neural network is obtained.

[0123] This application provides an automated collaborative scheduling method for aeration equipment based on reinforcement learning. It measures and corrects the deviation between the predicted and actual dissolved oxygen values ​​by calculating the fitting loss of observed values. Combined with the physical regularization loss constraint model to fit the physical laws of the aeration tank, the two are summed to obtain the total loss. Then, the backpropagation algorithm is used to iteratively update the initial model parameters of the deep neural network until the total loss converges, and finally a pre-trained deep neural network is obtained. This effectively improves the model's fitting accuracy of key parameters, ensures that the model output conforms to the actual working conditions, and provides reliable data-driven support for the construction of gray box digital twin models.

[0124] In one embodiment of the present invention, multi-source observation data is input into a finite volume grid physical model, and ensemble Kalman filtering assimilation and Lagrange inverse tracking spatiotemporal correction are performed on the multi-source observation data to obtain the spatiotemporal state field of the aeration tank, including:

[0125] S31. Spatial discretization of the aeration tank and establishment of a prediction model based on fluid transport and biochemical reaction mechanisms.

[0126] For example, the collaborative scheduling terminal can discretize the space of the aeration tank based on the actual geometric size and structural layout of the aeration tank using the finite volume method, dividing it into several regular grid computing units, and clarifying the boundary range and spatial coordinates of each unit. The collaborative scheduling terminal can combine the dissolved oxygen mass transfer law and fluid flow characteristics in the aeration tank to construct a prediction model, and integrate the discretized grid unit parameters and fluid dynamic characteristics into the model to complete the construction of the prediction model.

[0127] S32. Run the set of Kalman filter prediction steps with a fixed control beat, take the spatiotemporal state field of the aeration tank output in the previous beat as the current posterior state field, and take the cooperative action command generated in the previous beat as the control input. Input the current posterior state field and the control input into the prediction model to deduce the prediction state set.

[0128] Specifically, the ensemble Kalman filter prediction step can be a step in the model derivation process. The ensemble Kalman filter prediction step can be used to predict the current state of the aeration tank based on historical states and control inputs. The posterior state field can be the state result of the previous control cycle. The control input can be the action command used to drive the aeration equipment. The prediction state set is a set containing multiple possible prediction results, which is used for subsequent correction and optimization.

[0129] For example, the collaborative scheduling terminal can initiate the ensemble Kalman filter forecasting process according to a preset fixed control cycle: the collaborative scheduling terminal can extract the spatiotemporal state field of the aeration tank output in the previous cycle as the current posterior state field, and retrieve the collaborative action command generated in the previous cycle as the control input; the collaborative scheduling terminal can input the current posterior state field and the control input into the forecasting model, and deduce a variety of possible prediction results through the forecasting model, and integrate these prediction results to form a forecast state set.

[0130] S33. Extract each dissolved oxygen observation value and the corresponding observation time and location from the multi-source observation data, and extract the flow velocity information at the observation location corresponding to the observation time from the forecast state set. Combine each dissolved oxygen observation value and the flow velocity information corresponding to each dissolved oxygen observation value to obtain the set of observation values ​​to be corrected.

[0131] Specifically, the set of observations to be corrected can be data used to verify the accuracy of the forecast state set. The set of observations to be corrected can be composed of dissolved oxygen observations and flow velocity information at the corresponding locations in multi-source observation data. It can comprehensively reflect the actual operating status of the aeration tank and provide data support for subsequent correction work.

[0132] For example, the collaborative scheduling terminal can extract all dissolved oxygen observation values ​​from multi-source observation data and record the observation time and specific grid location corresponding to each dissolved oxygen observation value; the collaborative scheduling terminal can extract the flow velocity information corresponding to the observation time and the corresponding grid location from the forecast state set; the collaborative scheduling terminal can match and combine each set of dissolved oxygen observation values ​​and the flow velocity information corresponding to the dissolved oxygen observation values ​​to obtain the set of observation values ​​to be corrected.

[0133] S34. Perform Lagrange back-tracking spatiotemporal correction on the set of observations to be corrected to obtain spatiotemporally corrected observations.

[0134] Specifically, Lagrange backward tracking spatiotemporal correction can be used to correct spatiotemporal deviations between forecast states and actual observations. By tracing the migration trajectory of matter, it can calibrate the spatiotemporal matching accuracy of dissolved oxygen observations, ensuring that the observed data are consistent with the physical processes of the forecast model.

[0135] For example, the collaborative scheduling terminal can use the Lagrange inverse tracking method to trace the migration trajectory of dissolved oxygen at the corresponding observation time based on the dissolved oxygen observation value and flow velocity information in the set of observation values ​​to be corrected; the collaborative scheduling terminal can combine the fluid flow law in the aeration tank to correct the spatiotemporal deviation between the observed value and the predicted value, remove abnormal deviation data, and obtain the spatiotemporally corrected observation value.

[0136] S35. Calculate the ensemble Kalman gain matrix based on the residuals between the spatiotemporally corrected observations and the predicted state set.

[0137] For example, the collaborative scheduling terminal can calculate the difference between each predicted value and its corresponding spatiotemporally corrected observation in the forecast state set, obtain the residuals of each set of data, and calculate the ensemble Kalman gain matrix by statistically analyzing the distribution characteristics of the residuals and combining them with the physical process constraints of the aeration tank; wherein, the calculation expression of the ensemble Kalman gain matrix can be:

[0138] in, for The Kalman gain matrix of the time set, To predict the covariance matrix, For the observation matrix, To observe the noise covariance matrix.

[0139] S36. The predicted state set is weighted and corrected by the set of Kalman gain matrices to obtain the spatiotemporal state field of the aeration tank.

[0140] For example, the collaborative scheduling terminal can apply the set of Kalman gain matrices to the set of predicted states and perform weighted correction on each predicted value in the set of predicted states, amplifying the weight of the predicted results that are close to the actual observations and reducing the impact of the predicted results with large deviations; after the correction is completed, the collaborative scheduling terminal can integrate all the corrected predicted values ​​to obtain the spatiotemporal state field of the aeration tank.

[0141] This application provides an automated collaborative scheduling method for aeration equipment based on reinforcement learning. It constructs a forecast model by spatially discretizing the aeration tank, obtains a forecast state set by combining ensemble Kalman filter forecast steps, integrates dissolved oxygen observations and flow velocity information to form a set to be corrected, and performs spatiotemporal correction using Lagrange backtracking. Finally, it calculates the ensemble Kalman gain matrix to weight and correct the forecast state set, ultimately obtaining an accurate spatiotemporal state field for the aeration tank. This effectively corrects observation and forecast deviations, improves the accuracy of state field extrapolation, and provides precise state support for the collaborative scheduling of aeration equipment.

[0142] In one embodiment of the present invention, the spatiotemporal state field of the aeration tank is subjected to spatial feature compression and physical feature fusion to obtain a high-dimensional state feature vector. Then, action feature mapping is performed on the high-dimensional state feature vector to obtain an action set, including:

[0143] S41. Input the spatiotemporal state field of the aeration tank into the encoder of the collaborative scheduling strategy network to compress the spatial dimension of the spatiotemporal state field of the aeration tank, and obtain the local feature map of each spatial region on different physical quantity channels.

[0144] For example, the collaborative scheduling terminal can input the spatiotemporal state field data of the aeration tank into the encoder, and perform dimensionality compression and feature extraction on the data through operations such as convolution and pooling to obtain local feature maps of each region; wherein, the calculation expression of the local feature map can be:

[0145]

[0146] in, For local feature maps, For encoder convolution kernel weights, This is the spatiotemporal state data of the aeration tank. For bias terms, This is the activation function.

[0147] S42. Calculate the correlation weights between different physical quantities through the cross-attention mechanism between different physical quantity channels, and use the correlation weights to perform weighted fusion of local feature maps to obtain the global physical context vector.

[0148] For example, the collaborative scheduling terminal can perform correlation analysis on the local feature maps of different physical quantity channels through the cross-attention mechanism and calculate the correlation weight of each channel; the collaborative scheduling terminal can assign corresponding weights according to the correlation degree of each physical quantity, and obtain global features that can reflect the overall operating status through weighted fusion.

[0149] S43. Perform global average pooling on the local feature map to obtain the global feature vector. Concatenate the global feature vector and the global physical context vector, and map them to a fixed-dimensional vector through a fully connected network to obtain the high-dimensional state feature vector.

[0150] For example, the collaborative scheduling terminal can perform global average pooling on the local feature maps of each spatial region to integrate them into global features; the collaborative scheduling terminal can concatenate the global features and the global physical context vector, and perform dimension mapping through a fully connected network to obtain a high-dimensional state feature vector with fixed dimensions.

[0151] S44. Input the high-dimensional state feature vector into the decoder of the policy network to obtain the action distribution mean vector and covariance matrix, and randomly sample the multidimensional Gaussian distribution defined by the action distribution mean vector and covariance matrix to obtain the action set.

[0152] For example, the collaborative scheduling terminal can input a high-dimensional state feature vector into the decoder of the policy network, obtain predicted values ​​of control parameters such as fan frequency and valve opening through nonlinear transformation, and calculate the action condition probability distribution based on the predicted values; wherein, the calculation expression of the action condition probability distribution can be:

[0153] in, For the conditional probability distribution of actions, It is a high-dimensional state feature vector. The mean of the action predictions. The action prediction covariance; the collaborative scheduling terminal can randomly sample the multidimensional Gaussian distribution defined by the action condition probability distribution. During the sampling process, combined with the actual operating parameter range of the aeration equipment, the sampling results that meet the working conditions are selected. Each sampling result corresponds to a set of control parameters such as fan frequency and valve opening, that is, a candidate action. All candidate actions that meet the requirements are summarized and integrated to obtain the action set.

[0154] This application provides an automated collaborative scheduling method for aeration equipment based on reinforcement learning. The method extracts features and compresses the dimensionality of the spatiotemporal state field of the aeration tank through an encoder to obtain a local feature map. It then uses a cross-attention mechanism to fuse the associated features of various physical quantities, generates a high-dimensional state feature vector through global pooling and fully connected mapping, and finally selects candidate actions that meet the working conditions by random sampling through the action conditional probability distribution to form an action set. This achieves efficient integration of spatiotemporal state features and action mapping, providing diversified and precise candidate action support for the optimal collaborative scheduling of aeration equipment.

[0155] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0156] Based on the same inventive concept, this application also provides a reinforcement learning-based automated collaborative scheduling system for aeration equipment to implement the reinforcement learning-based automated collaborative scheduling method for aeration equipment described above. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more reinforcement learning-based automated collaborative scheduling system embodiments provided below can be found in the limitations of the reinforcement learning-based automated collaborative scheduling method for aeration equipment described above, and will not be repeated here.

[0157] In one exemplary embodiment, such as Figure 3 As shown, an automated collaborative scheduling system 500 for aeration equipment based on reinforcement learning is provided to implement the methods in the above-described embodiments, including:

[0158] The time unification module 501 is used to acquire heterogeneous measurement data from the aeration tank and obtain multi-source observation data with a globally unified timestamp.

[0159] The fusion correction module 502 is used to input multi-source observation data into a finite volume grid physical model, perform ensemble Kalman filtering assimilation and Lagrange back-tracking spatiotemporal correction on the multi-source observation data, and obtain the spatiotemporal state field of the aeration tank.

[0160] The feature processing module 503 is used to perform spatial feature compression and physical feature fusion on the spatiotemporal state field of the aeration tank to obtain a high-dimensional state feature vector, and to perform action feature mapping on the high-dimensional state feature vector to obtain an action set.

[0161] The simulation and deduction module 504 is used to input the action set into the pre-constructed gray box digital twin model, perform differentiable dynamic forward deduction on the candidate actions in the action set, and obtain the expected reward corresponding to the candidate actions; wherein, the gray box digital twin model uses a finite volume mesh physical model as the nominal dynamic skeleton;

[0162] The decision execution module 505 is used to select the candidate action with the highest expected return from the candidate action set as the optimal action, and convert the optimal action into a coordinated action instruction and input it into the aeration equipment control system.

[0163] In one embodiment of the present invention, the gray-box digital twin model is constructed through the following steps:

[0164] S11. Based on the convection-diffusion mass transfer equation and microbial oxygen consumption kinetics, the calculation relationship of the nominal predicted value is obtained;

[0165] S12. Based on multi-source observation data, obtain sensor observations, dissolved oxygen concentration sequences, flow velocity sequences, actuator action sequences, and water temperature sequences to construct a training sample set;

[0166] S13. Construct an initial model of a deep neural network. Input the training sample set into the initial model of the deep neural network, perform nonlinear transformation, feature fusion and data fitting to obtain the oxygen mass transfer coefficient, microbial oxygen consumption rate and additive source term.

[0167] S14. Based on the calculation relationship of the nominal predicted value, the oxygen mass transfer coefficient, the microbial oxygen consumption rate and the additive source term are corrected and predicted to obtain the corrected dissolved oxygen predicted value.

[0168] S15. Fit the loss function to the input data of the corrected dissolved oxygen prediction value, calculate the physical regularization loss, and train the initial model of the deep neural network based on the physical regularization loss, the training sample set and the corrected dissolved oxygen prediction value to obtain the pre-trained deep neural network.

[0169] S16. Combine the finite volume grid physical model and the pre-trained deep neural network to obtain the gray box digital twin model; wherein, the gray box digital twin model uses the finite volume grid physical model as the nominal dynamic skeleton.

[0170] Based on the above embodiments, a loss function is fitted to the input data of the corrected dissolved oxygen prediction value to calculate the physical regularization loss. Then, based on the physical regularization loss, the training sample set, and the corrected dissolved oxygen prediction value, the initial model of the deep neural network is trained to obtain a pre-trained deep neural network, including:

[0171] S21. Calculate the squared mean of the difference between the sensor observations and the corrected dissolved oxygen predictions in the training sample set at the corresponding grid positions to obtain the observation fitting loss.

[0172] S22. Fit the loss function to the input data of the corrected dissolved oxygen prediction value and calculate the physical regularization loss.

[0173] S23. Add the observation fitting loss and the physical regularization loss to obtain the total loss;

[0174] S24. Update the parameters of the initial deep neural network model with the total loss using the backpropagation algorithm. Repeat S21 to S23 until the total loss converges to obtain the pre-trained deep neural network.

[0175] In one embodiment, the fusion correction module 502 can also be used for:

[0176] S31. Discretize the aeration tank space to obtain the prediction model;

[0177] S32. Run the set of Kalman filter prediction steps with a fixed control beat, take the spatiotemporal state field of the aeration tank output in the previous beat as the current posterior state field, and take the cooperative action command generated in the previous beat as the control input. Input the current posterior state field and the control input into the prediction model to deduce and obtain the prediction state set.

[0178] S33. Extract each dissolved oxygen observation value and the corresponding observation time and location from the multi-source observation data, and extract the flow velocity information at the observation location corresponding to the observation time from the forecast state set. Combine each dissolved oxygen observation value and the flow velocity information corresponding to each dissolved oxygen observation value to obtain the set of observation values ​​to be corrected.

[0179] S34. Perform Lagrange backward tracking spatiotemporal correction on the set of observations to be corrected to obtain spatiotemporally corrected observations;

[0180] S35. Calculate the ensemble Kalman gain matrix based on the residuals between the spatiotemporally corrected observations and the predicted state set;

[0181] S36. The predicted state set is weighted and corrected by the set of Kalman gain matrices to obtain the spatiotemporal state field of the aeration tank.

[0182] In one embodiment, the feature processing module 503 can also be used for:

[0183] S41. Input the spatiotemporal state field of the aeration tank into the encoder of the collaborative scheduling strategy network to compress the spatial dimension of the spatiotemporal state field of the aeration tank and obtain the local feature map of each spatial region on different physical quantity channels.

[0184] S42. Calculate the correlation weights between different physical quantities through the cross-attention mechanism between different physical quantity channels, and use the correlation weights to perform weighted fusion of local feature maps to obtain the global physical context vector.

[0185] S43. Perform global average pooling on the local feature map to obtain the global feature vector. Concatenate the global feature vector and the global physical context vector, and map them to a fixed-dimensional vector through a fully connected network to obtain a high-dimensional state feature vector.

[0186] S44. Input the high-dimensional state feature vector into the decoder of the policy network to obtain the action distribution mean vector and covariance matrix, and randomly sample the multidimensional Gaussian distribution defined by the action distribution mean vector and covariance matrix to obtain the action set.

[0187] In one embodiment, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0188] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0189] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0190] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. An automated collaborative scheduling method for aeration equipment based on reinforcement learning, characterized in that, The method includes: S1. Obtain heterogeneous measurement data from the aeration tank to obtain multi-source observation data with a globally unified timestamp; S2. Input the multi-source observation data into the finite volume grid physical model, perform ensemble Kalman filtering assimilation and Lagrange inverse tracking spatiotemporal correction on the multi-source observation data, and obtain the spatiotemporal state field of the aeration tank. S3. Perform spatial feature compression and physical feature fusion on the spatiotemporal state field of the aeration tank to obtain a high-dimensional state feature vector, and perform action feature mapping on the high-dimensional state feature vector to obtain an action set. S4. Input the action set into a pre-constructed gray-box digital twin model, perform differentiable dynamics forward extrapolation on the candidate actions in the action set to obtain a future state prediction sequence, and calculate the expected reward corresponding to the candidate action based on the future state prediction sequence. S5. Select the candidate action with the highest expected return in the action set as the optimal action, and convert the optimal action into a collaborative action command and input it into the aeration equipment control system.

2. The method according to claim 1, characterized in that, The gray-box digital twin model is constructed through the following steps: S11. Based on the convection-diffusion mass transfer equation and microbial oxygen consumption kinetics, the calculation relationship of the nominal predicted value is obtained; S12. Based on the multi-source observation data, obtain sensor observation values, dissolved oxygen concentration sequence, flow rate sequence, actuator action sequence and water temperature sequence, and construct a training sample set; S13. Construct an initial model of a deep neural network, input the training sample set into the initial model of the deep neural network, perform nonlinear transformation, feature fusion and data fitting to obtain the oxygen mass transfer coefficient, microbial oxygen consumption rate and additive source term; S14. Based on the nominal predicted value calculation relationship, the oxygen mass transfer coefficient, the microbial oxygen consumption rate and the additive source term are corrected and predicted to obtain the corrected dissolved oxygen predicted value. S15. Input the corrected dissolved oxygen prediction value into the data fitting loss function, calculate the physical regularization loss, and train the initial model of the deep neural network based on the physical regularization loss, the training sample set and the corrected dissolved oxygen prediction value to obtain a pre-trained deep neural network. S16. The finite volume grid physical model and the pre-trained deep neural network are combined to obtain the gray box digital twin model; wherein the gray box digital twin model uses the finite volume grid physical model as the nominal dynamic skeleton.

3. The method according to claim 2, characterized in that, The step of fitting a loss function to the input data of the corrected dissolved oxygen prediction value, calculating the physical regularization loss, and training the initial model of the deep neural network based on the physical regularization loss, the training sample set, and the corrected dissolved oxygen prediction value to obtain a pre-trained deep neural network includes: S21. Calculate the squared mean of the difference between the sensor observations and the corrected dissolved oxygen predictions in the training sample set at the corresponding grid positions to obtain the observation fitting loss. S22. Input the corrected dissolved oxygen prediction value into the data fitting loss function to calculate the physical regularization loss; wherein, the expression of the data fitting loss function is: ； in, The physical regularization loss is... For the corrected dissolved oxygen prediction value, for Dissolved oxygen concentration observation at time [time]. The time interval calculated between two consecutive dissolved oxygen concentration observations. For flow velocity vectors, The turbulent diffusion coefficient is... The oxygen mass transfer coefficient is... This represents the saturated dissolved oxygen concentration. The oxygen consumption rate of the microorganisms. This indicates summing over all grid cells; S23. Add the observation fitting loss and the physical regularization loss to obtain the total loss; S24. Update the parameters of the initial model of the deep neural network with the total loss using the backpropagation algorithm, and repeat S21 to S23 until the total loss converges to obtain the pre-trained deep neural network.

4. The method according to claim 1, characterized in that, The process of inputting the multi-source observation data into a finite volume grid physical model, performing ensemble Kalman filtering assimilation and Lagrange inverse tracking spatiotemporal correction on the multi-source observation data, and obtaining the spatiotemporal state field of the aeration tank includes: S31. Spatial discretization of the aeration tank is performed, and a prediction model is established based on fluid transport and biochemical reaction mechanisms; S32. Run the set of Kalman filter prediction steps with a fixed control beat, take the spatiotemporal state field of the aeration tank output in the previous beat as the current posterior state field, and take the cooperative action command generated in the previous beat as the control input. Input the current posterior state field and the control input into the prediction model for deduction to obtain the prediction state set. S33. Extract each dissolved oxygen observation value and the observation time and observation location corresponding to each dissolved oxygen observation value from the multi-source observation data, and extract the flow velocity information at the observation location corresponding to the observation time from the forecast state set. Combine each dissolved oxygen observation value and the flow velocity information corresponding to each dissolved oxygen observation value to obtain a set of observation values ​​to be corrected. S34. Perform Lagrange backward tracking spatiotemporal correction on the set of observations to be corrected to obtain spatiotemporally corrected observations; S35. Calculate the set Kalman gain matrix based on the residual between the spatiotemporal corrected observations and the predicted state set; S36. The predicted state set is weighted and corrected by the set Kalman gain matrix to obtain the spatiotemporal state field of the aeration tank.

5. The method according to claim 1, characterized in that, The spatial feature compression and physical feature fusion of the spatiotemporal state field of the aeration tank are performed to obtain a high-dimensional state feature vector. Then, action feature mapping is performed on the high-dimensional state feature vector to obtain an action set, including: S41. Input the spatiotemporal state field of the aeration tank into the encoder of the collaborative scheduling strategy network to compress the spatial dimension of the spatiotemporal state field of the aeration tank, and obtain the local feature map of each spatial region on different physical quantity channels. S42. Calculate the correlation weights between different physical quantities through the cross-attention mechanism between different physical quantity channels, and use the correlation weights to perform weighted fusion on the local feature map to obtain the global physical context vector. S43. Perform global average pooling on the local feature map to obtain a global feature vector. Concatenate the global feature vector and the global physical context vector, and map them into a fixed-dimensional vector through a fully connected network to obtain a high-dimensional state feature vector. S44. Input the high-dimensional state feature vector into the decoder of the policy network to obtain the action distribution mean vector and covariance matrix, and randomly sample the multidimensional Gaussian distribution defined by the action distribution mean vector and covariance matrix to obtain the action set.

6. An automated collaborative scheduling system for aeration equipment based on reinforcement learning, used to implement the method according to any one of claims 1 to 5, characterized in that, The system includes: The time unification module is used to acquire heterogeneous measurement data from the aeration tank and obtain multi-source observation data with a globally unified timestamp. The fusion correction module is used to input the multi-source observation data into the finite volume grid physical model, perform ensemble Kalman filter assimilation and Lagrange inverse tracking spatiotemporal correction on the multi-source observation data, and obtain the spatiotemporal state field of the aeration tank. The feature processing module is used to perform spatial feature compression and physical feature fusion on the spatiotemporal state field of the aeration tank to obtain a high-dimensional state feature vector, and to perform action feature mapping on the high-dimensional state feature vector to obtain an action set. The simulation and deduction module is used to input the action set into a pre-constructed gray-box digital twin model, perform differentiable dynamics forward deduction on the candidate actions in the action set, obtain a future state prediction sequence, and calculate the expected reward corresponding to the candidate action based on the future state prediction sequence. The decision execution module is used to select the candidate action with the highest expected return from the candidate action set as the optimal action, and convert the optimal action into a coordinated action command and input it into the aeration equipment control system.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images