Urban pollution source dynamic distribution regulation and control method based on reinforcement learning

Abstract

The invention discloses an urban pollution source dynamic distribution regulation and control method based on reinforcement learning, and the method comprises the following steps: S1, collecting source data of air pollution, carrying out the preprocessing, and constructing a pollution data set; s2, performing time sequence prediction on the pollution data set by using an improved Informer network; s3, calculating the influence degree of different pollution sources on the air quality, and constructing a pollution source regulation and control task data set; s4, establishing a game theory multi-agent reinforcement learning environment, optimizing a pollution source agent regulation and control strategy by using Nash equilibrium, optimizing a reward mechanism based on pollution historical data, and dynamically adjusting the pollution source regulation and control strategy; s5, calculating a pollution source adjustment scheme, and optimizing pollution regulation and control parameters; and S6, executing the pollution regulation and control strategy, and performing iterative optimization. According to the method, the improved Informer network and game theory multi-agent reinforcement learning are combined, accurate prediction of the pollutant concentration and intelligent regulation and control of the pollution source are realized, and the method has the advantages of high precision, high adaptability and global optimization capability.

Description

Technical Field

[0001] The present invention relates to the field of environmental pollution control, and in particular to a method for dynamic distribution control of urban pollution sources based on reinforcement learning. Background Art

[0002] The sources of air pollution are complex and diverse, including industrial emissions, motor vehicle exhaust, construction dust, and residents' daily emissions. The emission characteristics and diffusion patterns of different pollution sources vary significantly. Therefore, how to effectively monitor, predict and intelligently control emissions from urban pollution sources has become one of the core issues in the current environmental governance field.

[0003] Existing pollution source control methods are mainly based on atmospheric physical diffusion models, such as the Gaussian plume model, the Lagrangian particle model and the Eulerian grid model. These methods rely on mathematical modeling of the diffusion laws of pollutants. Although they can simulate the propagation process of pollutants in the atmosphere to a certain extent, their applicability is limited by the assumptions of the model. For example, the Gaussian plume model assumes that the diffusion of pollutants is a uniform and continuous process, which makes it difficult to cope with complex terrain, non-steady-state meteorological conditions and the interaction of multiple sources of pollution. In addition, traditional air pollution prediction methods rely on statistical analysis and machine learning models, such as the autoregressive moving average model, support vector machine and random forest. Although these methods have a certain degree of accuracy in short-term predictions, they still have great limitations in long-term predictions, modeling of nonlinear pollution evolution processes and coordinated prediction of multiple pollutants.

[0004] In recent years, deep learning technology has demonstrated superior performance in time series prediction tasks. Among them, long short-term memory networks and Transformer structures have been widely used in pollutant concentration prediction. However, the standard Transformer has huge computational overhead when processing long time series data. The Informer network reduces the computational complexity through the sparse self-attention mechanism and improves the prediction ability of long sequence data. It has been applied to traffic forecasting, weather forecasting and other fields, but its application in air pollution prediction still has room for improvement, such as optimizing model training for sparse data and improving the accuracy of pollutant spatial interpolation.

[0005] In terms of pollution source control, traditional pollutant control measures mainly rely on fixed control strategies, such as setting industry emission standards, demarcating low-emission areas, and implementing odd-even number restrictions. Although these methods can reduce pollutant emissions to a certain extent, they lack dynamic adaptability and cannot flexibly adjust control plans according to real-time pollution conditions and meteorological conditions. In addition, the optimal control of a single pollution source often cannot take into account the coordinated control of multiple pollution sources. For example, the interaction between industrial pollution sources and motor vehicle emissions has not been fully considered. Therefore, it is difficult for existing methods to achieve global optimal control of pollution sources.

[0006] In recent years, multi-agent reinforcement learning has been widely used in distributed decision-making optimization tasks. For example, in the fields of intelligent traffic control, drone collaboration, and smart grid dispatching, multi-agent systems optimize overall benefits and improve decision-making efficiency through collaborative learning. However, in the field of pollution source control, research based on multi-agent reinforcement learning is still in its early stages and faces the following challenges: (1) The nonlinear dynamic characteristics of pollutant diffusion lead to a complex state space of agent decision-making; (2) The interaction between pollution sources makes the optimization of the benefit function non-convex, making it difficult to find the optimal solution; (3) How to design a reasonable reward mechanism so that the agent can optimize its own benefits while taking into account the overall pollution control goals. Most existing studies use a single reinforcement learning method, such as deep Q-network and proximal strategy optimization, but do not fully consider the game characteristics of pollution control, resulting in the strategy learned by the agent converging to a local optimal solution and failing to achieve the optimal strategy for global pollution control. Summary of the invention

[0007] One purpose of the present invention is to propose a method for dynamic distribution control of urban pollution sources based on reinforcement learning. The improved Informer network of the present invention performs spatiotemporal prediction of pollutant concentrations, and optimizes pollution control strategies based on game theory multi-agent reinforcement learning. The improved Informer network is used to process long time series data to improve the accuracy of pollutant concentration prediction, and the diffusion trend of pollutants is calculated in combination with the spatial interpolation method. A game theory multi-agent reinforcement learning environment is established, and the control strategy of the pollution source agent is optimized using Nash equilibrium. The reward mechanism is optimized in combination with pollution historical data, so that each agent can continuously learn and optimize in a dynamic environment, and finally converge to the global optimal strategy, thereby improving the executability and effectiveness of the pollution source control scheme.

[0008] A method for dynamic distribution control of urban pollution sources based on reinforcement learning according to an embodiment of the present invention includes the following steps: S1. Collect source data of air pollution, and perform data cleaning, format conversion and standardization on the source data to construct a pollution data set; S2. Use the improved Informer network to perform time series prediction on pollution data sets, calculate the future concentration trend of different pollutants, and combine the spatial interpolation method to calculate the diffusion of pollutants in different regions to generate pollutant spatiotemporal distribution prediction data; S3. Based on the predicted spatial and temporal distribution data of pollutants, a pollution source impact factor matrix is ​​constructed to calculate the impact of different pollution sources on air quality, and a pollution source control task data set is constructed in combination with the pollution source emissions and pollution diffusion characteristics; S4. Based on the pollution source control task data set, a game theory multi-agent reinforcement learning environment is established. The state space, action space and benefit function of the pollution source agent are set. The control strategy of the pollution source agent is optimized using Nash equilibrium. The reward mechanism is optimized based on pollution historical data. In combination with pollution control resource constraints, the pollution source control strategy is dynamically adjusted to generate the global optimal control strategy. S5. Combine the global optimal control strategy, calculate the pollution source adjustment plan, and optimize the pollution control parameters; S6. Execute the pollution control strategy and iteratively optimize the pollution control strategy.

[0009] Optionally, the source data includes air pollutant concentration data and meteorological data.

[0010] Optionally, the S2 specifically includes: S21. Construct a pollutant time series data matrix, process the time series data, and calculate the rate of change and acceleration of pollutant concentration: ; ; in, represents the pollutant time series data matrix, Indicates time Location The pollutant concentration at represents the total number of time steps, Represents the geographic location index, Indicates time Location The rate of change of pollutant concentration at Indicates time Location The acceleration of the pollutant concentration at Indicates time Location The pollutant concentration at Indicates time Location The pollutant concentration at represents the time interval between adjacent time steps, Indicates time Location The rate of change of pollutant concentration at the location; S22, set the input sequence length to , define the input matrix: ; in, Represents the input matrix, describing the input pollutant time series data, represents the input vector, Indicates time Location The pollutant concentration at Indicates time Location The rate of change of pollutant concentration at Indicates time Location The acceleration of the pollutant concentration at Indicates time Meteorological data matrix at ; S23, using the improved Informer network to predict pollutant time series, the input matrix Perform a linear transformation and map it to a high-dimensional embedding vector. The improved Informer network includes a data embedding layer, a sparse self-attention mechanism, and a decoder: ; in, represents a high-dimensional embedding vector, represents the input matrix, represents the embedding layer weight matrix, represents the bias term of the embedding layer; S24. Use the sparse self-attention mechanism to calculate the high-dimensional embedding vector, obtain the query, key, and value vectors, and calculate the attention score: ; in, represents the attention score, represents the query vector, represents the key vector, represents the dimension of the key vector, represents the indicator function, only when the attention score is greater than the threshold Only when it is retained, Indicates the threshold value; S25, use Sampling strategy, only retaining the most influential Attention scores are used to calculate the attention-weighted high-dimensional embedding vector: ; in, represents the high-dimensional embedding vector after attention weighting, Represents the attention score after filtering, only retaining the top The attention scores corresponding to important time steps, represents the residual connection coefficient, represents a value vector, Represents a sampling strategy that retains only the most influential Attention scores; S26, decoder based on improved Informer network, high-dimensional embedding vector after attention weighting Decoding is performed through reverse calculation to generate pollutant concentration predictions. The decoder includes a multi-head attention mechanism, a feedforward network, a residual connection, and a normalization layer. The feedforward network is a multi-layer perceptron: ; in, represents the predicted pollutant concentration series, Indicates time Location The predicted value of pollutant concentration at represents the prediction step length; S27. Combined prediction of pollutant concentration The spatial diffusion trend of pollutants is calculated using the spatial interpolation method, and the spatial diffusion equation of pollutants is defined as: ; in, represents the temporal variation of pollutant concentration, Indicates wind speed The convection and diffusion of pollutants under the influence of represents the diffusion coefficient, Indicates wind speed, represents the Laplace diffusion term of the pollutant concentration, Indicates the concentration of pollutants; S28, based on predicted pollutant concentrations And the pollutant spatial diffusion equation, calculate the concentration distribution of pollutants in different regions in the future, obtain the predicted values ​​of the temporal and spatial distribution of pollutants, and generate the predicted data of the temporal and spatial distribution of pollutants: ; in, Indicates time Location The predicted value of pollutant concentration at Indicates time Location The pollutant concentration at Indicates time Location The rate of change of pollutant concentration at represents the time step, represents the spatial diffusion gradient of pollutants, represents the spatial step length, represents the diffusion coefficient, represents the diffusion term, A Laplace diffusion term representing the pollutant concentration.

[0011] Optionally, the S4 specifically includes: S41. Based on the pollution source control task dataset, a reinforcement learning environment is established to construct a Markov decision process: ; in, represents the Markov decision process of the pollution source control task, represents the set of pollution source agents, Represents the state space, including pollutant concentration, pollution change rate and environmental carrying capacity, Represents the pollution source agent action space, adjust the strategy for pollution, represents the state transition probability, describing the impact of different adjustment strategies adopted by the pollution source agent on the pollutant concentration. represents the benefit function, which measures the effect of pollution control. represents the discount factor, which controls the weight of future returns; S42, setting the state space and action space of the pollution source agent, wherein the state space includes pollutant concentration, pollutant time change rate, pollutant acceleration, environmental carrying capacity and pollution source control weight, and the action space selects to adjust the emission; S43, constructing a benefit function of the pollution source agent, wherein the benefit function takes into account the pollutant concentration, the time change rate of the pollutant and the pollution source control weight: ; in, represents the benefit function of the pollution source agent, , and They represent the influence weights of pollutant concentration, pollutant time change rate and pollution source control weight respectively. represents the pollutant concentration, represents the time change rate of the pollutant, represents the pollution source control weight; S44. Generate pollution source control strategies based on reinforcement learning, use the policy gradient method to optimize the pollution source agent control strategy, and calculate the control strategy gradient: ; in, Represents the control policy gradient of the policy objective function of the pollution source agent, represents the strategic objective function of the pollution source agent, represents the mathematical expectation, represents the benefit function of the pollution source agent, represents the policy function, represents the parameters of the policy function, represents the gradient calculation, represents the action space of the pollution source agent, represents the state space of the pollution source agent, represents the time step; S45. Use Nash equilibrium to optimize the pollution source agent control strategy. When the strategies of other agents are fixed, calculate the optimal strategy of each pollution source agent: ; in, Represents the pollution source agent The optimal strategy, represents the policy function, represents the optimal strategy for all other agents, represents the benefit function of the pollution source agent, represents the discount factor; S46. Optimize the reward mechanism based on historical pollution data and construct a time-weighted reward function: ; in, represents the time-weighted reward function, represents the historical reward impact factor, represents the time decay factor, controlling the weight of historical pollution data, A variable representing the accumulation operation, represents the natural exponential function; S47. Combined with the resource constraints of pollution control, dynamically adjust the pollution source control strategy and optimize the pollution source control strategy based on the gradient descent method: ; in, represents the control strategy of the adjusted pollution source, represents the control strategy of pollution sources, Represents the learning rate, which controls the step size of the adjustment of the control strategy. represents the gradient of the time-weighted reward function with respect to the control policy; S48. Generate the global optimal control strategy for pollution sources based on the intelligent agent strategy trained by reinforcement learning: ; in, represents the global optimal control strategy set of all pollution source agents, Represents the number of pollution source agents.

[0012] Optionally, the S5 specifically includes: S51, based on the optimal strategy of each pollution source agent in the global optimal control strategy, generating a pollution source adjustment vector, wherein the pollution source adjustment vector includes an emission adjustment range, a control start and end time, and a regional execution weight; S52, constructing a pollution source control parameter optimization objective function, wherein the objective function takes minimizing the pollutant concentration prediction error, reducing the pollution diffusion intensity and improving the resource utilization efficiency as the optimization goals; S53, using Lagrange multiplier method to calculate the optimal solution of pollution source control parameters; S54. Generate a pollution source adjustment plan based on the calculated optimal solution, and determine the emission limit value, execution time period and regional control priority of each pollution source.

[0013] The beneficial effects of the present invention are: First, the improved Informer network proposed in the present invention can effectively overcome the limitations of traditional time series prediction methods, and use the sparse self-attention mechanism to reduce the computational complexity, so that it has stronger modeling capabilities when dealing with long-term pollutant concentration prediction tasks. In addition, by constructing a pollutant time series data matrix and calculating the pollutant concentration change rate and acceleration, the present invention can more comprehensively consider the temporal dynamic characteristics of pollutants when predicting future pollutant concentration trends. Combined with the spatial interpolation method, the present invention further makes up for the shortcomings of existing models in modeling the spatial diffusion of pollutants, making the prediction of the diffusion trend of pollutants in different regions more accurate, and providing reliable data support for subsequent pollution source control.

[0014] Secondly, in terms of pollution source control, the present invention adopts a game theory multi-agent reinforcement learning method, which has stronger adaptability and global optimization capabilities compared to traditional rule-based pollution control strategies. By constructing the state space, action space and benefit function of the pollution source agent, the present invention can enable the agent to learn autonomously in a complex pollution environment, and use Nash equilibrium to optimize the pollution source control strategy, so that each agent can optimize its own benefits without harming the overall environmental interests. Unlike the existing single-agent reinforcement learning method, the present invention is based on a game theory framework, which enables different pollution source agents to collaboratively optimize the control strategy, avoids local optimal problems, and improves the overall effect of pollution source control. In addition, the present invention combines pollution history data to optimize the reward mechanism, so that the agent can better adapt to different pollution evolution patterns, thereby improving the stability and generalization ability of the pollution control strategy.

[0015] Finally, the present invention also introduces pollution control resource constraints in the pollution source control process, so that the pollution control strategy finally generated has theoretical optimality. At the same time, the control strategy of the present invention can dynamically adapt to environmental changes. For example, it can strengthen pollution source control under unfavorable meteorological conditions (such as high humidity and low wind speed) and appropriately relax restrictions when the air self-purification capacity is strong, thereby maximizing the economic and environmental benefits of pollution control.

[0016] In summary, the present invention improves the accuracy of pollutant concentration prediction by introducing an improved Informer network, and optimizes the pollution source control strategy by using game theory multi-agent reinforcement learning, thereby overcoming the problems of insufficient pollution prediction accuracy and weak pollution control adaptability in the prior art. The present invention improves the spatiotemporal resolution of pollutant concentration prediction, and through continuous optimization of the reinforcement learning strategy, enables the pollution source control scheme to dynamically adapt to environmental changes, thereby realizing intelligent, autonomous and global optimization of pollution control. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a method for dynamic distribution control of urban pollution sources based on reinforcement learning proposed by the present invention; Figure 2 This is a schematic diagram of the improved Informer network structure of a reinforcement learning-based urban pollution source dynamic distribution control method proposed in the present invention. DETAILED DESCRIPTION

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0019] refer to Figure 1 and Figure 2 , a method for dynamic distribution control of urban pollution sources based on reinforcement learning, comprising the following steps: S1. Collect source data related to air pollution, and perform data cleaning, format conversion and standardization on the source data to construct a pollution data set; S2. Use the improved Informer network to perform time series prediction on pollution data sets, calculate the future concentration trend of different pollutants, and combine the spatial interpolation method to calculate the diffusion of pollutants in different regions to generate pollutant spatiotemporal distribution prediction data; S3. Based on the predicted spatial and temporal distribution data of pollutants, a pollution source impact factor matrix is ​​constructed to calculate the impact of different pollution sources on air quality, and a pollution source control task data set is constructed in combination with the pollution source emissions and pollution diffusion characteristics; S4. Based on the pollution source control task data set, a game theory multi-agent reinforcement learning environment is established. The state space, action space and benefit function of the pollution source agent are set. The control strategy of the pollution source agent is optimized using Nash equilibrium. The reward mechanism is optimized based on pollution historical data. In combination with pollution control resource constraints, the pollution source control strategy is dynamically adjusted to generate the global optimal control strategy. S5. Combine the global optimal control strategy, calculate the pollution source adjustment plan, and optimize the pollution control parameters; S6. Execute pollution control strategies and iteratively optimize pollution control plans.

[0020] In this implementation, the source data includes air pollutant concentration data and meteorological data.

[0021] In this implementation, S2 specifically includes: S21. Construct a pollutant time series data matrix, process the time series data, and calculate the rate of change and acceleration of pollutant concentration: ; ; in, represents the pollutant time series data matrix, Indicates time Location The pollutant concentration at represents the total number of time steps, Represents the geographic location index, Indicates time Location The rate of change of pollutant concentration at Indicates time Location The acceleration of the pollutant concentration at Indicates time Location The pollutant concentration at Indicates time Location The pollutant concentration at represents the time interval between adjacent time steps, Indicates time Location The rate of change of pollutant concentration at the location; S22, set the input sequence length to , define the input matrix: ; in, Represents the input matrix, describing the input pollutant time series data, represents the input vector, Indicates time Location The pollutant concentration at Indicates time Location The rate of change of pollutant concentration at Indicates time Location The acceleration of the pollutant concentration at Indicates time Meteorological data matrix at ; S23, using the improved Informer network to predict pollutant time series, the input matrix Perform a linear transformation and map it to a high-dimensional embedding vector. The improved Informer network includes a data embedding layer, a sparse self-attention mechanism, and a decoder: ; in, represents a high-dimensional embedding vector, represents the input matrix, represents the embedding layer weight matrix, represents the bias term of the embedding layer; S24. Use the sparse self-attention mechanism to calculate the high-dimensional embedding vector, obtain the query, key, and value vectors, and calculate the attention score: ; in, represents the attention score, represents the query vector, represents the key vector, represents the dimension of the key vector, represents the indicator function, only when the attention score is greater than the threshold Only when it is retained, Indicates the threshold value; S25, use Sampling strategy, only retaining the most influential Attention scores are used to calculate the attention-weighted high-dimensional embedding vector: ; in, represents the high-dimensional embedding vector after attention weighting, Represents the attention score after filtering, only retaining the top The attention scores corresponding to important time steps, represents the residual connection coefficient, represents a value vector, Represents a sampling strategy that retains only the most influential Attention scores; S26, decoder based on improved Informer network, high-dimensional embedding vector after attention weighting Decoding is performed through reverse calculation to generate pollutant concentration predictions. The decoder includes a multi-head attention mechanism, a feedforward network, a residual connection, and a normalization layer. The feedforward network is a multi-layer perceptron: ; in, represents the predicted pollutant concentration series, Indicates time Location The predicted value of pollutant concentration at represents the prediction step length; S27. Combined prediction of pollutant concentration The spatial diffusion trend of pollutants is calculated using the spatial interpolation method, and the spatial diffusion equation of pollutants is defined as: ; in, represents the temporal variation of pollutant concentration, Indicates wind speed The convection and diffusion of pollutants under the influence of represents the diffusion coefficient, Indicates wind speed, represents the Laplace diffusion term of the pollutant concentration, Indicates the concentration of pollutants; S28, based on predicted pollutant concentrations And the pollutant spatial diffusion equation, calculate the concentration distribution of pollutants in different regions in the future, obtain the predicted values ​​of the temporal and spatial distribution of pollutants, and generate the predicted data of the temporal and spatial distribution of pollutants: ; in, Indicates time Location The predicted value of pollutant concentration at Indicates time Location The pollutant concentration at Indicates time Location The rate of change of pollutant concentration at represents the time step, represents the spatial diffusion gradient of pollutants, represents the spatial step length, represents the diffusion coefficient, represents the diffusion term, A Laplace diffusion term representing the pollutant concentration.

[0022] In this implementation manner, the S4 specifically includes: S41. Based on the pollution source control task dataset, a reinforcement learning environment is established to construct a Markov decision process: ; in, represents the Markov decision process of the pollution source control task, represents the set of pollution source agents, Represents the state space, including pollutant concentration, pollution change rate and environmental carrying capacity, Represents the pollution source agent action space, adjust the strategy for pollution, represents the state transition probability, describing the impact of different adjustment strategies adopted by the pollution source agent on the pollutant concentration. represents the benefit function, which measures the effect of pollution control. represents the discount factor, which controls the weight of future returns; S42, setting the state space and action space of the pollution source agent, wherein the state space includes pollutant concentration, pollutant time change rate, pollutant acceleration, environmental carrying capacity and pollution source control weight, and the action space selects to adjust the emission; S43, constructing a benefit function of the pollution source agent, wherein the benefit function takes into account the pollutant concentration, the time change rate of the pollutant and the pollution source control weight: ; in, represents the benefit function of the pollution source agent, , and They represent the influence weights of pollutant concentration, pollutant time change rate and pollution source control weight respectively. represents the pollutant concentration, represents the time change rate of the pollutant, represents the pollution source control weight; S44. Generate pollution source control strategies based on reinforcement learning, use the policy gradient method to optimize the pollution source agent control strategy, and calculate the control strategy gradient: ; in, Represents the control policy gradient of the policy objective function of the pollution source agent, represents the strategic objective function of the pollution source agent, represents the mathematical expectation, represents the benefit function of the pollution source agent, represents the policy function, represents the parameters of the policy function, represents the gradient calculation, represents the action space of the pollution source agent, represents the state space of the pollution source agent, represents the time step; S45. Use Nash equilibrium to optimize the pollution source agent control strategy. When the strategies of other agents are fixed, calculate the optimal strategy of each pollution source agent: ; in, Represents the pollution source agent The optimal strategy, represents the policy function, represents the optimal strategy for all other agents, represents the benefit function of the pollution source agent, represents the discount factor; S46. Optimize the reward mechanism based on historical pollution data and construct a time-weighted reward function: ; in, represents the time-weighted reward function, represents the historical reward impact factor, represents the time decay factor, controlling the weight of historical pollution data, A variable representing the accumulation operation, represents the natural exponential function; S47. Combined with the resource constraints of pollution control, dynamically adjust the pollution source control strategy and optimize the pollution source control strategy based on the gradient descent method: ; in, represents the control strategy of the adjusted pollution source, represents the control strategy of pollution sources, Represents the learning rate, which controls the step size of the adjustment of the control strategy. represents the gradient of the time-weighted reward function with respect to the control policy; S48. Generate the global optimal control strategy for pollution sources based on the intelligent agent strategy trained by reinforcement learning: ; in, represents the global optimal control strategy set of all pollution source agents, Represents the number of pollution source agents.

[0023] In this implementation manner, S5 specifically includes: S51, based on the optimal strategy of each pollution source agent in the global optimal control strategy, generating a pollution source adjustment vector, wherein the pollution source adjustment vector includes an emission adjustment range, a control start and end time, and a regional execution weight; S52, constructing a pollution source control parameter optimization objective function, wherein the objective function takes minimizing the pollutant concentration prediction error, reducing the pollution diffusion intensity and improving the resource utilization efficiency as the optimization goals; S53, using Lagrange multiplier method to calculate the optimal solution of pollution source control parameters; S54. Generate a pollution source adjustment plan based on the calculated optimal solution, and determine the emission limit value, execution time period and regional control priority of each pollution source.

[0024] Embodiment 1: In order to verify the feasibility of the present invention in implementation, the present invention is applied to a typical urban pollution source control scenario, and a city with a high degree of industrialization is selected.

[0025] At the city's air pollution monitoring stations, multiple sensors are deployed to collect pollutant concentrations, meteorological data, and pollution source emission data, including PM2.5, PM10, , 、CO、 The concentration data of major pollutants such as wind speed and wind direction are collected by ground monitoring stations and updated every hour. Meteorological data including wind speed, wind direction, temperature, humidity, etc. are provided by local meteorological stations. Pollution source emission data are obtained through online monitoring systems of industrial enterprises and traffic flow monitoring systems. After data collection is completed, the data is first cleaned, formatted and standardized to construct a pollution data set, and then input into the improved Informer network for spatiotemporal prediction of pollutant concentrations.

[0026] In practical applications, the present invention uses an improved Informer network to predict the concentration of major pollutants in the city, and combines the spatial interpolation method to calculate the diffusion of pollutants in different areas. In terms of pollution source control, based on the pollution source control task data set, a game theory multi-agent reinforcement learning environment is established, and the state space, action space and benefit function of the pollution source agent are set. In the experiment, the agents represent different pollution sources in the city, including industrial enterprises, motor vehicles, construction dust, etc., and a reinforcement learning algorithm is used to optimize the pollution source emission strategy. The agent gradually reduces pollutant emissions and optimizes the overall air quality through interactive learning with the environment.

[0027] In addition, in order to evaluate the feasibility of pollution source control solutions, the present invention compares the implementation effects of different pollution control strategies.

[0028] Table 1 Experimental comparison data table

[0029] It can be seen from the above experimental data comparison table that the present invention has significant advantages in both pollutant prediction and pollution source control. First, in terms of pollutant concentration prediction, after using the improved Informer network, PM2.5, and The prediction errors (MSE) of PM2.5 were significantly reduced, among which the prediction error of PM2.5 decreased by 36.7% compared with the traditional method. The forecast error decreased by 34.9%. The prediction error decreased by 34.2%. This result shows that compared with the traditional time series prediction method, the Informer network of the present invention uses the sparse self-attention mechanism and The sampling strategy improves the modeling capability of long time series data and optimizes the prediction accuracy of pollutant concentrations.

[0030] In terms of pollution source control, the present invention uses game theory multi-agent reinforcement learning to optimize pollution emissions, which greatly increases the reduction in pollutant concentration. Compared with traditional pollution control methods, the PM2.5 reduction rate is increased from 9.8% to 22.7%, an increase of 131.6%; The decline rate increased from 7.5% to 17.4%, an increase of 132.0%; The decline rate increased from 5.9% to 13.9%, an increase of 135.6%. This shows that in the process of pollutant control, reinforcement learning methods can effectively optimize the emission strategies of pollution sources, making pollution control more precise and intelligent, and able to adapt to different meteorological conditions and pollution evolution trends.

[0031] In terms of the treatment effect of heavy pollution weather, when using traditional pollution control methods, the PM2.5 reduction rate is 9.8%, while the intelligent control strategy of the present invention makes the PM2.5 reduction rate reach 19.3%, an increase of 96.9%. This shows that in extremely polluted weather, traditional methods may be difficult to achieve effective pollution control, while the present invention optimizes the pollution source emission strategy through reinforcement learning, so that the pollution control plan has stronger dynamic adaptability and global optimality, which can further reduce pollutant concentrations and improve air quality during critical periods.

[0032] In summary, the present invention uses an improved Informer network to accurately predict pollutant concentrations, and combines game theory multi-agent reinforcement learning to optimize pollution source emission strategies, effectively overcoming the problems of low accuracy of traditional pollution predictions and poor adaptability of pollution control, and achieving more accurate pollution predictions and more efficient pollution control strategies.

[0033] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for dynamic distribution control of urban pollution sources based on reinforcement learning, characterized in that: The steps include: S1. Collect source data of air pollution, and perform data cleaning, format conversion and standardization on the source data to construct a pollution data set; S2. Use the improved Informer network to perform time series prediction on pollution data sets, calculate the future concentration trend of different pollutants, and combine the spatial interpolation method to calculate the diffusion of pollutants in different regions to generate pollutant spatiotemporal distribution prediction data; S3. Based on the predicted spatial and temporal distribution data of pollutants, a pollution source impact factor matrix is ​​constructed to calculate the impact of different pollution sources on air quality, and a pollution source control task data set is constructed in combination with the pollution source emissions and pollution diffusion characteristics; S4. Based on the pollution source control task data set, a game theory multi-agent reinforcement learning environment is established. The state space, action space and benefit function of the pollution source agent are set. The control strategy of the pollution source agent is optimized using Nash equilibrium. The reward mechanism is optimized based on pollution historical data. In combination with pollution control resource constraints, the pollution source control strategy is dynamically adjusted to generate the global optimal control strategy. S5. Combine the global optimal control strategy, calculate the pollution source adjustment plan, and optimize the pollution control parameters; S6. Execute the pollution control strategy and iteratively optimize the pollution control strategy.

2. According to claim 1, a method for dynamic distribution control of urban pollution sources based on reinforcement learning is characterized in that: The source data include air pollutant concentration data and meteorological data.

3. According to claim 1, a method for dynamic distribution control of urban pollution sources based on reinforcement learning is characterized in that: The S2 specifically includes: S21. Construct a pollutant time series data matrix, process the time series data, and calculate the rate of change and acceleration of pollutant concentration: ； ； in, represents the pollutant time series data matrix, Indicates time Location The pollutant concentration at represents the total number of time steps, Represents the geographic location index, Indicates time Location The rate of change of pollutant concentration at Indicates time Location The acceleration of the pollutant concentration at Indicates time Location The pollutant concentration at Indicates time Location The pollutant concentration at represents the time interval between adjacent time steps, Indicates time Location The rate of change of pollutant concentration at the location; S22, set the input sequence length to , define the input matrix: ； in, Represents the input matrix, describing the input pollutant time series data, represents the input vector, Indicates time Location The pollutant concentration at Indicates time Location The rate of change of pollutant concentration at Indicates time Location The acceleration of the pollutant concentration at Indicates time Meteorological data matrix at ; S23, using the improved Informer network to predict pollutant time series, the input matrix Perform a linear transformation and map it to a high-dimensional embedding vector. The improved Informer network includes a data embedding layer, a sparse self-attention mechanism, and a decoder: ； in, represents a high-dimensional embedding vector, represents the input matrix, represents the embedding layer weight matrix, represents the bias term of the embedding layer; S24. Use the sparse self-attention mechanism to calculate the high-dimensional embedding vector, obtain the query, key, and value vectors, and calculate the attention score: ； in, represents the attention score, represents the query vector, represents the key vector, represents the dimension of the key vector, represents the indicator function, only when the attention score is greater than the threshold Only when it is retained, Indicates the threshold value; S25, use Sampling strategy, only retaining the most influential Attention scores are used to calculate the attention-weighted high-dimensional embedding vector: ； in, represents the high-dimensional embedding vector after attention weighting, Represents the attention score after filtering, only retaining the top The attention scores corresponding to important time steps, represents the residual connection coefficient, represents a value vector, Represents a sampling strategy that retains only the most influential Attention scores; S26, decoder based on improved Informer network, high-dimensional embedding vector after attention weighting Decoding is performed through reverse calculation to generate pollutant concentration predictions. The decoder includes a multi-head attention mechanism, a feedforward network, a residual connection, and a normalization layer. The feedforward network is a multi-layer perceptron: ； in, represents the predicted pollutant concentration series, Indicates time Location The predicted value of pollutant concentration at represents the prediction step length; S27. Combined prediction of pollutant concentration The spatial diffusion trend of pollutants is calculated using the spatial interpolation method, and the spatial diffusion equation of pollutants is defined as: ； in, represents the temporal variation of pollutant concentration, Indicates wind speed The convection and diffusion of pollutants under the influence of represents the diffusion coefficient, Indicates wind speed, represents the Laplace diffusion term of the pollutant concentration, Indicates the concentration of pollutants; S28, based on predicted pollutant concentrations And the pollutant spatial diffusion equation, calculate the concentration distribution of pollutants in different regions in the future, obtain the predicted values ​​of the temporal and spatial distribution of pollutants, and generate the predicted data of the temporal and spatial distribution of pollutants: ； in, Indicates time Location The predicted value of pollutant concentration at Indicates time Location The pollutant concentration at Indicates time Location The rate of change of pollutant concentration at represents the time step, represents the spatial diffusion gradient of pollutants, represents the spatial step length, represents the diffusion coefficient, represents the diffusion term, A Laplace diffusion term representing the pollutant concentration.

4. According to claim 1, a method for dynamic distribution control of urban pollution sources based on reinforcement learning is characterized in that: The S4 specifically includes: S41. Based on the pollution source control task dataset, a reinforcement learning environment is established to construct a Markov decision process: ； in, represents the Markov decision process of the pollution source control task, represents the set of pollution source agents, Represents the state space, including pollutant concentration, pollution change rate and environmental carrying capacity, Represents the pollution source agent action space, adjust the strategy for pollution, represents the state transition probability, describing the impact of different adjustment strategies adopted by the pollution source agent on the pollutant concentration. represents the benefit function, which measures the effect of pollution control. represents the discount factor, which controls the weight of future returns; S42, setting the state space and action space of the pollution source agent, wherein the state space includes pollutant concentration, pollutant time change rate, pollutant acceleration, environmental carrying capacity and pollution source control weight, and the action space selects to adjust the emission; S43, constructing a benefit function of the pollution source agent, wherein the benefit function takes into account the pollutant concentration, the time change rate of the pollutant and the pollution source control weight: ； in, represents the benefit function of the pollution source agent, , and They represent the influence weights of pollutant concentration, pollutant time change rate and pollution source control weight respectively. represents the pollutant concentration, represents the time change rate of the pollutant, represents the pollution source control weight; S44. Generate pollution source control strategies based on reinforcement learning, use the policy gradient method to optimize the pollution source agent control strategy, and calculate the control strategy gradient: ； in, Represents the control policy gradient of the policy objective function of the pollution source agent, represents the strategic objective function of the pollution source agent, represents the mathematical expectation, represents the benefit function of the pollution source agent, represents the policy function, represents the parameters of the policy function, represents the gradient calculation, represents the action space of the pollution source agent, represents the state space of the pollution source agent, represents the time step; S45. Use Nash equilibrium to optimize the pollution source agent control strategy. When the strategies of other agents are fixed, calculate the optimal strategy of each pollution source agent: ； in, Represents the pollution source agent The optimal strategy, represents the policy function, represents the optimal strategy for all other agents, represents the benefit function of the pollution source agent, represents the discount factor; S46. Optimize the reward mechanism based on historical pollution data and construct a time-weighted reward function: ； in, represents the time-weighted reward function, represents the historical reward impact factor, represents the time decay factor, controlling the weight of historical pollution data, A variable representing the accumulation operation, represents the natural exponential function; S47. Combined with the resource constraints of pollution control, dynamically adjust the pollution source control strategy and optimize the pollution source control strategy based on the gradient descent method: ； in, represents the control strategy of the adjusted pollution source, represents the control strategy of pollution sources, Represents the learning rate, which controls the step size of the adjustment of the control strategy. represents the gradient of the time-weighted reward function with respect to the control policy; S48. Generate the global optimal control strategy for pollution sources based on the intelligent agent strategy trained by reinforcement learning: ； in, represents the global optimal control strategy set of all pollution source agents, Represents the number of pollution source agents.

5. According to claim 1, a method for dynamic distribution control of urban pollution sources based on reinforcement learning is characterized in that: The S5 specifically includes: S51, based on the optimal strategy of each pollution source agent in the global optimal control strategy, generating a pollution source adjustment vector, wherein the pollution source adjustment vector includes an emission adjustment range, a control start and end time, and a regional execution weight; S52, constructing a pollution source control parameter optimization objective function, wherein the objective function takes minimizing the pollutant concentration prediction error, reducing the pollution diffusion intensity and improving the resource utilization efficiency as the optimization goals; S53, using Lagrange multiplier method to calculate the optimal solution of pollution source control parameters; S54. Generate a pollution source adjustment plan based on the calculated optimal solution, and determine the emission limit value, execution time period and regional control priority of each pollution source.

Images