A water resource demand optimization method and system based on deep reinforcement learning

Abstract

The application discloses a water resource demand optimization method based on deep reinforcement learning, and relates to the technical field of water resource management, comprising the following steps: acquiring original data and real-time data; preprocessing the original data; extracting parameters of extreme climate events according to the original data, and generating extreme scene data of a plurality of disturbance scenes based on a pre-trained environment simulator according to historical climate data and the parameters of the extreme climate events; training a preset deep reinforcement learning model based on DQN according to an optimization training set, wherein the network architecture of the deep reinforcement learning model based on DQN is CNN; optimizing the deep reinforcement learning model based on DQN in a multi-objective reinforcement learning strategy to obtain a trained deep reinforcement learning model based on DQN; inputting real-time climate data into the trained deep reinforcement learning model based on DQN; and outputting real-time optimized regional water resource demand data. The application significantly improves the decision efficiency and precision of water resource demand optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of water resource management, and particularly relates to a water resource demand optimization method and system based on deep reinforcement learning. BACKGROUND

[0002] With the increasing global climate change and water resource demand, water resource management is facing unprecedented pressure. Traditional water resource allocation methods rely on static models and manual adjustment, which cannot effectively respond to complex and dynamic environmental conditions. Especially in the face of sudden climate change, seasonal demand fluctuations and other factors, traditional methods often cannot make timely optimization decisions, leading to waste or shortage of water resources, affecting the sustainable development of ecology, agriculture and cities. In order to cope with these challenges, in recent years, water resource management methods combining deep reinforcement learning (DRL) and multi-objective optimization technology have gradually attracted attention.

[0003] Most existing water resource management methods use experience-based rules and historical data models, which have certain effects in some stable environments, but when facing complex and variable environments, these methods are difficult to cope with changing water resource demand. For example, precipitation fluctuations caused by climate change, seasonal changes in agricultural water consumption and other factors will affect the allocation of water resources.

[0004] Therefore, the applicant has developed a water resource demand optimization method based on deep reinforcement learning to solve the above problems. SUMMARY

[0005] The present application provides a water resource demand optimization method based on deep reinforcement learning to solve the problem that existing water resource management methods are difficult to cope with changing water resource demand when facing complex and variable environments.

[0006] The present application achieves the above-mentioned purposes through the following technical solutions:

[0007] The present application provides a water resource demand optimization method based on deep reinforcement learning, comprising:

[0008] Obtaining original data and real-time data, the original data including collecting historical climate data, historical regional water resource demand data, the real-time data including real-time climate data;

[0009] Preprocessing the original data to obtain preprocessed data, and performing data labeling on the preprocessed data to obtain a training data set;

[0010] According to the original data, parameters of extreme climate events are extracted, and according to the historical climate data and the parameters of extreme climate events, extreme scene data of a plurality of disturbance scenes is generated based on a pre-trained environment simulator, the environment simulator being used to simulate the change process of other related environmental parameters according to the input parameters of extreme events and the historical climate data;

[0011] The extreme scene data is randomly inserted into the training data set to obtain an optimized training set;

[0012] A pre-set deep reinforcement learning model based on DQN is trained according to the optimized training set, and the network architecture of the deep reinforcement learning model based on DQN is CNN;

[0013] The deep reinforcement learning model based on DQN is optimized based on a multi-objective reinforcement learning strategy to obtain a trained deep reinforcement learning model based on DQN, and the reward function of the multi-objective reinforcement learning strategy includes a water resource reward function, an irrigation efficiency reward function and an ecological demand reward function;

[0014] The real-time climate data is input into the trained deep reinforcement learning model based on DQN, and real-time optimized regional water resource demand data is output.

[0015] Further, the environment simulator includes an input layer, an intermediate layer and an output layer, the input of the input layer is a feature vector with a length of 260, the intermediate layer includes a fully connected layer containing 260 hidden units, a ReLU activation function layer, a fully connected layer containing 64 hidden units, a ReLU activation function layer, a fully connected layer containing 256 hidden units and a ReLU activation function layer arranged in sequence, and the output layer is a fully connected layer.

[0016] Further, the loss function of the pre-training process of the environment simulator is:

[0017] ;

[0018] Wherein, 、 、 、 are generated soil moisture, humidity, wind speed and water demand, 、 、 、 are the real recorded soil moisture, humidity, wind speed and water demand, N is the total number of detection points, and i is the serial number of the detection point.

[0019] Further, the reward function is:

[0020] ;

[0021] ;

[0022] ;

[0023] ;

[0024] wherein, is the reward of water saving, is the current water consumption of the region, is the reward of irrigation efficiency, is the yield of crops, is the amount of water used for irrigation, is the reward of ecological demand, is the water level of the ecological region, is the minimum water level required by the ecological region, w1, w2, w3 represent the weights of the three objectives of water saving, irrigation efficiency and ecological demand, respectively.

[0025] Further, it also includes handling the conflict between the three objectives based on the Pareto optimization method, and the Pareto optimization process is represented as:

[0026] .

[0027] Further, after the weighted reward mechanism and Pareto optimization, the total reward value is passed into the Q value function:

[0028] ;

[0029] wherein, s t represents the current environmental state, including soil humidity, air humidity, wind speed, current water resource reserves, crop growth stage, current irrigation demand; a t represents the current water resource management decision, including irrigation water allocation, ecological flow regulation, regional water resource scheduling; s t+1 represents the next moment of environmental state, including soil humidity, air humidity, wind speed, current water resource reserves, crop growth stage, current irrigation demand; a' represents the next moment of water resource management decision, including irrigation water allocation, ecological flow regulation, regional water resource scheduling; R total is the total reward calculated by the weighted reward value and Pareto optimization after considering water saving, irrigation efficiency and ecological demand, Q old is the current Q value, representing the value under the state s t and action a t , is the updated Q value, representing the evaluation of water resource allocation after the optimization of the new strategy, is a discount factor representing the importance of future rewards, and alpha is a learning rate that controls the influence of new experiences on model updates.

[0030] Further, the original data is preprocessed, including:

[0031] The original environmental data collected is denoised;

[0032] The data after denoising is processed for filtering;

[0033] The data after filtering is standardized or normalized;

[0034] The data after standardization or normalization is processed using linear interpolation and mean filling method to obtain preprocessed data.

[0035] The application also provides a water resource demand optimization system based on deep reinforcement learning, comprising:

[0036] An acquisition module is configured to acquire original data and real-time data, wherein the original data includes collected historical climate data and historical regional water resource demand data, and the real-time data includes real-time climate data;

[0037] A preprocessing module is configured to preprocess the original data to obtain preprocessed data, and to label the preprocessed data to obtain a training data set;

[0038] A simulation module is configured to extract parameters of extreme climate events according to the original data, and to generate extreme scenario data of a plurality of disturbance scenarios based on a pre-trained environmental simulator according to the historical climate data and the parameters of extreme climate events, wherein the environmental simulator is configured to simulate the change process of other related environmental parameters according to the input parameters of extreme events and historical climate data;

[0039] An insertion module is configured to randomly insert the extreme scenario data into the training data set to obtain an optimized training set;

[0040] A training module is configured to train a pre-set deep reinforcement learning model based on DQN according to the optimized training set, wherein the network architecture of the deep reinforcement learning model based on DQN is CNN;

[0041] An optimization module is configured to optimize the deep reinforcement learning model based on DQN based on a multi-objective reinforcement learning strategy to obtain a trained deep reinforcement learning model based on DQN, wherein the reward function of the multi-objective reinforcement learning strategy includes a water resource reward function, an irrigation efficiency reward function and an ecological demand reward function;

[0042] an output module for inputting the real-time climate data into the trained deep Q-network (DQN) based deep reinforcement learning model and outputting real-time optimized regional water resource demand data.

[0043] The present application has the following beneficial effects:

[0044] The water resource demand optimization method based on deep reinforcement learning significantly improves the decision efficiency and accuracy of water resource demand optimization, especially in the face of complex and variable environmental conditions, it exhibits excellent robustness and adaptability, can monitor and optimize water resource allocation scheme in real time, effectively cope with climate change, agricultural demand fluctuations and water shortage and other uncertain factors, so as to realize efficient utilization and sustainable management of water resources. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A flowchart of the water resource demand optimization method based on deep reinforcement learning.

[0046] Figure 2 A data preprocessing flowchart in the embodiment of the present application.

[0047] Figure 3 A deep reinforcement learning model design and training flowchart in the embodiment of the present application.

[0048] Figure 4 An environmental adaptability training flowchart in the embodiment of the present application.

[0049] Figure 5 A system feedback and incremental learning flowchart in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0052] It should be noted that similar reference numerals and letters refer to like items throughout the accompanying drawings, and once an item is defined in one drawing, it is not necessary to further define and explain it in subsequent drawings.

[0053] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "outer", "left", "right", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship commonly placed when the product of the application is used, or the orientation or positional relationship commonly understood by those skilled in the art, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application.

[0054] In addition, the terms "first", "second", and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0055] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "provided", "connected" and the like should be broadly understood, for example, "connected" can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, or electrically connected; can be directly connected, or indirectly connected through an intermediate medium, or the internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0056] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0057] The application will be further described below with reference to the accompanying drawings and examples.

[0058] As Figure 1 shown, a flow chart of a water resource demand optimization method based on deep reinforcement learning of the present application, the steps are as follows:

[0059] S1: obtaining original data and real-time data, the original data including collecting historical climate data, historical regional water resource demand data, the real-time data including real-time climate data;

[0060] S2: pre-processing the original data to obtain pre-processed data, and data labeling the pre-processed data to obtain a training data set;

[0061] S3: extracting parameters of extreme climate events according to the original data, and generating extreme scenario data of a plurality of perturbation scenarios based on a pre-trained environment simulator according to the historical climate data and the parameters of the extreme climate events, the environment simulator being used to simulate a variation process of other related environmental parameters according to the input parameters of the extreme events and the historical climate data;

[0062] S4: inserting the extreme scenario data into the training data set randomly to obtain an optimized training set;

[0063] S5: training a preset deep reinforcement learning model based on DQN according to the optimized training set, a network architecture of the deep reinforcement learning model based on DQN being CNN;

[0064] S6: optimizing the deep reinforcement learning model based on DQN based on a multi-objective reinforcement learning strategy to obtain a trained deep reinforcement learning model based on DQN, a reward function of the multi-objective reinforcement learning strategy including a water resource reward function, an irrigation efficiency reward function and an ecological demand reward function;

[0065] S7: inputting the real-time climate data into the trained deep reinforcement learning model based on DQN to output real-time optimized regional water resource demand data.

[0066] The application also provides a water resource demand optimization system based on deep reinforcement learning, comprising:

[0067] an acquisition module, which is used to acquire original data and real-time data, the original data including collected historical climate data and historical regional water resource demand data, and the real-time data including real-time climate data;

[0068] a preprocessing module, which is used to preprocess the original data to obtain preprocessed data, and perform data labeling on the preprocessed data to obtain a training data set;

[0069] a simulation module, which is used to extract parameters of extreme climate events according to the original data, and generate extreme scenario data of a plurality of perturbation scenarios based on a pre-trained environment simulator according to the historical climate data and the parameters of the extreme climate events, the environment simulator being used to simulate a variation process of other related environmental parameters according to the input parameters of the extreme events and the historical climate data;

[0070] an insertion module, which is used to insert the extreme scenario data into the training data set randomly to obtain an optimized training set;

[0071] a training module configured to train a preset deep reinforcement learning model based on DQN according to the optimized training set, a network architecture of the deep reinforcement learning model based on DQN being CNN;

[0072] an optimization module configured to optimize the deep reinforcement learning model based on DQN based on a multi-objective reinforcement learning strategy to obtain a trained deep reinforcement learning model based on DQN, a reward function of the multi-objective reinforcement learning strategy including a water resource reward function, an irrigation efficiency reward function and an ecological demand reward function;

[0073] an output module configured to input the real-time climate data into the trained deep reinforcement learning model based on DQN to output real-time optimized regional water resource demand data.

[0074] The specific operation steps are as follows:

[0075] Step 1: The system optimizes data collection frequency by prioritizing different regions. For high-priority areas such as agricultural irrigation zones and ecological protection zones, the system maintains a higher collection frequency through a dynamic tiering mechanism, typically set to every 5 minutes. These areas are crucial for water resource management, so frequent data collection is necessary to ensure timely access to weather data (temperature, humidity, precipitation, wind speed), soil moisture, water demand, and other information. Through this high-frequency collection, the system can accurately monitor water resource demand in the region, especially during sudden weather events (such as heavy rain or drought), allowing for rapid response and timely adjustment of water resource management strategies to prevent waste and ensure reasonable allocation of water resources. For low-priority areas, the system dynamically adjusts the collection frequency based on environmental changes, typically set to every 15 minutes. Low-priority areas such as general urban water use areas can have a relatively lower data collection frequency, as these areas have more stable environmental changes. In these areas, the main data collected includes weather data, water demand, and water source reserves, which help assess long-term water resource demand and assist the system in future water source prediction. When the system detects sudden weather events (such as heavy rain, drought, or high temperature), the collection frequency for related areas will automatically increase, typically increasing to every 1 minute. This is to capture data changes caused by sudden weather events, such as frequent monitoring of precipitation and soil moisture during heavy rain, and increasing soil moisture and temperature collection frequency during drought, ensuring rapid scheduling under extreme weather conditions to avoid resource waste. In some extreme weather conditions or when sensors fail, lose signal, or other reasons, the system will automatically call on drones for supplementary monitoring. Drones can be quickly deployed and can collect data that ground sensors cannot, especially in complex terrain or weather conditions, ensuring the continuity and integrity of data collection. For example, when weather sensor data is abnormal (such as sensor damage or instability), drones will fly to the relevant area to collect high-precision weather data (such as temperature, humidity, precipitation, etc.). When soil moisture sensors cannot work due to changes in soil conditions, the system will send drones to measure soil moisture accurately, ensuring the reliability of soil moisture data in that area. In addition, in extreme weather conditions, drones can also help collect data from remote areas that are difficult to obtain through ground sensors, especially during extreme weather such as heavy rain, drought, and cold waves. The supplementary monitoring of drones is not limited to sensor failure situations, but also plays a key role in extreme weather or hard-to-reach areas, ensuring the comprehensiveness and accuracy of data collection. All data collected by sensors and drones will be transmitted in real time to the data center through a wireless transmission system. The system uses high-performance wireless transmission devices such as NB-IoT modules to ensure stable and efficient data transmission, with a data transmission distance of more than 10 kilometers, ensuring monitoring of a large area.In addition, the system sets a reasonable data collection interval, typically 15 minutes, and ensures data safety and reliability through redundant storage and backup mechanisms. Data storage uses high-capacity hard disk devices such as 5TB Seagate Exos X14 hard disks to ensure large-scale data storage needs. In terms of data storage, standard JSON format is used for data storage to ensure data compatibility and scalability for subsequent data analysis and model training.

[0076] Step 2: As shown in Figure 2 , in this step, the main operation is to preprocess the collected raw environmental data to ensure data quality, accuracy and consistency, providing reliable data support for subsequent model training. The specific process includes denoising, standardization or normalization, missing value processing, data labeling and data division, each step making necessary preparations for the input of deep reinforcement learning models.

[0077] Step 2.1: Denoising the collected raw environmental data to eliminate noise data interference and ensure data accuracy and consistency. In this implementation, a median filter-based denoising method is used to denoise temperature and humidity sensor, precipitation sensor and other data. A 3x3 window size median filter is selected for processing, which can effectively remove random noise while preserving the effective information in the signal. After denoising, the Pandas library in Python is used to process the data, and the scipy.signal.medfilt function is applied to filter the output data of each sensor.

[0078] After denoising, in order to ensure that the data adapt to the input requirements of the deep reinforcement learning model, the data is standardized or normalized. Specifically, the standardization method is used to transform the data to zero mean and unit variance to adapt to the training requirements of the neural network model. For each data set, calculate the mean and standard deviation, and use the formula:

[0079] ;

[0080] where X is the original data, μ is the mean of the data, and σ is the standard deviation of the data. This operation can make the distribution of input data more balanced, avoiding the influence of data scale difference on model training.

[0081] Step 2.2: Dealing with missing values. Dealing with missing values is an important step in data preprocessing. Since environmental sensors may lose data due to device failure, network problems, or external interference in some cases, it is necessary to handle missing values to ensure the integrity and consistency of the dataset. For missing values, we use two commonly used methods: interpolation and mean filling. First, we use linear interpolation to handle missing data from temperature and humidity sensors, precipitation sensors, and other sensors. Specifically, we use the pandas.DataFrame.interpolate function to fill in missing values according to the time sequence of the data. In cases where the data volume is large and the change is relatively flat, linear interpolation can better restore the trend of the data. For soil moisture sensors and other sensors that may lose data for a long time, we use the mean filling method to fill in missing values. By calculating the mean value of the data column, we fill in the missing values with the mean value to ensure the integrity of the dataset. We use the pandas.DataFrame.fillna function to perform mean filling operations to ensure that each data point has a valid value.

[0082] Step 2.3: Data labeling. In the model training of deep reinforcement learning, data labeling is to provide supervision information for subsequent training. According to the actual situation of water resource demand and the demand of the target task, the data is labeled to mark the part of water resource demand that needs to be optimized. The labeling process is based on different water resource allocation strategies and monitoring targets, combined with meteorological data, geographical data, and historical water resource usage. First, by analyzing meteorological data and hydrological data, we determine the key features of water resource demand. For example, according to different climate conditions (such as dry or humid environments) and crop irrigation needs in different regions, we label the water resource demand level in different regions. In the labeling process, the label of each dataset item is: water resource demand level (such as from 0 to 1 representing demand from low to high).

[0083] Step 2.4: Data division. Data division is to ensure the effectiveness of model training and avoid overfitting. The processed dataset is divided into training set, validation set and test set according to certain proportion. Generally, the data division ratio is 70% for training set, 15% for validation set and 15% for test set. Ensure that each subset can represent the overall characteristics of the dataset. In this process, a data shuffling mechanism is used to ensure the randomness of the division result. For different categories of data, ensure that the proportion of each category in the training set, validation set and test set is relatively balanced, to prevent the influence of class imbalance on model performance. After dividing the dataset, cross-validation method is used to evaluate the performance of the model. The cross-validation process further divides the training set into multiple subsets for model training and evaluation, avoiding model bias caused by unreasonable data division, and ensuring the generalization ability of the model.

[0084] Step 3: Enhance the model's extreme scenario adaptability by embedding a lightweight environmental simulator.

[0085] Step 3.1: The system sets artificial-defined extreme event parameters based on historical climate data and regional water resource demand. For example, set the following extreme climate events: extreme high temperature event: when the temperature is higher than 38℃ for consecutive days, the system considers it an extreme high temperature event, affecting water resource demand and soil moisture. Drought event: the system sets 10 consecutive days of zero precipitation, which is a drought state. Rainstorm event: the system sets the precipitation within 24 hours to be greater than 150mm as a rainstorm, which greatly affects water sources.

[0086] Step 3.2: The system generates multiple perturbation scenarios using a lightweight environmental simulator by combining historical climate data and set extreme event parameters. The simulator simulates the change process of other related environmental parameters (such as soil moisture, wind speed, water demand, and water level) according to the input artificial-defined extreme event temperature and precipitation, combined with the initial values of other meteorological data.

[0087] First, use the data processed in step 2 to train an environmental simulator, whose network structure is as follows:

[0088] Input layer: feature vector of length 260;

[0089] Intermediate layer: fully connected layer (260), ReLU, fully connected layer (64), ReLU, fully connected layer (256), ReLU;

[0090] Output layer: fully connected layer (384)

[0091] The network inputs the temperature and precipitation data of 128 consecutive detection points, as well as the initial values of humidity, wind speed, and soil moisture, to generate specific values of humidity, wind speed, and soil moisture at 128 detection points, and its loss function is as follows:

[0092] ;

[0093] where, , , , are the generated soil moisture, humidity, wind speed, and water demand, , , , To record the actual soil moisture, humidity, wind speed, and water demand. By minimizing this loss function, the system can optimize the generated perturbation scenarios to better match the actual extreme weather conditions, providing real and comprehensive training data for the deep reinforcement learning model, N is the total number of detection points, and i is the serial number of the detection point.

[0094] Step 4: The generated extreme scenario data is manually annotated for water resource demand, and the generated extreme perturbation scenarios are randomly inserted into the training dataset. By alternating the use of these extreme environmental data with normal environmental data, the system ensures that the deep reinforcement learning model can learn to cope with both normal and extreme weather conditions during the training process.

[0095] Step 5: As shown in Figure 3 , design a deep reinforcement learning model and train it using the processed dataset. The entire model design and optimization process is as follows:

[0096] Based on the input types mentioned above, a convolutional neural network (CNN) is chosen as the deep reinforcement learning model. Set DQN as the reinforcement learning framework, combined with convolutional neural network (CNN) for data feature extraction and decision optimization, to process meteorological and water resource data, including:

[0097] Meteorological data: temperature, humidity, precipitation, wind speed, etc.

[0098] Soil moisture: the amount of water in the soil

[0099] Water level data: used to reflect the water level of the current water source

[0100] Water demand: the demand for agricultural, industrial, and urban water

[0101] The input data will be standardized and passed into the network. Each time step input will contain all the above data and be processed by the convolutional neural network (CNN). The convolutional neural network (CNN) is used to extract the spatial features of meteorological data (temperature, humidity, precipitation, wind speed, soil moisture, and water demand), helping the model understand the climate patterns between different regions. It consists of multiple convolutional layers and pooling layers, which extract local features in meteorological data :

[0102] ;

[0103] where, is the input meteorological data, and are the weights and biases of the convolutional layer, respectively.

[0104] The DQN model is trained using the Q-learning algorithm combined with the policy gradient method to optimize water resource allocation schemes. During the training process, if the model does not converge, the learning rate, batch size, and other hyperparameters are adjusted and the training continues until the model converges, entering the model evaluation stage: calculating the model's performance on water resource optimization scheduling tasks, including: prediction accuracy, water resource allocation efficiency, and adaptability to extreme climate conditions. After the model converges, the objective function is optimized to improve the adaptability of water resource scheduling in different environments.

[0105] Step 6: In water resource management, the system adopts a multi-objective reinforcement learning strategy to optimize multiple objectives (such as saving water resources, improving irrigation efficiency, and ensuring ecological needs). To balance these objectives, the system uses a weighted reward mechanism and a Pareto optimization method. Each objective has a corresponding reward function, and the system adjusts the weights of each objective according to changes in the environment.

[0106] Water resource reward function: The reward function gives rewards based on the degree of water consumption reduction. The reward function is designed as the inverse of water consumption, aiming to encourage the model to reduce water consumption while ensuring demand:

[0107] ;

[0108] where, is the reward for saving water resources, is the current regional water consumption.

[0109] Irrigation efficiency reward function: The reward function gives rewards based on the ratio of irrigation water and crop yield, encouraging the system to maximize irrigation efficiency under limited water resources:

[0110] ;

[0111] where, is the reward for irrigation efficiency, is the yield of crops, is the amount of water used for irrigation.

[0112] Ecological demand reward function: The reward function optimizes ecological demand based on the water level of the ecological region to ensure that the ecological region receives sufficient water resources:

[0113] ;

[0114] where, is the reward for ecological demand, is the water level of the ecological region, is the minimum water level required by the ecological region, and abs is the absolute value function.

[0115] The total reward is calculated as:

[0116] ;

[0117] where w1, w2, w3 are the weights of the objectives, representing the relative importance of saving water resources, improving irrigation efficiency, and ensuring ecological needs. The system dynamically adjusts these weights based on the current environmental state to ensure optimal water resource allocation.

[0118] To further optimize multiple objectives, the system also employs a Pareto optimization method to handle conflicts between multiple objectives. The core idea of the Pareto optimization method is to find an optimal balance point between multiple objectives, so that the improvement of a certain objective will not come at the expense of other objectives. The specific Pareto optimization process is implemented through the following formula:

[0119] ;

[0120] The system ensures that the optimization of one objective will not greatly affect the optimization effect of other objectives by calculating the relative trade-offs between different objectives.

[0121] During the training process, the system uses a deep Q network (DQN) to estimate the Q value of each state-action pair, and the goal of the reinforcement learning model is to maximize the cumulative reward. After using the weighted reward mechanism and Pareto optimization, the total reward value will be passed into the Q value function:

[0122] ;

[0123] where R total is the total reward calculated based on the weighted reward mechanism and Pareto optimization; Q old is the current Q value, representing the value in state s t and action a t ; Q new is the updated Q value, representing the evaluation of water resource allocation after the new policy optimization; γ is the discount factor, representing the importance of future rewards; α is the learning rate, used to control the influence of new experiences on model updates. s t represents the current environmental state, including soil moisture, air humidity, wind speed, current water resource reserves, crop growth stage, and current irrigation demand; a t represents the current water resource management decision, including irrigation water allocation, ecological flow regulation, and regional water resource scheduling; s t+1 represents the next environmental state, including soil moisture, air humidity, wind speed, current water resource reserves, crop growth stage, and current irrigation demand; a' represents the next water resource management decision, including irrigation water allocation, ecological flow regulation, and regional water resource scheduling.

[0124] Through this training mechanism, the system continuously optimizes water resource allocation strategies, ensuring that in practical applications it can balance between multiple goals and respond to environmental changes, achieving efficient and rational use of water resources. Target networks, experience replay, and other methods are used during training to ensure the stability of reinforcement learning training.

[0125] Step 7: As shown in Figure 4 , the trained deep reinforcement learning model is applied to the actual water resource management system, combining real-time climate data and water demand data to dynamically adjust the allocation scheme of water resources. The system monitors real-time climate change, precipitation, soil moisture, and other environmental parameters, and calculates the optimal allocation scheme based on the current environmental state. The data collected by the meteorological sensors, soil moisture sensors, and precipitation sensors deployed within the system are transmitted to the central control unit and preprocessed as input data for model analysis. In specific applications, the water resource management system obtains real-time data within the region every hour. When the model receives these input data, it calculates the optimal water resource allocation scheme. Whenever there is a change in climate or soil moisture, the system will immediately adjust the allocation strategy based on the new data to ensure rational allocation of water resources and effectively respond to emergencies. At the same time, the real-time collected data and allocation strategy are saved locally for subsequent use.

[0126] As shown in Figure 5 , in addition, the system regularly evaluates the optimization effect and conducts incremental learning combined with newly collected data. The previously saved data is manually corrected, and the performance of the model is judged according to the saved allocation strategy. When the performance does not meet the target requirements, the model is incrementally trained using the newly saved data and the previous data, and the model is trained to meet the current performance using the above optimization method, and the model is put into use again to repeat the above steps and continuously improve the model performance.

[0127] In summary, the present application provides an efficient and sustainable water resource management optimization method by combining a deep reinforcement learning model and real-time environmental data. By monitoring environmental parameters such as climate change, soil moisture, and water demand in real time, the system can dynamically adjust water resource allocation schemes to ensure efficient use and sustainable management of water resources. Based on the deep reinforcement learning model and multi-objective optimization strategy, the present application significantly improves the adaptability of water resource management and exhibits strong robustness and real-time performance in different climate conditions, changes in water demand, and unexpected events. At the same time, the system continuously optimizes water resource allocation strategies through incremental learning and feedback mechanisms to ensure long-term efficient water resource management capabilities. Through the collaborative design of software and hardware, combined with real-time data flow and decision support of deep reinforcement learning, the system can quickly respond to environmental changes and make reasonable water resource scheduling decisions, effectively avoiding resource waste and supply shortages.

[0128] The present application provides an intelligent solution for water resource management, with wide application prospects. The system can be widely used in agricultural irrigation, industrial water, ecological protection and other fields, providing important technical support for improving the rational allocation of water resources and environmental protection.

[0129] The above is only the preferred embodiment of the present application, and it should be noted that for ordinary skilled persons in the art, without departing from the technical principles of the present application, several improvements and refinements can be made, which should also be considered within the protection scope of the present application.

Claims

1. A water resource demand optimization method based on deep reinforcement learning, characterized in that: include: Acquiring raw data and real-time data, wherein the raw data includes collecting historical climate data and historical regional water resource demand data, and the real-time data includes real-time climate data; Preprocessing the original data to obtain preprocessed data, and labeling the preprocessed data to obtain a training data set; Extracting parameters of extreme climate events from the raw data, and generating extreme scenario data for a plurality of disturbance scenarios based on a pre-trained environmental simulator according to the historical climate data and the parameters of extreme climate events, wherein the environmental simulator is used to simulate the change process of other relevant environmental parameters according to the input parameters of extreme events and historical climate data; randomly inserting the extreme scenario data into the training data set to obtain an optimized training set; Training a preset DQN-based deep reinforcement learning model according to the optimized training set, wherein the network architecture of the DQN-based deep reinforcement learning model is CNN; Optimizing the DQN-based deep reinforcement learning model based on a multi-objective reinforcement learning strategy to obtain a trained DQN-based deep reinforcement learning model, wherein the reward function of the multi-objective reinforcement learning strategy includes a water resource reward function, an irrigation efficiency reward function, and an ecological demand reward function; Input the real-time climate data into a trained DQN-based deep reinforcement learning model to output real-time optimized regional water resource demand data; The reward function is: ； ； ； ； in, Rewards for saving water resources, is the current regional water consumption, is a reward for irrigation efficiency, is the crop yield, is the amount of water used for irrigation, It is a reward for ecological needs. is the water level in the ecoregion, is the minimum water level required for the ecological region, abs is the absolute value function, w1, w2, and w3 represent the weights of the three goals of saving water resources, improving irrigation efficiency, and ensuring ecological needs, respectively.

2. The water resource demand optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The environment simulator includes an input layer, an intermediate layer, and an output layer. The input of the input layer is a feature vector with a length of 260. The intermediate layer includes a fully connected layer containing 260 hidden units, a ReLU activation function layer, a fully connected layer containing 64 hidden units, a ReLU activation function layer, a fully connected layer containing 256 hidden units, and a ReLU activation function layer, which are arranged in sequence. The output layer is a fully connected layer.

3. The water resource demand optimization method based on deep reinforcement learning according to claim 2 is characterized in that: The loss function of the environment simulator pre-training process is: ； in, 、 、 、 are the generated soil moisture, humidity, wind speed and water demand, 、 、 、 is the actual recorded soil moisture, humidity, wind speed and water demand, N is the total number of detection points, and i is the sequence number of the detection point.

4. The water resource demand optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The method also includes dealing with conflicts among the three objectives based on the Pareto optimization method. The Pareto optimization process is expressed as follows: 。 5. The water resource demand optimization method based on deep reinforcement learning according to claim 4 is characterized in that: After weighted reward mechanism and Pareto optimization, the total reward value Pass in the Q value function: ； Among them, s t Indicates the current environmental status, including soil moisture, air humidity, wind speed, current water resource reserves, crop growth stage, and current irrigation needs; a t Represents current water resource management decisions, including irrigation water allocation, ecological flow regulation, and regional water resource scheduling; t+1 represents the environmental state at the next moment, including soil moisture, air humidity, wind speed, current water resource reserves, crop growth stage, and current irrigation demand; a' represents the water resource management decision at the next moment, including irrigation water allocation, ecological flow regulation, and regional water resource scheduling; R total It is the total reward calculated by the weighted reward value and Pareto optimization after comprehensively considering water conservation, irrigation efficiency, and ecological needs. old is the current Q value, indicating that in state s t and action a t The value of is the updated Q value, which represents the evaluation of water resource allocation after the new strategy is optimized. is a discount factor that indicates the importance of future rewards, and α is the learning rate, which is used to control the impact of new experience on model updates.

6. The water resource demand optimization method based on deep reinforcement learning according to claim 1 is characterized in that: Preprocessing the raw data includes: De-noising the collected raw environmental data; Process the denoised data for filtering; Standardize or normalize the filtered data; The standardized or normalized data are processed using linear interpolation and mean filling methods to obtain preprocessed data.

7. A water resource demand optimization system based on deep reinforcement learning, characterized in that: include: An acquisition module, the acquisition module is used to acquire original data and real-time data, the original data includes historical climate data and historical regional water resource demand data, and the real-time data includes real-time climate data; A preprocessing module is used to preprocess the raw data to obtain preprocessed data, and to label the preprocessed data to obtain a training data set; A simulation module, the simulation module is used to extract parameters of extreme climate events based on the raw data, and generate extreme scenario data for a plurality of disturbance scenarios based on the historical climate data and the parameters of extreme climate events based on a pre-trained environmental simulator, the environmental simulator is used to simulate the change process of other relevant environmental parameters based on the input extreme event parameters and historical climate data; An insertion module, configured to randomly insert the extreme scenario data into the training data set to obtain an optimized training set; A training module, wherein the training module is used to train a preset DQN-based deep reinforcement learning model according to the optimized training set, wherein the network architecture of the DQN-based deep reinforcement learning model is CNN; An optimization module, wherein the optimization module is configured to optimize the DQN-based deep reinforcement learning model based on a multi-objective reinforcement learning strategy to obtain a trained DQN-based deep reinforcement learning model, wherein the reward function of the multi-objective reinforcement learning strategy includes a water resource reward function, an irrigation efficiency reward function, and an ecological demand reward function; An output module, configured to input the real-time climate data into a trained DQN-based deep reinforcement learning model and output real-time optimized regional water resource demand data; The reward function is: ； ； ； ； in, Rewards for saving water resources, is the current regional water consumption, is a reward for irrigation efficiency, is the crop yield, is the amount of water used for irrigation, It is a reward for ecological needs. is the water level in the ecoregion, is the minimum water level required for the ecological region, abs is the absolute value function, w1, w2, and w3 represent the weights of the three goals of saving water resources, improving irrigation efficiency, and ensuring ecological needs, respectively.

8. The water resource demand optimization system based on deep reinforcement learning according to claim 7 is characterized in that: The environment simulator includes an input layer, an intermediate layer, and an output layer. The input of the input layer is a feature vector with a length of 260. The intermediate layer includes a fully connected layer containing 260 hidden units, a ReLU activation function layer, a fully connected layer containing 64 hidden units, a ReLU activation function layer, a fully connected layer containing 256 hidden units, and a ReLU activation function layer, which are arranged in sequence. The output layer is a fully connected layer.

Images