Flood control scheduling method combining convolutional neural network and Monte Carlo tree search

Abstract

The invention provides a flood control scheduling method combining a convolutional neural network and Monte Carlo tree search, which comprises the following steps of: firstly, collecting and sorting original data of historical flood control scheduling and simulated flood control scheduling, and obtaining a flood characteristic matrix data set by using a designed data conversion method; the flood characteristic matrix data set is used for training two convolutional neural networks, namely a strategy network and a value network, the strategy network is used for predicting the delivery flow, and the value network is used for evaluating the comprehensive flood control effect; and finally, taking the optimal comprehensive flood control effect as a target, and combining Monte Carlo tree search with a strategy network and a value network to find an optimal flood control scheduling scheme. When the method is applied to an actual flood control dispatching process, the outbound flow of the dispatching process is stable, and flood with different frequencies can be coped with; meanwhile, the flood control dispatching effect is improved to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent flood control scheduling, and in particular relates to a flood control scheduling method combining a convolutional neural network and a Monte Carlo tree search. Background Art

[0002] Floods are one of the natural disasters that human society faces, which is highly destructive, has a wide range of disasters, is long-lasting, and occurs frequently. Studies have shown that due to global climate change, the intensity, frequency, and severity of floods may increase. How to resist the increasingly severe floods has become a problem. The construction and operation and dispatching of reservoir groups are important measures for flood prevention and disaster reduction. With the gradual saturation of reservoir construction, it has become the key to further strengthen flood control capabilities and improve the level of operation and dispatching. Since flood control dispatching problems have the characteristics of multiple objectives, multiple constraints, and multiple stages. Although a good dispatching plan can be formulated by relying on artificial experience and dispatching rules, this method has poor versatility and lacks flexibility in practical applications. How to quickly formulate a reasonable flood control dispatching plan and give full play to the storage function of the reservoir is a practical problem that my country's hydropower industry needs to solve urgently.

[0003] Conventional flood control scheduling of reservoirs generally includes the following three steps: (1) Establish a flood control scheduling model including objective functions and constraints. (2) Use optimization algorithms to solve the model and obtain a flood control scheduling plan. (3) Determine the final flood control scheduling plan based on factors such as rainfall forecasts and flood forecasts. Optimization algorithms have been a hot topic of research in recent years. Optimization algorithms can be roughly divided into two categories: traditional flood control scheduling methods and intelligent flood control scheduling methods.

[0004] Traditional optimization methods usually include empirical rule method, linear programming, nonlinear programming and dynamic programming. With the development of computer technology and the Internet, intelligent flood control scheduling methods have received more and more attention. Intelligent algorithms in the field of flood control scheduling can be roughly divided into three categories: evolutionary algorithms, case-based reasoning algorithms and machine learning algorithms. These algorithms have achieved good results in solving reservoir flood control scheduling problems. However, there are still several problems that have not been well solved: (1) Displaying the characteristics of the reservoir scheduling process can be achieved by drawing flow curves and water level curves. We can obtain features by observing flow and water level diagrams, but this is very difficult for computers. (2) Although hydropower stations now have a lot of historical scheduling data, they have not been fully utilized in actual flood control scheduling. (3) Existing flood control scheduling methods based on deep learning are only committed to finding solutions similar to historical solutions. Few studies have gone a step further and explored finding flood control scheduling solutions that are better than historical solutions. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a flood control scheduling method combining a convolutional neural network and a Monte Carlo tree search. The device can solve the problem that traditional 3D printers cannot quickly switch materials during printing, cannot easily and quickly make models of the interfaces of different materials, and the materials contained in the manufactured models are single, or the manufacturing process is cumbersome and complicated; and the printing material can be a liquid-solid combination, and gold material can be added to the model to improve the strength of the printed model.

[0006] In order to realize the above technical features, the purpose of the present invention is to achieve: a flood control scheduling method combining convolutional neural network and Monte Carlo tree search, the specific operation steps are as follows: Step 1: Create a flood characteristic matrix dataset: Collect and organize the original data of historical flood control dispatch and simulated flood control dispatch, and use the designed data conversion method to obtain the flood characteristic matrix data set; Step 2: Train the strategy network and value network: Two convolutional neural networks are trained using the flood feature matrix dataset: a policy network and a value network. The policy network is used to predict outflow flow, and the value network is used to evaluate the comprehensive flood control effect. Step 3: Search for the best flood control scheduling solution: With the goal of achieving the best comprehensive flood control effect, the Monte Carlo tree search is combined with the strategy network and the value network to find the best flood control scheduling plan.

[0007] The detailed steps of making the flood characteristic matrix data set in step 1 are as follows: Step 1.1, select flood characteristics: The flood characteristics are divided into three categories: inflow flood characteristics, outflow discharge characteristics and reservoir water level characteristics. Step 1.2, convert flood characteristics: The flood characteristic matrix is ​​a three-dimensional matrix that represents different characteristic quantities in layers. The matrix consists of numbers of 0 or 1. The conversion methods of different types of characteristic quantities are also different. Specifically, they are divided into three categories; Step 1.3, create flood characteristic matrix: After the selected flood characteristic quantities are converted using the conversion method in step 1.2, the flood characteristic matrix is ​​obtained.

[0008] Step 1.1 specifically includes: Step 1.1.1, the flood characteristics of the reservoir can be described by qualitative and quantitative indicators. Among them, the qualitative indicators first describe the characteristics of the flood from a macro perspective, mainly reflected in the size of the flood and the location of the flood peak, which are composed of flood frequency, flood shape and flood peak time; the quantitative indicators describe the flood characteristics from the size of the flood flow value and the rapidity of the change, which are composed of the reservoir flow, the rising rate of the reservoir flow and the precipitation rate of the flood inflow; Step 1.1.2, the outflow characteristic is used to describe the magnitude and change trend of the downstream flow value. Flood control scheduling is a time series-based problem. The inflow flood and the outflow follow each other. The outflow characteristic is composed of the downstream flow, the downstream flow rising rate, and the downstream flow falling rate. Step 1.1.3, the reservoir water level characteristic quantity describes the flood process through the reservoir water level. The reservoir water level is strongly correlated with the storage capacity and can reflect the current adjustable storage capacity of the reservoir. The reservoir water level characteristic quantity is composed of the water level above the dam and the water level below the dam.

[0009] Step 1.2 specifically includes: Step 1.2.1: For the numerical features that change over time, use a multi-layer matrix to represent them. The number of matrix layers depends on how many digits the data consists of. First, extract the digits at different positions. Then fill the matrix and use 10 squares to store a number. The 10 squares represent 0 to 9 in sequence. 1 is stored in the corresponding number grid, and 0 is stored in the remaining grids. Step 1.2.2, for numerical features that do not change with time, use a single-layer matrix to represent them; first convert the value of the feature to a decimal; then extract the numbers at different digits; finally use 10 squares to store a number, store 1 in the corresponding number grid, and store 0 in the remaining grids. Use continuous squares of size 10*n to store data, and the value of n is equal to the number of extracted numbers. In order to better identify the features, fill multiple times in the single-layer matrix; Step 1.2.3, for enumerated feature quantities, use a single-layer matrix to represent them; first, conduct a qualitative analysis of the feature quantities, and divide the flood peak occurrence time into three categories: early, middle, and late; then use -1, 0, and 1 to represent the enumeration values ​​of the flood peak occurrence time, with -1 representing the early, 0 representing the middle, and 1 representing the late.

[0010] Step 1.3 specifically includes: The flood feature matrix is ​​a three-dimensional 0 / 1 digital matrix with a shape of n*m*m. n represents the number of feature matrix layers, which is determined by the selected flood feature quantity. m*m represents the length and width of the feature matrix, which is determined by the length of historical scheduling. A square with equal length and width is selected as the single-layer matrix shape to reduce the risk of data loss when extracting data features. By converting all historical flood dispatch data, we can obtain a flood characteristic matrix data set, which can reflect the flood characteristics and dispatch characteristics of different periods. The flood characteristic matrix data set is used as input to train the convolutional neural network.

[0011] The detailed steps of training the strategy network and the value network in step 2 are as follows: Step 2.1, determine the input and output of the convolutional neural network: The input of the strategy network is the flood feature matrix dataset constructed in step 1, and the output is the outflow flow. To train the neural network, each input feature matrix needs a corresponding label. The outflow flow is divided into equidistant values, and the 0-10000 The outbound flow is 50 The difference is divided into two types, and the category of outbound traffic is used as the label of the policy network. The policy network uses a classified convolutional neural network. The input of the value network is the flood feature matrix dataset constructed in step 1, and the output is the comprehensive flood control effect, which is a specific value. The comprehensive flood control effect is used as the label of the value network; the comprehensive flood control effect indicates the effectiveness of the flood control scheduling plan, and the calculation formula is as follows: (1); In the formula, It indicates the comprehensive flood control effect; It represents the normalized value of the combined minimum flow target of the downstream flood control point, the minimum flood control storage capacity target of the reservoir, and the minimum difference target between the final water level and the ideal water level; The weight coefficients corresponding to the combined flow minimum target of the downstream flood control control point, the minimum target of the flood control storage capacity of the reservoir, and the minimum target of the difference between the final water level and the ideal water level are expressed; the value network uses a regression-type convolutional neural network; Step 2.2, design the convolutional neural network structure: Convolutional layers and pooling layers appear in combination. A more complex deep convolutional neural network is composed of multiple such combinations. Then a fully connected layer is added to flatten the data. Different output shapes are used according to actual problems. The policy network and value network are trained separately. The policy network consists of two parts, namely the public layer of the network and the policy network layer. The value network consists of two parts, namely the public layer of the network and the value network layer. Step 2.3, set the hyperparameters of the training network: In the process of training convolutional neural networks, the selection of loss functions and optimizers, the setting of the number of iterations and the learning rate hyperparameters are particularly important. The trial and error method is used to compare the model accuracy of different schemes to determine the hyperparameters of the training network; Step 2.4, start training the convolutional neural network: The network structure and hyperparameters of network training are adjusted according to the training results until the best training results are obtained, and the network structure and weight parameters are saved.

[0012] The detailed steps of searching for the best flood control scheduling scheme in step 3 are: using a method combining convolutional neural network and Monte Carlo tree search to make strategic decisions on flood control scheduling, using the strategy network to find feasible outflow flow values ​​with higher probability, and using the value network to evaluate the comprehensive flood control effect, so as to improve the simulation efficiency of the Monte Carlo tree search algorithm.

[0013] The step 3 specifically includes: Step 3.1, select decision tree nodes: The parameter for selecting nodes uses the parallel upper bound confidence of the tree to select child nodes. The calculation formula is: (2); In the formula, Represents the parallel ascending confidence interval of the tree, which will be selected when making decisions The path with the largest value; Representation Node k The average reward value, i.e., the comprehensive flood control effect value; Representation Node k Number of visits; Representation Node k The number of times the parent node is visited; represents the probability of node k being selected; is a constant indicating the level of exploration of new nodes; Step 3.2, expand the decision tree node: The starting node in the Monte Carlo tree is called the root node, and the nodes below the root node are called child nodes. The terminal node in the Monte Carlo tree, that is, the node without child nodes, is called a leaf node. k Not a leaf node and has been simulated , then all feasible outbound flow values ​​satisfying the constraints under node k will be expanded to node k The ordinary Monte Carlo tree search adopts the random expansion method, which will lead to low search efficiency and difficulty in finding a better solution. The policy network is used to output the possible outbound traffic probability set, and the outbound traffic probability set is used to guide the Monte Carlo tree search, which reduces the waste of computing power of the Monte Carlo tree search on unnecessary solutions and improves the efficiency of the Monte Carlo tree search. Step 3.3, simulate the decision tree node, if the node k The node expanded is not a leaf node and has not been explored , the simulation will be executed, taking this node as the initial node, and selecting new actions to generate new states according to the parallel upper bound confidence of the computation tree until a leaf node is encountered; Step 3.4, back propagate decision tree nodes: After the simulation is completed, all nodes are leaf nodes, but the scheduling process may not be completed at this time, because some nodes may not be fully expanded. In order to improve the accuracy of the model, the value network is introduced to evaluate the comprehensive flood control effect of the endgame; for nodes whose scheduling process has not ended, the value network is used to evaluate the comprehensive flood control effect value. ; For the node where the scheduling process ends, its real comprehensive flood control effect value is calculated ; Comprehensive flood control effect The node access information starts from the leaf node and gradually propagates upward to the root node, while updating the information of each node in the following way: (3); In the formula, Representation Node k The average reward value, i.e., the comprehensive flood control effect value; Representation Node k Number of visits; and Respectively represent nodes k Updated comprehensive flood control effect value and exploration times; It represents the comprehensive flood prevention effect value of leaf nodes back propagated after simulation.

[0014] Flood control dispatch is the process of staggered outbound and inbound flows, and the comprehensive flood control effect is for outbound flows, so for outbound flow nodes, use Update, and for the incoming traffic node, use renew; Repeat steps 3.1 to 3.4 until the maximum number of iterations is reached. Finally, the node with the most visits is selected as the best child node, and the corresponding outbound flow is used as the decision flow at that moment. The above process is used for each scheduling time interval to obtain a decision outbound flow, which is combined into a complete flood control scheduling plan.

[0015] The present invention has the following beneficial effects: 1. The present invention proposes to select flood characteristic quantities and use three different conversion methods to convert one-dimensional flood control dispatch data into a three-dimensional flood characteristic matrix. The flood characteristic matrix can fully reflect the characteristics of the flood control dispatch process at different time periods, which helps the computer to better obtain these characteristics.

[0016] 2. The present invention proposes to use the flood feature matrix data set to train two convolutional neural networks, the strategy network and the value network. The strategy network is used to predict the outflow flow, and the value network is used to evaluate the comprehensive flood control effect. The strategy network and the value network make full use of historical data to improve the efficiency and response speed of flood control scheduling.

[0017] 3. The present invention proposes a flood control scheduling method that combines Monte Carlo tree search with convolutional neural network. The method can explore scheduling schemes that are better than historical schemes based on historical schemes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] Figure 1 It is a flow chart of the intelligent flood control dispatching method of the present invention.

[0020] Figure 2 It is the flood characteristic quantity conversion diagram.

[0021] Figure 3 It is a schematic diagram of a convolutional neural network.

[0022] Figure 4 This is a diagram of a Monte Carlo tree search.

[0023] Figure 5 It is a diagram of the policy network training process.

[0024] Figure 6 It is a diagram of the value network training process.

[0025] Figure 7 This is a comparison chart of flood control scheduling plans using different methods. DETAILED DESCRIPTION

[0026] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0027] Embodiment 1: like Figure 1 As shown, the flood control scheduling method proposed by the present invention combines a convolutional neural network with a Monte Carlo tree search, comprising the following steps: Step 1: Create a flood characteristic matrix dataset: Collect and organize the original data of historical flood control dispatch and simulated flood control dispatch, and use the designed data conversion method to obtain the flood characteristic matrix data set; Step 2: Train the strategy network and value network: Two convolutional neural networks, a policy network and a value network, are trained using the flood feature matrix dataset. The policy network is used to predict outflow flow, and the value network is used to evaluate the comprehensive flood control effect. Step 3: Search for the best flood control scheduling solution: With the goal of achieving the best comprehensive flood control effect, the Monte Carlo tree search is combined with the strategy network and the value network to find the best flood control scheduling plan.

[0028] Furthermore, the detailed steps of making the flood dataset in step 1 are as follows: Step 1.1, select flood characteristic quantities. Flood characteristic quantities are divided into three categories: inflow flood characteristic quantities, outflow discharge characteristic quantities and reservoir water level characteristic quantities.

[0029] (1) The flood characteristics of the reservoir can be described by qualitative and quantitative indicators. Qualitative indicators first describe the characteristics of the flood from a macro perspective, mainly in terms of the size of the flood and the location of the flood peak, and are composed of flood frequency, flood shape and flood peak time. Quantitative indicators describe the flood characteristics from the size of the flood flow value and the rapidity of the change, and are composed of the inflow flow, the inflow flow rising rate and the inflow flood precipitation rate.

[0030] (2) The characteristic quantity of outflow can be used to describe the magnitude and changing trend of the downstream flow. Flood control dispatch is a time series-based problem. Inflow flood and outflow follow each other. Therefore, an accurate description of the downstream flow is also very necessary. The characteristic quantity of outflow is composed of the downstream flow, the downstream flow rising rate and the downstream flow falling rate.

[0031] (3) Reservoir water level characteristics can be used to describe the flood process. The reservoir water level is strongly correlated with the storage capacity and can reflect the current adjustable storage capacity of the reservoir. The reservoir water level characteristics consist of the water level above the dam and the water level below the dam.

[0032] Step 1.2, convert flood feature quantities. The flood feature matrix is ​​a three-dimensional matrix that represents different feature quantities in layers. The matrix consists of numbers of 0 or 1. The conversion methods of different types of feature quantities are also different, and can be divided into the following three categories.

[0033] (1) For numerical features that change over time, such as flow, water level, rising and falling rates, etc., a multi-layer matrix is ​​used to represent them. The number of matrix layers depends on how many numbers the data consists of. Taking the outbound and inbound flows as an example, the filling method is as follows: Figure 2 (a) First, extract the numbers at different digits (for example, 1234 is extracted into 4 numbers 1, 2, 3, and 4); then fill the matrix, use 10 squares to store a number, and the 10 squares represent 0 to 9 in sequence. 1 is stored in the corresponding number grid, and 0 is stored in the remaining grids.

[0034] (2) For numerical features that do not change over time, such as flood frequency, a single-layer matrix is ​​used to represent them. Taking flood frequency as an example, the filling method is as follows: Figure 2 (b) is shown. First, convert the feature value into a decimal; then extract the numbers at different digits; finally, use 10 squares to store a number, store 1 in the corresponding number grid, and store 0 in the remaining grids. Use continuous squares of size 10*n to store data, and the value of n is equal to the number of extracted numbers. In order to better identify the features, multiple fillings are performed in the single-layer matrix.

[0035] (3) For enumerated feature quantities, such as flood shape and flood peak occurrence time, a single-layer matrix is ​​used to represent them. Taking the peak occurrence time as an example, the filling method is as follows: Figure 2 (c) As shown. First, the characteristic quantity is qualitatively analyzed, and the flood peak occurrence time is divided into three categories: early, middle and late. In this example, the flood peak occurrence time is defined as early. Then, -1, 0 and 1 are used to represent the enumeration value of the flood peak occurrence time, and -1 is used to represent the early. Finally, the whole layer matrix is ​​filled with the enumeration value -1 using the staggered filling method.

[0036] Step 1.3, make a flood feature matrix. After using the conversion method of step 1.2 on the selected flood feature quantities, you can get the flood feature matrix. The flood feature matrix is ​​a three-dimensional 0 / 1 digital matrix with a shape of n*m*m. n represents the number of feature matrix layers, which is determined by the selected flood feature quantities. m*m represents the length and width of the feature matrix, which is determined by the length of the historical scheduling time. Selecting a square with equal length and width as the single-layer matrix shape can reduce the risk of data loss when extracting data features.

[0037] By converting all historical flood dispatch data, we can get the flood characteristic matrix data set. The flood characteristic matrix data set can reflect the flood characteristics and dispatch characteristics of different periods, and the flood characteristic matrix data set is used as input to train the convolutional neural network.

[0038] Furthermore, the detailed steps of training the strategy network and the value network in step 2 are: Step 2.1, determine the input and output of the convolutional neural network.

[0039] The input of the strategy network is the flood feature matrix dataset constructed in step 1, and the output is the outflow. To train a neural network, each input feature matrix needs a corresponding label. Directly using the outflow as a label has certain defects. Because the value of the outflow changes rapidly and it is difficult to grasp the internal rules, the outflow is divided into arithmetic intervals. The outbound flow is 50 The difference between the two can be divided into 200 categories. The category of the outbound traffic is used as the label of the policy network. This form is very consistent with the convolutional neural network for classification problems, so the policy network uses a classification convolutional neural network.

[0040] The input of the value network is the flood feature matrix dataset constructed in step 1, and the output is the comprehensive flood control effect, which is a specific value. The comprehensive flood control effect is used as the label of the value network. The comprehensive flood control effect indicates the effectiveness of the flood control scheduling plan. The calculation formula is as follows: (1); in, It indicates the comprehensive flood control effect; It represents the normalized value of the combined minimum flow target of the downstream flood control point, the minimum flood control storage capacity target of the reservoir, and the minimum difference target between the final water level and the ideal water level; It represents the weight coefficients corresponding to the combined minimum flow target of the downstream flood control control point, the minimum flood control storage capacity of the reservoir, and the minimum difference between the final water level and the ideal water level.

[0041] This form is very suitable for the convolutional neural network of regression problems, so the value network uses a regression-type convolutional neural network.

[0042] Step 2.2, design the convolutional neural network structure. Figure 3 As shown in the figure, it is the basic structure of convolutional neural network. Convolutional layers and pooling layers usually appear in combination, and more complex deep convolutional neural networks are composed of multiple such combinations. Then a fully connected layer is added to flatten the data, and different output shapes are used according to actual problems. The policy network and value network are trained separately. The policy network consists of two parts, namely the common layer of the network and the policy network layer. The value network consists of two parts, namely the common layer of the network and the value network layer.

[0043] Step 2.3, set the hyperparameters of the training network. In the training process of convolutional neural networks, the selection of loss function and optimizer, the setting of hyperparameters such as the number of iterations and learning rate are also particularly important. Refer to similar model parameters and use trial and error to compare the model accuracy of different solutions to determine the hyperparameters of the training network.

[0044] Step 2.4, start training the convolutional neural network. Adjust the network structure and hyperparameters of network training according to the training results until the best training results are obtained, and save the network structure and weight parameters.

[0045] Furthermore, the detailed steps of searching for the best flood control scheduling scheme in step 3 are: A method combining convolutional neural network and Monte Carlo tree search is used to make strategic decisions for flood control scheduling. The strategy network is used to find feasible outflow values ​​with higher probability, and the value network is used to evaluate the comprehensive flood control effect to improve the simulation efficiency of the Monte Carlo tree search algorithm.

[0046] Step 3.1, select the decision tree node. Figure 4 As shown in the selection part, for a node, there will be multiple child nodes after discretization, and the selection is how to choose the best child node. The parameter for selecting nodes uses the Parallelized Upper Confidence Bounds for Trees (PUCT) to select child nodes, and the calculation formula is: (2); In the formula, Represents the parallel ascending confidence interval of the tree, which will be selected when making decisions The path with the largest value. Representation Node k The average reward value is the comprehensive flood control effect value. Representation Node k The number of times it has been visited. Representation Node k The number of times the parent node is visited. Representation Node k The probability of being selected. is a constant representing the exploration level for new nodes and is selected as 2.0.

[0047] Step 3.2, expand the decision tree nodes. Figure 4 As shown in the selection expansion, the starting node in the Monte Carlo tree is called the root node, and the root node is called the child node. The terminal node in the Monte Carlo tree, that is, the node without child nodes, is called a leaf node. If node k is not a leaf node and has been simulated , then all feasible outbound flow values ​​under node k that meet the constraints will be expanded to child nodes of node k. Ordinary Monte Carlo tree search uses random expansion, which will lead to low search efficiency and difficulty in finding a better solution. Use the policy network to output the possible outbound flow probability set, and use the outbound flow probability set to guide the Monte Carlo tree search. Reduce the waste of computing power of Monte Carlo tree search on unnecessary solutions and improve the efficiency of Monte Carlo tree search.

[0048] Step 3.3, simulate the decision tree node. Figure 4 As shown in the simulation part, if the node expanded by node k is not a leaf node and has not been explored , the simulation will be executed. Taking this node as the initial node, a new action is selected to generate a new state according to the parallel upper limit confidence of the computation tree until a leaf node is encountered.

[0049] Step 3.4, back propagate the decision tree nodes. Figure 4As shown in the back propagation section, after the simulation is completed, all nodes are leaf nodes, but the scheduling process may not be completed at this time. Because there may be nodes that have not been fully expanded, in order to improve the accuracy of the model, the value network is introduced to evaluate the comprehensive flood control effect of the endgame. For nodes whose scheduling process has not ended, the value network is used to evaluate the comprehensive flood control effect value. ; For the node where the scheduling process ends, its real comprehensive flood control effect value is calculated . The comprehensive flood control effect The node access information starts from the leaf node and gradually propagates upward to the root node. At the same time, the information of each node is updated as follows: (3); In the formula Represents the average reward value of node k, that is, the comprehensive flood control effect value. Indicates the number of times node k is visited. and They represent the updated comprehensive flood control effect value and exploration times of node k respectively. It represents the comprehensive flood prevention effect value of leaf nodes back propagated after simulation.

[0050] It should be noted that flood control scheduling is a process of staggered outbound and inbound flows, while the comprehensive flood control effect is for outbound flows. Update, and for the incoming traffic node, use renew.

[0051] Repeat steps 3.1 to 3.4 until the maximum number of iterations is reached. Finally, the node with the most visits is selected as the best child node, and the corresponding outbound flow is used as the decision flow at that moment. The above process is used for each scheduling time interval to obtain a decision outbound flow, which is combined into a complete flood control scheduling plan.

[0052] Embodiment 2: The Chitan Hydropower Station is taken as the research object of flood control dispatch. The Chitan Hydropower Station is the first hydropower station in the cascade development of the Jinxi River Basin. The basin catchment area is 4,766 square kilometers, accounting for 66% of the Jinxi River Basin. The project is mainly for power generation, and also has comprehensive benefits such as flood control, shipping, and fish farming. The total installed capacity of the Chitan Hydropower Station is 208,000 kilowatts. It is the backbone power plant of the Fujian Power Grid, with a maximum storage capacity of 870 million m 3 、Normal storage capacity 700 million m 3 The normal water storage level is 275.0m, the design flood level is 275.03m, and the verification flood level is 277.10m. The average annual natural runoff is 5.982 billion cubic meters. The main flood season is from April to September, with a natural runoff of 4.442 billion m 3 , accounting for 74.26% of the whole year.

[0053] First, the training results of the policy network and the value network are shown as follows: (1) Training results of the policy network: The training process of the policy network used to predict outbound traffic is as follows Figure 5 As shown in the figure, the horizontal axis represents the number of network training iterations, the vertical axis on the left represents the cross entropy loss function, and the vertical axis on the right represents the probability that the predicted outbound flow is the same as the historical outbound flow. As the number of training times increases, the loss function continues to decrease, and the accuracy of the training set and the validation set is gradually increasing. This shows that the model has continuously learned the task of predicting outbound flow. It can be seen that the accuracy of the training set tends to be stable at 75%, and the accuracy of the validation set tends to be stable at 52%. The performance of the policy network is reflected in the accuracy of the validation set, which shows that after training, the accuracy of the policy network for predicting outbound flow can reach 52%. At the same time, it can be seen that the accuracy of the training set is significantly higher than that of the validation set, but the accuracy of the validation set has not decreased, which means that the performance of the policy network has not decreased. The phenomenon of model overfitting is that when the accuracy of the training set increases, the accuracy of the validation set no longer changes or even decreases, so the model does not overfit. The main reason why the accuracy of the training set is greater than the accuracy of the validation set is that the problem of uneven distribution of the two data sets cannot be overcome. That is, the "knowledge" learned in the training set does not appear in the corresponding "test questions" in the validation set.

[0054] (2) Training results of the value network: The training process of the value network used to predict the comprehensive flood control effect is as follows: Figure 6 As shown, the horizontal axis represents the number of iterations of network training, the left vertical axis represents the mean squared error loss function (Mean Squared Error Loss), and the right vertical axis represents the probability that the predicted comprehensive flood control effect is correct. Since the comprehensive flood control effect is a value belonging to (-1,1), the predicted value and the actual value are difficult to be exactly the same, so the present invention will use an error interval of 0.05 to calculate the probability that the predicted comprehensive flood control effect is correct. With the increase in the number of training times, the loss function is constantly decreasing, and the accuracy of the training set and the validation set is also gradually increasing. It shows that the model has continuously learned the task of evaluating the comprehensive flood control effect. It can be seen that the accuracy of the training set tends to stabilize at 50%, and the accuracy of the validation set tends to stabilize at 57%. The performance of the value network is mapped to the accuracy of the validation set, which shows that after training, the accuracy of the value network for evaluating the comprehensive flood control effect can reach 57%.

[0055] The results of different flood control dispatch methods are then presented as follows: In order to verify the effectiveness of the proposed flood control scheduling method based on the combination of convolutional neural network and Monte Carlo tree search, floods of different frequencies (5%, 2%, 1% and 0.2%) were selected in the Jinxi River Basin for flood control scheduling research. The proposed method was compared with the historical flood control scheduling results. At the same time, two other methods were added, namely, using convolutional neural network alone and Monte Carlo tree search alone for flood control scheduling. The comparison results are shown in Figure 2. Figure 7 shown. Figure 7 The flood control dispatch process for floods of different frequencies is shown in the figure. The upper part is the water level process line, and the lower part is the inflow and outflow process lines.

[0056] First, we look at the dispatch plan generated by the convolutional neural network method alone. The CNN method uses historical dispatch data to train the network and uses the network to generate a dispatch plan. Similar to the above conclusion, this method shows an obvious follow-up relationship with the actual value, and can cope with some floods when the flood is small or the flood control reservoir capacity is sufficient. However, it performs poorly when the flood is large or the flood control reservoir capacity is insufficient. Figure 7 As shown in (c), the CNN method deviates far from the actual value in the second half. It is observed that the water level has exceeded the verified flood level of 277.10m, but the CNN method is still causing the water level to rise, which is obviously inconsistent with the actual scheduling. Since the CNN method only generates values ​​close to the historical scheduling plan as the outflow flow, it does not have the ability to correct the outflow flow based on the current water level, which makes the generated scheduling plan unreasonable in some extreme cases.

[0057] Next, we will see the scheduling solution generated by the Monte Carlo tree search method alone. The Monte Carlo tree search method selects the optimal scheduling solution from feasible solutions through continuous assumptions and simulations. Figure 7 As shown in the figure, observing the outflow of the four floods, it can be seen that the MCTS method always uses a lower outflow as the decision value in the early stage. This causes the reservoir water level to rise continuously until it reaches the flood control high water level of 275.0m. After that, the same value as the inflow will be used as the outflow to keep the water level from breaking the limit value. This scheduling scheme is obviously not reasonable and difficult to apply in the actual scheduling process.

[0058] Finally, we see the scheduling plan generated by the flood control scheduling method based on the combination of convolutional neural network and Monte Carlo tree search proposed in the present invention. The method of the present invention uses a convolutional neural network to generate a probability set of possible outbound flows, and then uses the probability set to guide the simulation direction of the Monte Carlo tree search to more accurately find the optimal scheduling plan. Unlike the CNN method, the scheduling method of the present invention has the same trend as historical floods, but it is very different, as if this is a brand new scheduling plan. By observing the outbound flow of the four floods, it can be seen that the outbound flow of the method of the present invention is not very volatile and can be well applied to actual scheduling. Figure 7 In (a), (b) and (c), this method makes full use of the existing flood control reservoir capacity, and the water level at the end of the scheduling period is closer to the target water level compared with the historical scheme.

[0059] In general, the flood control effects of the CNN method and the historical scheme are roughly equivalent, while the MCTS method is less effective. The method of the present invention performs steadily in the face of floods of different frequencies, and has a certain improvement in the comprehensive flood control effect compared with the historical scheme. This method can be well applied to actual flood control scheduling.

Claims

1. A flood control scheduling method combining convolutional neural network and Monte Carlo tree search, characterized in that: The specific steps are as follows: Step 1: Create a flood characteristic matrix dataset: Collect and organize the original data of historical flood control dispatch and simulated flood control dispatch, and use the designed data conversion method to obtain the flood characteristic matrix data set; Step 2: Train the strategy network and value network: Two convolutional neural networks are trained using the flood feature matrix dataset: a policy network and a value network. The policy network is used to predict outflow flow, and the value network is used to evaluate the comprehensive flood control effect. Step 3: Search for the best flood control scheduling solution: With the goal of achieving the best comprehensive flood control effect, the Monte Carlo tree search is combined with the strategy network and the value network to find the best flood control scheduling plan.

2. The flood control scheduling method combining convolutional neural network and Monte Carlo tree search according to claim 1, characterized in that: The detailed steps of making the flood characteristic matrix data set in step 1 are as follows: Step 1.1, select flood characteristics: The flood characteristics are divided into three categories: inflow flood characteristics, outflow discharge characteristics and reservoir water level characteristics. Step 1.2, convert flood characteristics: The flood characteristic matrix is ​​a three-dimensional matrix that represents different characteristic quantities in layers. The matrix consists of numbers of 0 or 1. The conversion methods of different types of characteristic quantities are also different. Specifically, they are divided into three categories; Step 1.3, create flood characteristic matrix: After the selected flood characteristic quantities are converted using the conversion method in step 1.2, the flood characteristic matrix is ​​obtained.

3. The flood control scheduling method combining convolutional neural network and Monte Carlo tree search according to claim 2 is characterized in that: Step 1.1 specifically includes: Step 1.1.1, the flood characteristics of the reservoir can be described by qualitative and quantitative indicators. Among them, the qualitative indicators first describe the characteristics of the flood from a macro perspective, mainly reflected in the size of the flood and the location of the flood peak, which are composed of flood frequency, flood shape and flood peak time; the quantitative indicators describe the flood characteristics from the size of the flood flow value and the rapidity of the change, which are composed of the reservoir flow, the rising rate of the reservoir flow and the precipitation rate of the flood inflow; Step 1.1.2, the outflow characteristic is used to describe the magnitude and change trend of the downstream flow value. Flood control scheduling is a time series-based problem. The inflow flood and the outflow follow each other. The outflow characteristic is composed of the downstream flow, the downstream flow rising rate, and the downstream flow falling rate. Step 1.1.3, the reservoir water level characteristic quantity describes the flood process through the reservoir water level. The reservoir water level is strongly correlated with the storage capacity and can reflect the current adjustable storage capacity of the reservoir. The reservoir water level characteristic quantity is composed of the water level above the dam and the water level below the dam.

4. The flood control scheduling method combining convolutional neural network and Monte Carlo tree search according to claim 3 is characterized in that: Step 1.2 specifically includes: Step 1.2.1: For the numerical features that change over time, use a multi-layer matrix to represent them. The number of matrix layers depends on how many digits the data consists of. First, extract the digits at different positions. Then fill the matrix and use 10 squares to store a number. The 10 squares represent 0 to 9 in sequence. 1 is stored in the corresponding number grid, and 0 is stored in the remaining grids. Step 1.2.2, for numerical features that do not change with time, use a single-layer matrix to represent them; first convert the value of the feature to a decimal; then extract the numbers at different digits; finally use 10 squares to store a number, store 1 in the corresponding number grid, and store 0 in the remaining grids. Use continuous squares of size 10*n to store data, and the value of n is equal to the number of extracted numbers. In order to better identify the features, fill multiple times in the single-layer matrix; Step 1.2.3, for enumerated feature quantities, use a single-layer matrix to represent them; first, conduct a qualitative analysis of the feature quantities, and divide the flood peak occurrence time into three categories: early, middle, and late; then use -1, 0, and 1 to represent the enumeration values ​​of the flood peak occurrence time, with -1 representing the early, 0 representing the middle, and 1 representing the late.

5. The flood control scheduling method combining convolutional neural network and Monte Carlo tree search according to claim 4 is characterized in that: Step 1.3 specifically includes: The flood feature matrix is ​​a three-dimensional 0 / 1 digital matrix with a shape of n*m*m. n represents the number of feature matrix layers, which is determined by the selected flood feature quantity. m*m represents the length and width of the feature matrix, which is determined by the length of historical scheduling. A square with equal length and width is selected as the single-layer matrix shape to reduce the risk of data loss when extracting data features. By converting all historical flood dispatch data, we can obtain a flood characteristic matrix data set, which can reflect the flood characteristics and dispatch characteristics of different periods. The flood characteristic matrix data set is used as input to train the convolutional neural network.

6. The flood control dispatching method combining convolutional neural network and Monte Carlo tree search according to claim 5, characterized in that: The detailed steps of training the strategy network and the value network in step 2 are as follows: Step 2.1, determine the input and output of the convolutional neural network: The input of the strategy network is the flood feature matrix dataset constructed in step 1, and the output is the outflow flow. To train the neural network, each input feature matrix needs a corresponding label. The outflow flow is divided into equidistant values, and the 0-10000 The outbound flow is 50 The difference is divided into two types, and the category of outbound traffic is used as the label of the policy network. The policy network uses a classified convolutional neural network. The input of the value network is the flood feature matrix dataset constructed in step 1, and the output is the comprehensive flood control effect, which is a specific value. The comprehensive flood control effect is used as the label of the value network; the comprehensive flood control effect indicates the effectiveness of the flood control scheduling plan, and the calculation formula is as follows: （1）； In the formula, It indicates the comprehensive flood control effect; It represents the normalized value of the combined minimum flow target of the downstream flood control point, the minimum flood control storage capacity target of the reservoir, and the minimum difference target between the final water level and the ideal water level; The weight coefficients corresponding to the combined flow minimum target of the downstream flood control control point, the minimum target of the flood control storage capacity of the reservoir, and the minimum target of the difference between the final water level and the ideal water level are expressed; the value network uses a regression-type convolutional neural network; Step 2.2, design the convolutional neural network structure: Convolutional layers and pooling layers appear in combination. More complex deep convolutional neural networks are composed of multiple such combinations, and then a fully connected layer is added to flatten the data, using different output shapes based on actual problems. The policy network and value network were trained separately. The policy network consists of two parts, namely the public layer of the network and the policy network layer, and the value network consists of two parts, namely the public layer of the network and the value network layer; Step 2.3, set the hyperparameters of the training network: In the training process of convolutional neural networks, the selection of loss function and optimizer, the setting of the number of iterations and learning rate hyperparameters are particularly important. The trial and error method is used to compare the model accuracy of different schemes to determine the hyperparameters of the training network; Step 2.4, start training the convolutional neural network: The network structure and hyperparameters of network training are adjusted according to the training results until the best training results are obtained, and the network structure and weight parameters are saved.

7. The flood control dispatching method combining convolutional neural network and Monte Carlo tree search according to claim 6, characterized in that: The detailed steps of searching for the best flood control scheduling scheme in step 3 are: using a method combining convolutional neural network and Monte Carlo tree search to make strategic decisions on flood control scheduling, using the strategy network to find feasible outflow flow values ​​with higher probability, and using the value network to evaluate the comprehensive flood control effect, so as to improve the simulation efficiency of the Monte Carlo tree search algorithm.

8. The flood control dispatching method combining convolutional neural network and Monte Carlo tree search according to claim 6, characterized in that: The step 3 specifically includes: Step 3.1, select decision tree nodes: The parameter for selecting nodes uses the parallel upper bound confidence of the tree to select child nodes. The calculation formula is: （2）； In the formula, Represents the parallel ascending confidence interval of the tree, which will be selected when making decisions The path with the largest value; Representation Node k The average reward value, i.e., the comprehensive flood control effect value; Representation Node k Number of visits; Representation Node k The number of times the parent node is visited; represents the probability of node k being selected; is a constant indicating the level of exploration of new nodes; Step 3.2, expand the decision tree node: The starting node in the Monte Carlo tree is called the root node, and the nodes below the root node are called child nodes. The terminal node in the Monte Carlo tree, that is, the node without child nodes, is called a leaf node. k Not a leaf node and has been simulated , then all feasible outbound flow values ​​satisfying the constraints under node k will be expanded to node k The ordinary Monte Carlo tree search adopts the random expansion method, which will lead to low search efficiency and difficulty in finding a better solution. The policy network is used to output the possible outbound traffic probability set, and the outbound traffic probability set is used to guide the Monte Carlo tree search, which reduces the waste of computing power of the Monte Carlo tree search on unnecessary solutions and improves the efficiency of the Monte Carlo tree search. Step 3.3, simulate the decision tree node, if the node k The expanded node is not a leaf node and has not been explored , the simulation will be executed, taking this node as the initial node, and selecting new actions to generate new states according to the parallel upper bound confidence of the computation tree until a leaf node is encountered; Step 3.4, back propagate decision tree nodes: After the simulation is completed, all nodes are leaf nodes, but the scheduling process may not be completed at this time, because some nodes may not be fully expanded. In order to improve the accuracy of the model, the value network is introduced to evaluate the comprehensive flood control effect of the endgame; for nodes whose scheduling process has not ended, the value network is used to evaluate the comprehensive flood control effect value. ; For the node where the scheduling process ends, its real comprehensive flood control effect value is calculated ; Comprehensive flood control effect The node access information starts from the leaf node and gradually propagates upward to the root node, while updating the information of each node in the following way: （3）； In the formula, Representation Node k The average reward value, i.e., the comprehensive flood control effect value; Representation Node k Number of visits; and Respectively represent nodes k Updated comprehensive flood control effect value and exploration times; It represents the comprehensive flood prevention effect value of leaf nodes back propagated after simulation.

9. The flood control dispatching method combining convolutional neural network and Monte Carlo tree search according to claim 8, characterized in that: Flood control dispatch is the process of staggered outbound and inbound flows, and the comprehensive flood control effect is for outbound flows, so for outbound flow nodes, use Update, and for the incoming traffic node, use renew; Repeat steps 3.1 to 3.4 until the maximum number of iterations is reached; finally, select the node with the most visits as the best child node, and use the corresponding outbound flow as the decision flow at that moment. Use the above process for each scheduling time interval to obtain a decision-making outbound flow, which together form a complete flood control scheduling plan.

Images