A pavement maintenance decision-making method based on rainbow dqn reinforcement learning

Abstract

The application discloses a kind of based on Rainbow DQN reinforcement learning's road maintenance decision-making method, specifically include: S1, collection road performance historical data and the parameter of influence road performance;S2, according to the data of step S1, construct the road use performance prediction model based on LSTM neural network;S3, according to the data of step S1 and the prediction result of step S2, construct the road maintenance decision-making model based on Rainbow DQN reinforcement learning;S4, set prediction life, input corresponding step's road performance historical data, the parameter of influence road performance, by road use performance prediction model and road maintenance decision-making model, output gets the maintenance action of road in prediction life and corresponding comprehensive benefit cost ratio;S5, select the maintenance action of maximum comprehensive benefit cost ratio as the final road maintenance scheme.The application realizes the intelligentization of road maintenance decision-making method, improves the comprehensive benefit ratio of road maintenance.

Description

Technical Field

[0001] The present invention belongs to the technical field of highway maintenance, and in particular relates to a pavement maintenance decision-making method based on Rainbow DQN reinforcement learning. Background Art

[0002] Pavement maintenance is a crucial step in road management, maintaining pavement performance and ensuring road safety. Repairing severely damaged pavement not only consumes significant resources but can also shorten its service life. Pavement performance prediction, by revealing the patterns of pavement degradation, can provide data support and a scientific basis for subsequent pavement disease prediction and maintenance management, enabling preventive maintenance, improving the utilization efficiency of pavement maintenance resources, and mitigating road damage.

[0003] In specific engineering applications, accurate evaluation and scientifically sound prediction of pavement performance are essential foundational tasks for analyzing pavement maintenance needs and making maintenance decisions. With advances in information technology, the practical application of machine learning algorithms in engineering is becoming increasingly common. Traditional road maintenance decision-making methods, such as decision trees and ranking methods, rely primarily on engineers' experience and subjective opinions. Compared to machine learning-based road maintenance prediction methods, these traditional methods often yield suboptimal maintenance decisions. Summary of the Invention

[0004] The first purpose of the invention is to overcome the shortcomings and deficiencies in the existing technology, provide a pavement maintenance decision-making method based on RainbowDQN reinforcement learning, realize the intelligence of the pavement maintenance decision-making method, improve the comprehensive benefit ratio of pavement maintenance, and reduce resource waste.

[0005] The purpose of the present invention is achieved through the following technical solution: a pavement maintenance decision-making method based on Rainbow DQN reinforcement learning, comprising the steps of:

[0006] S1. Collect historical data on pavement performance and parameters affecting pavement performance;

[0007] S2. Training and optimizing the LSTM neural network model using the pavement performance historical data and parameters affecting pavement performance, and constructing a pavement performance prediction model based on the LSTM neural network;

[0008] S3. Constructing a pavement maintenance decision model based on Rainbow DQN reinforcement learning based on the pavement performance historical data, parameters affecting pavement performance, and prediction results obtained by the pavement performance prediction model;

[0009] S4. Setting a prediction period, inputting pavement performance data corresponding to a step length and parameters influencing pavement performance, obtaining a prediction result using a pavement performance prediction model, inputting historical pavement performance data, parameters influencing pavement performance, and the prediction result into a pavement maintenance decision model, and outputting a maintenance action for the pavement within the prediction period and the corresponding comprehensive benefit-cost ratio;

[0010] S5. Based on the maintenance actions and corresponding comprehensive benefit-cost ratios within the predicted lifespan, select the maintenance action with the highest comprehensive benefit-cost ratio as the final pavement maintenance plan;

[0011] Preferably, step S2 specifically includes:

[0012] S21. Preprocessing historical pavement performance data and parameters affecting pavement performance into a training set and a test set, and determining input parameters and output parameters of an LSTM neural network model;

[0013] S22. Use the Sequential model in the Keras library to build a block model, add LSTM layers and fully connected layers in sequence, compile the block model, specify the loss function and optimizer, train the LSTM neural network model using the training set, and then verify it using the test set to obtain the trained LSTM neural network model, and use the error analysis method to analyze the prediction results of the trained LSTM neural network model;

[0014] S23. Use the GridSearchCV method provided by the Sklearn library to adjust the hyperparameters of the trained LSTM neural network model, and use the mean square error (MSE) to perform error analysis to obtain the optimized LSTM neural network model. After the optimization is completed, the road performance prediction model is obtained.

[0015] Preferably, in step S21, the input parameters include traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade and pavement performance historical data, and the output parameters are pavement performance evaluation indexes, which include pavement damage condition index PCI, pavement ride quality index RQI and crack rate index CRI.

[0016] Preferably, the expression of the crack rate index CRI is:

[0017]

[0018]

[0019] Among them, k is the type of pavement crack disease, k0 is the total number of pavement crack disease types, ω kwherein, k is the total area of the disease in the measured section, ω is the total area of the measured section, β0 and β1 are model coefficients.

[0020] Preferably, step S3 specifically comprises:

[0021] S31, determining the definition of each element in the Rainbow DQN model, the elements including: an agent, a state, a reward, an environment and an action;

[0022] S32, determining the input parameters of the Rainbow DQN model, the input parameters including historical data of pavement performance, parameters affecting pavement performance and prediction results obtained by a pavement performance prediction model, the input parameters being preprocessed and divided into a training set and a test set, the Q distribution being used as an output parameter, and the number of output parameters being equal to the number of maintenance actions;

[0023] S33, using a Sequential model building block in the Keras library to construct a block model, sequentially adding a convolution layer, a fully connected layer and an output layer, compiling the block model, specifying a loss function and an optimizer, training the Rainbow DQN model using the training set, verifying the Rainbow DQN model using the test set, obtaining the trained Rainbow DQN model, and analyzing the prediction results of the trained Rainbow DQN model using an error analysis method, so as to obtain a maintenance decision-making model based on the Rainbow DQN reinforcement learning,

[0024] The Rainbow DQN model comprehensively uses multiple DQN improved algorithms to improve the performance of reinforcement learning, specifically including: a prioritized experience replay method Prioritized Replay Buffer is used to store the experience of the interaction between the agent and the environment, and the samples are sampled according to the importance, and more attention is paid to the samples that are helpful for learning; the Q value of each maintenance action is determined by a state value function V and an advantage function H, and a target network Target Q-network is introduced to regularly update the parameters of the main network to stabilize the training process; a double deep Q network method Double DQN is used to reduce the estimation bias by using two independent neural networks for decision-making and evaluation; a multi-step learning method Multi-Step is used to more accurately estimate long-term rewards; a value distribution DQN method Distributional DQN uses multiple agents to interact with the environment in parallel; and a noise parameter in the Noisy Net is used to accelerate learning and policy exploration.

[0025] Preferably, in step S31, the various elements and corresponding definitions in the Rainbow DQN model specifically include: the intelligent agent is the road section that needs to make maintenance decisions; the status includes historical pavement performance data, traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade and pavement performance evaluation index output by the pavement performance prediction model, and the pavement performance evaluation index includes the pavement damage condition index PCI, the pavement ride quality index RQI and the crack rate index CRI; the reward is the corresponding increase in the comprehensive benefit-cost ratio of executing each maintenance action; the environment is the pavement performance prediction; and the action is each maintenance action performed by the intelligent agent.

[0026] Preferably, the input parameters in step S32 include historical pavement performance data, traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade, and pavement performance evaluation index output by the pavement performance prediction model, wherein the pavement performance evaluation index includes a pavement damage index PCI, a pavement ride quality index RQI, and a crack rate index CRI;

[0027] The preprocessing includes: for numerical values, using the Max-Min method in z-score standardization to limit the range of variation to [0, 1]; for categorical data, using one-hot encoding.

[0028] Preferably, in step S33, the using the training set to train the Rainbow DQN model specifically includes:

[0029] S01, the target Q-network established has the same structure as the Q-network, and the Q-network is used to select the action a at time t. t , using the Noisy DQN exploration algorithm, that is, adding noise Noisy to the network and outputting the action with the largest Q value; Target Q-network is used to update the action a at time t+1 t+1 , using the Noisy DQN exploration algorithm, that is, adding noise Noisy to the network;

[0030] S02. After obtaining the output values ​​of the Target Q-network and Q-network and the reward R of the environment feedback, the network loss function is calculated and the parameters of the Rainbow DQN model are updated. As the number of training epochs increases, the value of the network loss function tends to stabilize. When the preset error is reached, the training is completed, and a Rainbow DQN model that meets the accuracy requirements is obtained.

[0031] Preferably, the calculation method of the comprehensive benefit-cost ratio specifically includes:

[0032] Step 1, after selecting a maintenance action a for the road section l, output the road surface performance evaluation index i at the next moment through the performance prediction model, the road surface performance evaluation index i includes the road surface damage condition index PCI, the road surface driving quality index RQI and the crack rate index CRI, and the change curve of the road surface performance evaluation index is obtained;

[0033] Step 2, calculate the maintenance benefit Eff(t) of the road section l with the road surface performance evaluation index i at the t moment, wherein the maintenance benefit at the t moment is defined as the area surrounded by the road surface performance change curve after maintenance, the road surface performance change curve before maintenance and T=t0 and T=t, and is calculated by the following formula:

[0034]

[0035] Wherein, The road surface performance change curve after maintenance and the road surface performance change curve before maintenance respectively, and l represents the road section number;

[0036] Step 3, calculate the maintenance benefit cost ratio cost(t) of the road section l with the road surface performance evaluation index i at the t moment:

[0037]

[0038] Wherein, C(t) is the maintenance cost of the road section l from the starting point t0 of the maintenance time to the t moment;

[0039] Step 4, the maintenance benefit cost ratios of various road surface performance evaluation indexes are weighted and summed to obtain the comprehensive benefit ratio E(t) of the road section l at the t moment:

[0040]

[0041] Wherein, w i Is the weight of each road surface performance evaluation index i, and n is the total number of road surface performance evaluation indexes.

[0042] Preferably, the hyperparameters in step S23 include: the number of hidden layers, the number of hidden layer nodes and the Dropout layer.

[0043] The present application has the following advantages and effects relative to the prior art:

[0044] (1) The application introduces a long short-term memory network LSTM neural network which has excellent performance in predicting timing characteristics, is combined with a Rainbow DQN model, considers historical data of road surface performance and parameters affecting road surface performance, more truly simulates the actual environment of road surface use, and greatly improves the accuracy of the prediction result. The LSTM neural network and the Rainbow reinforcement learning method are applied to road surface maintenance management, provide technical support for scientifically formulating a road surface maintenance plan and optimizing maintenance decision-making, improve the comprehensive benefit ratio of road surface maintenance, and reduce resource waste.

[0045] (2) The Rainbow DQN model of the application comprehensively uses various DQN improved algorithms to improve the performance of reinforcement learning, specifically including: a priority experience replay method Prioritized Replay Buffer is used to store experiences of interaction of an agent with an environment, sampling is performed according to importance of samples, and samples that are more helpful to learning are paid more attention to; Q values of each maintenance action are determined by a state value function V and an advantage function H, a target network Target Q-network is introduced to regularly update parameters of a main network, so that the training process is stabilized; a double deep Q network method Double DQN is used to make decisions and evaluation by using two independent neural networks, so that estimation bias is reduced; a multi-step learning method Multi-Step is used to more accurately estimate long-term returns; a value distribution DQN method Distributional DQN uses multiple agents to interact with the environment in parallel; a noise parameter in a Noisy Net is used to speed up learning and strategy exploration. Different from a traditional artificial neural network, the Rainbow DQN reinforcement learning not only pursues minimization of training loss, but also pursues maximization of cumulative returns.

[0046] (3) The road surface maintenance decision-making model of the application simulates a use environment of a road surface (namely, inputs parameters affecting road surface performance), establishes a positive and negative feedback mechanism, gives an agent (a road section needing decision-making) human thinking, takes maximization of long-term maintenance benefit-cost ratio as a decision-making target, constantly “tries and errors-learns” by the agent to optimize and improve a strategy each time, and realizes intelligentization of road surface maintenance decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 It is a flowchart of a road surface maintenance decision-making method based on a Rainbow DQN reinforcement learning of the application.

[0048] Figure 2 It is a flowchart of a method for establishing a road surface use performance prediction model based on an LSTM neural network of the application.

[0049] Figure 3 It is a flowchart of a Rainbow DQN algorithm of the application.

[0050] Figure 4 Schematic diagram of the calculation method of the maintenance benefit of a single pavement performance evaluation index of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0052] Example 1

[0053] like Figure 1 The figure shows a process flow diagram of a pavement maintenance decision-making method based on Rainbow DQN reinforcement learning, including the following steps:

[0054] S1. Collect historical data on pavement performance and parameters affecting pavement performance;

[0055] S2. Training and optimizing the LSTM neural network using the pavement performance historical data and parameters affecting pavement performance, and constructing a pavement performance prediction model based on the LSTM neural network;

[0056] S3. Constructing a pavement maintenance decision model based on Rainbow DQN reinforcement learning based on the pavement performance historical data, parameters affecting pavement performance, and prediction results obtained by the pavement performance prediction model;

[0057] S4. Setting a prediction period, inputting pavement performance data corresponding to a step length and parameters influencing pavement performance, obtaining a prediction result using a pavement performance prediction model, inputting historical pavement performance data, parameters influencing pavement performance, and the prediction result into a pavement maintenance decision model, and outputting a maintenance action for the pavement within the prediction period and the corresponding comprehensive benefit-cost ratio;

[0058] S5. Based on the maintenance actions of the pavement within the predicted life and the corresponding comprehensive benefit-cost ratio, the maintenance action corresponding to the largest comprehensive benefit-cost ratio is selected as the final pavement maintenance plan.

[0059] Specifically, this invention combines an LSTM neural network with Rainbow DQN reinforcement learning to provide a pavement maintenance decision-making method, enabling the efficient application of big data and artificial intelligence technologies in intelligent highway maintenance management. This method comprises two models: the first is an LSTM neural network model, known as a pavement performance prediction model. This model receives basic data on the current pavement status of a specific road section as input and predicts the pavement performance evaluation index of that road section as output; the second is a Rainbow DQN model, known as a pavement maintenance decision-making model. First, an LSTM neural network model is trained by collecting historical pavement performance data and parameters influencing pavement performance. Based on this data, parameters influencing pavement performance, and the prediction results of the pavement performance prediction model, a Rainbow DQN model is trained to generate a pavement maintenance decision model. A prediction period is then set, and pavement performance data and parameters influencing pavement performance are input. The pavement performance prediction model performs actual prediction analysis. After outputting the results, the pavement maintenance decision model learns the results to determine the final maintenance action and comprehensive benefit-cost ratio. Decision-makers then select the maintenance action that maximizes the comprehensive benefit-cost ratio based on their needs. The historical pavement performance data includes various historical performance evaluation indicators. Parameters influencing pavement performance include traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, and highway technical grade. This invention integrates a Long Short-Term Memory (LSTM) neural network, which excels at predicting time series characteristics, with the Rainbow DQN model. This integration takes into account both historical pavement performance data and parameters influencing pavement performance, more realistically simulating the actual pavement usage environment and significantly improving the accuracy of the prediction results.

[0060] The method of the present invention realizes the intelligentization of road maintenance decision-making methods with the goal of minimizing maintenance funds and maximizing the benefit ratio. It applies the LSTM neural network and Rainbow reinforcement learning method to pavement maintenance management, provides technical support for the scientific formulation of pavement maintenance plans and the optimization of maintenance decisions, improves the comprehensive benefit ratio of pavement maintenance, and reduces resource waste.

[0061] Step S2 specifically includes:

[0062] S21. Preprocessing the historical data on road performance and parameters affecting road performance into a training set and a test set, and determining the input parameters and output parameters of the LSTM neural network model;

[0063] S22. Use the Sequential model in the Keras library to build a block model, add an LSTM layer and a fully connected layer in sequence, compile the block model, specify a loss function and an optimizer, train the LSTM neural network model using the training set, and then verify it using the test set to obtain a trained LSTM neural network model, and analyze the prediction results of the trained LSTM neural network model using an error analysis method;

[0064] S23. Use the GridSearchCV method provided by the Sklearn library to adjust the hyperparameters of the trained LSTM neural network model, and use the mean square error (MSE) to perform error analysis to obtain the optimized LSTM neural network model. After the optimization is completed, the road performance prediction model is obtained.

[0065] Specifically, Figure 2 The figure is a flow chart of the method for establishing a road performance prediction model based on an LSTM neural network of the present invention. In this embodiment, the optimizer adopts the Adam optimizer. In order to avoid overfitting, and since there are many parameters to be adjusted in the LSTM neural network, the GridSearchCV method provided by the Sklearn library is used in this embodiment to adjust the hyperparameters of the model. Compared with the traditional manual parameter adjustment, this method overcomes the defects of large workload of manual parameter adjustment and not necessarily obtaining the best parameter combination, and realizes the automation of hyperparameter adjustment. GridSearchCV is divided into two parts: GridSearch grid search and CV cross-validation. The parameters to be tried are arranged and combined, each combination is traversed, and the scores under all parameter combinations are returned by cross-validation. Finally, the parameters corresponding to the combination with the highest score are selected as the optimal value. The mean square error MSE is also used for evaluation during the model optimization process. The hyperparameters described in step S23 include: the number of hidden layers, the number of hidden layer nodes and the Dropout layer. The mean square error MSE is a commonly used indicator for evaluating fitting accuracy, and its formula is as follows:

[0066]

[0067] Among them, y j and represent the true value and the predicted value respectively, and m is the total number of samples.

[0068] In step S21, the input parameters include traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade and pavement performance historical data, and the output parameter is a pavement performance evaluation index, which includes a pavement damage condition index PCI, a pavement ride quality index RQI and a crack rate index CRI.

[0069] Specifically, the pavement performance evaluation index specifically includes the pavement damage condition index PCI, the pavement ride quality index RQI, the pavement rutting index RDI, the pavement bouncing index PBI, the pavement skid resistance index SRI, the pavement wear index PWI, and the pavement structural strength PSSI. The above are commonly used highway technical condition evaluation indicators, and can be selected in combination with the actual conditions of the highway grade and pavement type in the method of the present invention. In this embodiment, when the maintenance object is a rural road, the pavement damage condition index PCI and the pavement ride quality index RQI are selected and the crack rate index CRI is introduced as the pavement performance evaluation index, among which the pavement damage condition index PCI and the pavement ride quality index RQI are commonly used rural road pavement performance evaluation indicators, and the crack rate index CRI is an evaluation indicator introduced by the present invention to improve the rationality and comprehensiveness of the rural road pavement performance evaluation. The expression of the crack rate index CRI is:

[0070]

[0071] Among them, k is the type of pavement crack disease, k0 is the total number of pavement crack disease types, ω k is the total area of ​​k types of diseases in the measured section, ω is the total area of ​​the measured section, and β0 and β1 are model coefficients.

[0072] like Figure 3 The flowchart of the Rainbow DQN algorithm of the present invention is shown, and step S3 specifically includes:

[0073] S31. Determine the definitions of each element in the Rainbow DQN model, including: Agent, State, Reward, Environment, and Action.

[0074] S32. Determine input parameters for the Rainbow DQN model, where the input parameters include historical pavement performance data, parameters affecting pavement performance, and prediction results obtained by the pavement performance prediction model. Preprocess the input parameters and divide them into a training set and a test set. Use Q distribution as output parameters, where the number of output parameters is equal to the number of maintenance actions.

[0075] S33. Use the Sequential model in the Keras library to build a block model, add convolutional layers, fully connected layers, and output layers in sequence, compile the block model, specify the loss function and optimizer, use the training set to train the RainbowDQN model, and then use the test set to verify it to obtain the trained Rainbow DQN model. Use the error analysis method to analyze the prediction results of the trained Rainbow DQN model. After the error analysis is completed, a maintenance decision model based on RainbowDQN reinforcement learning is obtained.

[0076] The Rainbow DQN model uses a combination of various DQN-improved algorithms to improve the performance of reinforcement learning. These include: a Prioritized Replay Buffer method for storing the agent's experience with the environment, sampling based on sample importance and prioritizing samples that are helpful for learning; the Q-value of each maintenance action is determined by the state-value function V and the advantage function H, while introducing a target Q-network to regularly update the parameters of the main network to stabilize the training process; a Double DQN method using two independent neural networks for decision-making and evaluation to reduce estimation bias; a Multi-Step learning method for more accurately estimating long-term rewards; a Distributional DQN method using multiple agents to interact with the environment in parallel; and the use of noise parameters from the Noisy Net to accelerate learning and strategy exploration.

[0077] Specifically, the pavement maintenance decision-making model of the present invention simulates the pavement usage environment (i.e., inputs parameters that affect pavement performance) and establishes a positive and negative feedback mechanism, giving the intelligent agent (the road section requiring decision-making) human thinking, with maximizing the long-term maintenance benefit-cost ratio as the decision-making goal. Through the intelligent agent's continuous "trial and error-learning" to optimize and improve each strategy, the intelligent pavement maintenance decision-making is realized.

[0078] In step S31, the various elements and corresponding definitions in the Rainbow DQN model specifically include: the intelligent agent is the road section that requires maintenance decision-making; the status includes historical pavement performance data, traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade, and pavement performance evaluation index output by the pavement performance prediction model, wherein the pavement performance evaluation index includes the pavement damage condition index PCI, the pavement ride quality index RQI, and the crack rate index CRI; the reward is the corresponding increase in the comprehensive benefit-cost ratio for executing each maintenance action; the environment is the pavement performance prediction; and the action is each maintenance action performed by the intelligent agent.

[0079] Specifically, the present invention uses the comprehensive cost ratio as the reward function in reinforcement learning, and uses maximizing the comprehensive maintenance benefit ratio as a constraint. The determination of the reward function should accurately reflect the actual effectiveness of maintenance. To simplify calculations, in this embodiment, the reward is defined as the increased comprehensive benefit-cost ratio of a road section after each maintenance action. The environment can be considered the external environment of the road. In reinforcement learning, the environment receives the actions currently performed by the agent and provides feedback to the agent on the current instantaneous reward and the specific state at the next moment. In the pavement maintenance decision model of this embodiment, the environment can be described as a prediction of pavement performance.

[0080] Actions correspond to maintenance actions performed by the agent on the road surface. Taking maintenance actions or no maintenance actions is also considered an action. In this embodiment, 10 different maintenance plans are determined based on the survey conclusions, as shown in the following table:

[0081]

[0082]

[0083] The input parameters in step S32 include historical pavement performance data, traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade, and pavement performance evaluation index output by the pavement performance prediction model, wherein the pavement performance evaluation index includes a pavement damage index (PCI), a pavement ride quality index (RQI), and a crack rate index (CRI).

[0084] The preprocessing includes: for numerical values, using the Max-Min method in z-score standardization to limit the range of variation to [0, 1]; for categorical data, using one-hot encoding.

[0085] Specifically, the data required for the input parameters are preprocessed. The data includes numerical and categorical data, which need to be preprocessed separately because most of the samples involved in training are future prediction indicator values. There is a big difference between the numerical distribution of each input parameter in the existing data and the actual distribution in the subsequent training samples. Therefore, the Max-Min method in the z-score standardization is used to limit the range of variation to [0, 1]. For categorical data, such as discrete data features such as road technical grades, one-hot encoding is required.

[0086] In step S33, the use of the training set to train the Rainbow DQN model specifically includes:

[0087] S01, the target Q-network established has the same structure as the Q-network, and the Q-network is used to select the action a at time t. t , using the Noisy DQN exploration algorithm, that is, adding noise Noisy to the network and outputting the action with the largest Q value; Target Q-network is used to update the action a at time t+1 t+1 , using the Noisy DQN exploration algorithm, that is, adding noise Noisy to the network;

[0088] S02, after obtaining the output values of the Target Q-network and the Q-network and the reward R of the environmental feedback, a network loss function is calculated and the parameters of the Rainbow DQN model are updated, with the increase of the Epoch of training, the value of the network loss function tends to be stable, when the preset error is reached, the training is completed, and the Rainbow DQN model meeting the accuracy requirement is obtained.

[0089] The method for calculating the comprehensive benefit-cost ratio specifically comprises the following steps:

[0090] Step 1: after selecting a maintenance action a for a road section l, output the pavement performance evaluation index i at the next moment through a performance prediction model, the pavement performance evaluation index i comprises a pavement damage condition index PCI, a pavement riding quality index RQI and a crack rate index CRI, and a change curve of the pavement performance evaluation index is obtained;

[0091] Step 2: calculate the maintenance benefit Eff(t) of the road section l with the pavement performance evaluation index i at the moment t, wherein the maintenance benefit at the moment t is defined as the area surrounded by the change curve of the pavement performance after maintenance, the change curve of the pavement performance before maintenance and T=t0 and T=t, and is calculated by the following formula:

[0092]

[0093] wherein, respectively, the change curve of the pavement performance after maintenance and the change curve of the pavement performance before maintenance, and l represents the road section number;

[0094] Step 3: calculate the maintenance benefit-cost ratio cost(t) of the road section l with the pavement performance evaluation index i at the moment t:

[0095]

[0096] wherein, C(t) is the maintenance cost of the road section l from the starting point t0 of the maintenance moment to the moment t;

[0097] Step 4: weight and sum the maintenance benefit costs of all pavement performance evaluation indexes to obtain the comprehensive benefit ratio E(t) of the road section l at the moment t:

[0098]

[0099] wherein, w i is the weight of the pavement performance evaluation index i, and n is the total number of the pavement performance evaluation indexes.

[0100] Specifically, as Figure 4The shaded area shown is the maintenance benefit. That is, the maintenance benefit at time t is defined as the area enclosed by the pavement performance change curve after maintenance, the pavement performance change curve before maintenance, and T = t0 and T = t. The weights of the pavement condition index (PCI), the pavement ride quality index (RQI), and the crack rate index (CRI) in the pavement performance evaluation index are set based on the actual pavement conditions. In this embodiment, the weights of the PCI, RQI, and CRI are 0.6, 0.3, and 0.1, respectively. When the PCI, RQI, and CRI are selected as the pavement performance evaluation index, then n = 3. The weighted sum of the maintenance benefit-cost ratios of the PCI, RQI, and CRI indices yields the comprehensive benefit ratio E(t) for road section l at time t:

[0101]

[0102] The implementation principle of the present invention includes combining an LSTM neural network with Rainbow DQN reinforcement learning to provide a pavement maintenance decision-making method, which mainly includes two models. The first part is an LSTM neural network model, namely a pavement usage performance prediction model, which receives the current basic data of a certain road section as an input parameter, and then predicts the pavement performance index of the road as an output parameter. By simultaneously selecting different maintenance actions, namely maintenance plans, for the same road section, different pavement performance indicators at the next moment are obtained, the pavement performance indicators and maintenance costs after maintenance are recorded, and the maintenance benefit-cost ratio of the pavement after maintenance is calculated based on the maintenance costs required for selecting different maintenance plans. The maintenance plan with the largest maintenance benefit-cost ratio is selected as the maintenance plan that should be used currently. Maintenance is performed at time intervals of t. The pavement performance indicators, maintenance costs, and the pavement maintenance benefit-cost ratio after maintenance are recorded each time. After N maintenance operations, the cumulative pavement maintenance benefit-cost ratio is obtained. Finally, a pavement maintenance decision model combining an LSTM neural network and Rainbow DQN reinforcement learning is used to select the plan with the largest cumulative maintenance benefit-cost ratio as the optimal maintenance plan. The pavement performance indicators and cumulative maintenance costs after N maintenance operations are then obtained.

[0103] The specific process of making a decision on a road section of the present invention is as follows:

[0104] Step 01: pre-process the pavement performance data over the years and arrange the pavement performance data over the years into a matrix pattern according to the pavement performance evaluation index i (i = 1, 2, ..., n) and time t to obtain the pavement performance data sequence: X 1,t1 ,X 2,t1 ,…,X n,t1 ,…,X 1,tz ,X2,tz ,…,X n,tz ;

[0105] Step 02: randomly divide the sample data into a training set and a test set in a ratio of 7:3;

[0106] Step 03: Use the training set to train the model, set the iteration step to 1 year, and use the data X of the first year in the existing road condition data series to train the model. 1,t1 ,X 2,t1 ,…,X n,t1 To predict the road usage evaluation index X in the second year (n,t2) , and then use the road condition data sequence X of the second year 1,t2 ,X 2,t2 ,…,X n,t2 To predict the road condition evaluation index X in the third year (n,t3) In this way, all training set data are traversed until all the road performance data in the training set have been trained for prediction. Then, the training is stopped and the function relationship X is determined. (n,t(z+1)) =f(X 1,tz ,X 2,tz ,…,X n,tz ), and then the model is repeatedly tested through the optimizer, and the error analysis of the prediction effect of the prediction model is performed using the mean square error MSE as the evaluation index;

[0107] Step 4: Use the GridSearchCV method provided by the Sklearn library to adjust the model's hyperparameters, and also use the mean square error (MSE) as the evaluation metric for error analysis.

[0108] Step 05, save the LSTM neural network model;

[0109] Step 6: Use the trained LSTM neural network model to train the RainbowDQN model and predict the road condition data for the past year. The steps are as follows:

[0110] Step 61: All input parameters are used as network input parameters after PCA dimension reduction, and Q distribution is used as output parameters. The number of output parameters is equal to the number of maintenance actions.

[0111] Step 62, Q-network is used to select action a at time t t , using the NoisyDQN exploration algorithm, that is, adding noise Noisy to the network, where the Target Q-network does not need to be trained, and its parameters are completely copied from the Q-network at intervals. The LSTM neural network model is used to predict the action a at time t tAfter that, the road condition index at the next time t+1, Target Q-network updates the action a at time t+1 t+1 , and the same NoisyDQN exploration algorithm is used as Q-network, and then Q-network outputs the action with the largest Q value.

[0112] Step 63, record the return r t , that is, the comprehensive benefit-cost ratio of the road section after each maintenance action;

[0113] Step 64: When the output values ​​of Q-network and Target Q-network and the feedback from the environment are obtained, t After that, the network Loss is calculated and the model parameters are updated. As the number of training iterations Epoch increases, the loss function Loss gradually tends to a stable value.

[0114] Step 65: Output the maintenance plan and comprehensive benefit-cost ratio of each road section within the predicted period.

[0115] In step 66, when the loss is stable and lower than the required error value, the training is completed, and a Rainbow DQN model that meets the accuracy requirements is obtained.

[0116] The above embodiments are preferred implementations of the present invention and are not intended to limit the present invention. Any other changes or other equivalent replacements that do not deviate from the technical solutions of the present invention are included in the protection scope of the present invention.

Claims

1. A pavement maintenance decision-making method based on Rainbow DQN reinforcement learning, characterized by: Including steps: S1. Collect historical data on pavement performance and parameters affecting pavement performance; S2. Training and optimizing the LSTM neural network using the historical pavement performance data and parameters affecting pavement performance, and constructing a pavement performance prediction model based on the LSTM neural network; S3. Constructing a pavement maintenance decision model based on Rainbow DQN reinforcement learning based on the pavement performance historical data, parameters affecting pavement performance, and prediction results obtained by the LSTM pavement performance prediction model; S4. Setting a prediction period, inputting historical pavement performance data corresponding to a step length and parameters influencing pavement performance, obtaining a prediction result using a pavement performance prediction model, inputting the historical pavement performance data, parameters influencing pavement performance, and the prediction result into a pavement maintenance decision model, and outputting a maintenance action for the pavement within the prediction period and a corresponding comprehensive benefit-cost ratio; S5. Based on the maintenance actions and corresponding comprehensive benefit-cost ratios within the predicted lifespan, select the maintenance action with the highest comprehensive benefit-cost ratio as the final pavement maintenance plan; Step S3 specifically includes: S31. Determine the definitions of each element in the Rainbow DQN model, including: Agent, State, Reward, Environment, and Action. S32. Determine input parameters for the RainbowDQN model, where the input parameters include historical pavement performance data, parameters affecting pavement performance, and prediction results obtained by the pavement performance prediction model. Preprocess the input parameters and divide them into a training set and a test set. Use Q distribution as output parameters, where the number of output parameters is equal to the number of maintenance actions. S33. Use the Sequential model in the Keras library to build a block model, add convolutional layers, fully connected layers, and output layers in sequence, compile the block model, specify the loss function and optimizer, train the Rainbow DQN model using the training set, and then use the test set for verification to obtain the trained Rainbow DQN model. Use the error analysis method to analyze the prediction results of the trained Rainbow DQN model. After the error analysis is completed, a maintenance decision model based on Rainbow DQN reinforcement learning is obtained. The Rainbow DQN model uses a combination of various DQN-improved algorithms to improve the performance of reinforcement learning. These include: a Prioritized Replay Buffer method for storing the agent's experience with the environment, sampling based on sample importance and prioritizing samples that are helpful for learning; the Q-value of each maintenance action is determined by the state-value function V and the advantage function H, while introducing a target Q-network to regularly update the parameters of the main network to stabilize the training process; a Double DQN method using two independent neural networks for decision-making and evaluation to reduce estimation bias; a Multi-Step learning method for more accurately estimating long-term rewards; a Distributional DQN method using multiple agents to interact with the environment in parallel; and the use of noise parameters from the Noisy Net to accelerate learning and strategy exploration.

2. A pavement maintenance decision-making method based on Rainbow DQN reinforcement learning according to claim 1, characterized in that: Step S2 specifically includes: S21. Preprocessing the historical data on road performance and parameters affecting road performance into a training set and a test set, and determining the input parameters and output parameters of the LSTM neural network model; S22. Use the Sequential model in the Keras library to build a block model, add an LSTM layer and a fully connected layer in sequence, compile the block model, specify a loss function and an optimizer, train the LSTM neural network model using the training set, and then verify it using the test set to obtain a trained LSTM neural network model, and analyze the prediction results of the trained LSTM neural network model using an error analysis method; S23. Use the GridSearchCV method provided by the Sklearn library to adjust the hyperparameters of the trained LSTM neural network model, and use the mean square error (MSE) to perform error analysis to obtain the optimized LSTM neural network model. After the optimization is completed, the road performance prediction model is obtained.

3. The pavement maintenance decision-making method based on RainbowDQN reinforcement learning according to claim 2 is characterized in that: In step S21, the input parameters include traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade and pavement performance historical data, and the output parameter is a pavement performance evaluation index, which includes a pavement damage condition index PCI, a pavement ride quality index RQI and a crack rate index CRI.

4. The pavement maintenance decision-making method based on Rainbow DQN reinforcement learning according to claim 3 is characterized in that: The expression of the crack rate index CRI is: Among them, k is the type of pavement crack disease, k0 is the total number of pavement crack disease types, ω k is the total area of ​​k types of diseases in the measured section, ω is the total area of ​​the measured section, and β0 and β1 are model coefficients.

5. The pavement maintenance decision-making method based on Rainbow DQN reinforcement learning according to claim 1 is characterized in that: In step S31, the various elements and corresponding definitions in the Rainbow DQN model specifically include: the intelligent agent is the road section that requires maintenance decision-making; the status includes historical pavement performance data, traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade, and pavement performance evaluation index output by the pavement performance prediction model, wherein the pavement performance evaluation index includes the pavement damage condition index PCI, the pavement ride quality index RQI, and the crack rate index CRI; the reward is the corresponding increase in the comprehensive benefit-cost ratio for executing each maintenance action; the environment is the pavement performance prediction; and the action is each maintenance action performed by the intelligent agent.

6. The pavement maintenance decision-making method based on Rainbow DQN reinforcement learning according to claim 1 is characterized in that: The input parameters in step S32 include historical pavement performance data, traffic volume, road age, surface layer type, surface layer thickness, base layer type, base layer thickness, highway technical grade, and pavement performance evaluation index output by the pavement performance prediction model, wherein the pavement performance evaluation index includes a pavement damage index (PCI), a pavement ride quality index (RQI), and a crack rate index (CRI). The preprocessing includes: for numerical values, using the Max-Min method in z-score standardization to limit the range of variation to [0, 1]; for categorical data, using one-hot encoding.

7. The pavement maintenance decision-making method based on Rainbow DQN reinforcement learning according to claim 1 is characterized in that: In step S33, the use of the training set to train the Rainbow DQN model specifically includes: S01, the target Q-network established has the same structure as the Q-network, and the Q-network is used to select the action a at time t. t , using the Noisy DQN exploration algorithm, that is, adding noise Noisy to the network and outputting the action with the largest Q value; Target Q-network is used to update the action a at time t+1 t+1 , using the Noisy DQN exploration algorithm, that is, adding noise Noisy to the network; S02. After obtaining the output values ​​of the Target Q-network and Q-network and the reward R of the environment feedback, the network loss function is calculated and the parameters of the Rainbow DQN model are updated. As the number of training epochs increases, the value of the network loss function tends to stabilize. When the preset error is reached, the training is completed, and a Rainbow DQN model that meets the accuracy requirements is obtained.

8. The pavement maintenance decision-making method based on Rainbow DQN reinforcement learning according to claim 5 is characterized in that: The calculation method of the comprehensive benefit-cost ratio specifically includes: Step 1: After selecting a maintenance action a for road section l, the performance prediction model outputs the pavement performance evaluation index i at the next moment. The pavement performance evaluation index i includes the pavement damage index PCI, the pavement ride quality index RQI, and the crack rate index CRI, and a pavement performance evaluation index change curve is obtained; Step 2: Calculate the maintenance benefit Eff(t) of road section l at time t using the pavement performance evaluation index i. The maintenance benefit at time t is defined as the area enclosed by the pavement performance change curve after maintenance, the pavement performance change curve before maintenance, and T = t0 and T = t, and is calculated using the following formula: in, They are the pavement performance change curves after maintenance and before maintenance, and l represents the road section number; Step 3: Calculate the maintenance benefit-cost ratio cost(t) of road section l at time t using the pavement performance evaluation index i: Where C(t) is the maintenance cost of road section l from the maintenance starting time t0 to time t; Step 4: Take the weighted sum of the maintenance benefit costs of each pavement performance evaluation index to obtain the comprehensive benefit ratio E(t) of road section l at time t: Among them, w i is the weight of each pavement performance evaluation index i, and n is the total number of pavement performance evaluation indices.

9. The pavement maintenance decision-making method based on Rainbow DQN reinforcement learning according to claim 2 is characterized in that: The hyperparameters in step S23 include: the number of hidden layers, the number of hidden layer nodes, and the Dropout layer.

Images