Road surface performance prediction and maintenance decision optimization method based on machine learning

Abstract

The invention relates to a machine learning-based pavement performance prediction and maintenance decision optimization method, which comprises the following steps of: quantifying carbon emission cost, traffic delay cost and economic cost of asphalt pavement maintenance, and obtaining pavement life prediction age limit, asphalt pavement service data and environmental parameters of a road section to be detected; and through the asphalt pavement service performance prediction model and the asphalt pavement maintenance decision model, obtaining the optimal maintenance measure of the road section and the benefit-cost ratio of considering the carbon emission and the delay cost, cyclically carrying out the above process, and updating the maintenance measure. Compared with the prior art, the method is suitable for decision analysis of the asphalt pavement maintenance problem with a multi-source heterogeneous asphalt pavement data set needing to consider carbon emission and delay cost, a scientific pavement maintenance decision can be quickly generated, the maintenance efficiency and the resource utilization rate are improved, and traffic safety and smoothness are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent highway maintenance, and particularly to a method for predicting pavement performance and optimizing maintenance decisions based on machine learning. Background Art

[0002] In recent years, the total mileage of highways has increased rapidly. However, with the continuous increase in traffic volume, the overage service of highways, and the frequent occurrence of extreme weather, various performance indicators of asphalt pavements have been continuously declining under the combined action of various factors such as traffic loads and climatic conditions. Pavement diseases such as ruts, potholes, and cracks occur frequently, seriously affecting driving safety and comfort. In addition, highway construction and maintenance account for a large proportion of carbon emissions in the entire transportation industry. Therefore, it is necessary to comprehensively consider multiple factors such as the remaining pavement performance, carbon emission benefits of maintenance, maintenance delay costs, and maintenance funds to make scientific and intelligent maintenance decisions for asphalt pavements, so as to extend the service life of the pavement, reduce carbon emissions, and reduce capital consumption, and achieve the sustainable development of the pavement.

[0003] Regarding the maintenance decision-making technology for asphalt pavements, in the early stage, it mostly relied on professionals to make judgments based on engineering practice experience, which was greatly affected by subjective factors. With the development of computer technology, mathematical programming methods, decision trees or ranking methods based on expert experience are now mostly used. However, when facing large-scale road network-level problems, the solution speed of the mathematical programming method is very slow; the method based on expert experience is usually affected by the subjective factors of experts. In the optimization of pavement maintenance decisions, the reinforcement learning framework mostly uses value-based model algorithms. However, traditional reinforcement learning models have the problem of slow convergence speed. Chinese Patent Application CN118212103A discloses a pavement maintenance decision-making method based on Rainbow DQN reinforcement learning. The pavement performance is predicted through an LSTM neural network model based on the collected pavement historical data, and the predicted future pavement performance indicators are input into the reinforcement learning pavement maintenance decision-making model based on Rainbow DQN. The maintenance plan is determined according to the benefit-cost ratio, which can assist in formulating scientific and reasonable maintenance decision-making plans. However, in this method, the cleaning and correction of historical data are lacking, and it is difficult to ensure the accuracy of historical data; the pavement performance prediction model based on the LSTM neural network cannot fully utilize the correlation and time series of time series data, and the prediction accuracy of the network model needs to be further improved; the optimization goal lacks comprehensive consideration of the carbon emissions of maintenance measures and maintenance delay costs, and the evaluation index system is not sufficient; the decision-making algorithm can only make a decision on the pavement maintenance measures once, and lacks the decision on the optimal overall maintenance benefit in the long-term service life of the pavement.

[0004] Therefore, it is necessary to design an asphalt pavement maintenance decision-making method based on the proximal policy optimization method that considers the comprehensive benefits of carbon emissions, to assist in designing multiple maintenance decisions for asphalt pavements under long-term service life, and to reduce the comprehensive costs such as carbon emissions and delays while ensuring that the pavement maintains good service conditions. Summary of the Invention

[0005] The purpose of the present invention is to provide a pavement performance prediction and maintenance decision-making optimization method based on machine learning to overcome the defects of the above-mentioned existing technologies.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A pavement performance prediction and maintenance decision-making optimization method based on machine learning, the method includes:

[0008] Quantify the carbon emission cost, traffic delay cost and economic cost of asphalt pavement maintenance, obtain the pavement life prediction years, asphalt pavement service data and environmental parameters of the section to be measured, and through the asphalt pavement service performance prediction model and the asphalt pavement maintenance decision model, obtain the optimal maintenance measures for the section and their benefit-cost ratios considering carbon emissions and delay costs, and cycle the above process to update the maintenance measures, where,

[0009] The asphalt pavement service performance prediction model is an LSTM neural network model introducing the Attention mechanism, the asphalt pavement service performance prediction model is trained through the asphalt pavement data set, and the asphalt pavement maintenance decision model is optimized through the proximal policy optimization algorithm clipped by Clip.

[0010] Further, the construction process of the asphalt pavement data set includes:

[0011] Collect historical asphalt pavement heterogeneous data, including section number, inspection year, DR, PCI value, service life, annual rainfall, high temperature index, freezing index, existing maintenance number and maintenance thickness, AADTT, kESAL, surface layer thickness, base layer thickness, base layer and subbase layer types;

[0012] Perform outlier screening and data splicing processing on the asphalt pavement heterogeneous data to construct an asphalt pavement data set.

[0013] Further, the training process of the asphalt pavement service performance prediction model includes:

[0014] Preprocess the asphalt pavement data set and divide it into a training set and a test set;

[0015] Construct the asphalt pavement service performance prediction model;

[0016] Train the service performance prediction model of the asphalt pavement using the training set;

[0017] Verify the trained service performance prediction model of the asphalt pavement using the test set;

[0018] Use R 2 Evaluate the prediction effect of the service performance prediction model of the asphalt pavement using the evaluation index. If the evaluation result is lower than the threshold, repeat the above training and prediction; otherwise, output the service performance prediction model of the asphalt pavement.

[0019] Furthermore, the preprocessing includes:

[0020] Divide the data in the asphalt pavement dataset into maintenance measures, base course types, sub-base course types, and numerical data, and normalize the asphalt pavement dataset;

[0021] Perform one-hot encoding on the maintenance measures, base course types, and sub-base course types data, and perform min-max scaling on the numerical data to complete the preprocessing.

[0022] Furthermore, the construction process of the service performance prediction model of the asphalt pavement includes:

[0023] Build the basic structure of the service performance prediction model of the asphalt pavement, including an input layer, an LSTM layer, a fully connected layer, and a Self-Attention layer;

[0024] Determine the input and output of the service performance prediction model of the asphalt pavement according to the preprocessed asphalt pavement dataset to complete the construction of the service performance prediction model of the asphalt pavement, where the output is the PCI value of the asphalt pavement.

[0025] Furthermore, the loss function in the training of the service performance prediction model of the asphalt pavement is:

[0026]

[0027] where LOSS is the model loss function, RMSE is the root mean square error, y i is the measured value of the i-th sample, is the predicted value of the i-th sample, and n is the total number of data points.

[0028] Furthermore, the expression of the R 2 evaluation index is:

[0029]

[0030] where R 2 is the coefficient of determination, y i is the measured value of the i-th sample, is the average value of the samples, is the predicted value of the i-th sample.

[0031] Furthermore, the quantification process of the carbon emission cost, traffic delay cost and economic cost of asphalt pavement maintenance includes:

[0032] Obtain asphalt pavement maintenance measures, typical maintenance material parameters and transportation parameters of asphalt pavement, and determine the boundary range of assumed parameters;

[0033] Obtain the data of the preset carbon emission factor library, and calculate the carbon emissions and their costs in the raw material acquisition, transportation and construction stages according to the boundary range of the assumed parameters;

[0034] Based on the highway machinery work shift cost quota, calculate the traffic delay time and cost caused by maintenance construction according to the boundary range of the assumed parameters;

[0035] Integrate the carbon emission cost and traffic delay cost to form a comprehensive cost index for each maintenance measure.

[0036] Even further, the asphalt pavement maintenance decision model is constructed based on the proximal policy optimization algorithm, its framework is a Markov decision process framework, and the benefit-cost ratio of carbon emissions and delay costs is calculated based on the comprehensive cost index of each maintenance measure. The calculation formula of the benefit-cost ratio is:

[0037]

[0038] where B i is the change in performance of the road section after the maintenance measure i, C i is the cost corresponding to the i-th maintenance measure, and S is the PCI value of the road surface output by the asphalt pavement service performance prediction model.

[0039] Even further, the asphalt pavement maintenance decision model uses the Clip clipping technique to limit the update amplitude of the proximal policy optimization algorithm. The optimization objective function of the proximal policy optimization algorithm is:

[0040]

[0041] where p θ (a t ∣s t ) is the probability of selecting the action a t in the state s t according to the policy θ, and p θ ′(a t ∣s t ) is the probability of selecting the action a t in the state s t according to the old policy θ′, and Aθ′ (s t , a t ) is the advantage function, that is, the additional value of taking action a t under state s t relative to the average case, is the clipping function, which limits the value of x between 1 - ε and 1 + ε, where ε is a hyperparameter.

[0042] Compared with the prior art, the beneficial effects of the present invention include:

[0043] 1. The present invention can quickly generate scientific pavement maintenance decision measures at the section level within a certain number of years through the asphalt pavement service performance prediction model and the asphalt pavement maintenance decision model. At the same time, it also considers the carbon emission cost and traffic delay cost, improves the maintenance efficiency and resource utilization rate, and ensures smooth traffic safety;

[0044] 2. The present invention introduces the Attention mechanism into the LSTM neural network model, makes full use of the correlation and time series of time series data, improves the pavement performance prediction model, and effectively improves the pavement performance prediction accuracy;

[0045] 3. The present invention quantitatively evaluates the carbon emissions and maintenance delays caused by typical maintenance measures, calculates the comprehensive carbon emission cost and maintenance delay cost, and expands the benefit range in the benefit-cost ratio of maintenance decision-making;

[0046] 4. The present invention constructs a proximal policy optimization asphalt pavement maintenance decision model with Clip clipping for reinforcement learning, optimizes the policy function through gradient ascent, ensures the similarity of importance sampling results, and improves the convergence speed and average reward value of the decision model;

[0047] 5. When preprocessing asphalt pavement data, the present invention cleans and corrects the pavement performance (PCI value) and data of four types of influencing factors, improves the quality of the asphalt pavement performance data set, and helps to improve the applicability of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is the flowchart of the method of the present invention;

[0049] Figure 2 is the graph of randomly searching hyperparameter combinations in an embodiment of the present invention;

[0050] Figure 3 is the structural diagram of the LSTM network with the Self-Attention mechanism in an embodiment of the present invention;

[0051] Figure 4 is the graph of the carbon emission measurement range in an embodiment of the present invention;

[0052] Figure 5 Flow chart of the proximal policy optimization model iteration with Clip clipping in an embodiment of the present invention;

[0053] Figure 6 Line graph comparing PCI of sections where the method of the present invention is used to determine maintenance measures and sections where maintenance measures are not used in an embodiment of the present invention;

[0054] Figure 7 Maintenance measure diagram determined by the method of the present invention in an embodiment of the present invention. Detailed implementation manner

[0055] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0056] Embodiment 1

[0057] This embodiment aims to provide a pavement performance prediction and maintenance decision optimization method based on machine learning. The process of the method is as Figure 1 shown and includes:

[0058] S1. Collect multi-source heterogeneous data of asphalt pavement and perform preprocessing to construct a high-quality asphalt pavement data set;

[0059] S2. Obtain the time series features of pavement service data through the Attention mechanism, and construct and train an LSTM neural network asphalt pavement service performance prediction model incorporating the Attention mechanism;

[0060] S3. Calculate the carbon emissions of typical maintenance measures and calculate the construction and transportation machinery shifts, and quantify the comprehensive cost-benefit of asphalt pavement maintenance carbon emissions and traffic delays;

[0061] S4. Based on the content of steps S1 - S3, introduce a proximal policy optimization algorithm with a clipping operation, combine the LSTM neural network asphalt pavement service performance prediction model and the quantification result of the comprehensive cost-benefit of asphalt pavement maintenance carbon emissions and traffic delays, build the core environment for interacting with the agent in reinforcement learning, set a reasonable benefit evaluation function, and construct an asphalt pavement maintenance decision model with proximal policy optimization of reinforcement learning with Clip clipping;

[0062] S5. Set the pavement life prediction period for the road section to be measured, input the service data and environmental parameters of the asphalt pavement, and output the maintenance measures for this road section within a certain period and the benefit-cost ratio considering carbon emissions and delay costs through the asphalt pavement service performance prediction model and the asphalt pavement maintenance decision-making model;

[0063] S6. Select the maintenance measure with the optimal comprehensive benefit-cost ratio and cycle through the decision-making of long-term asphalt pavement maintenance measures.

[0064] Step S1 includes the following sub-steps:

[0065] S101. Collect and integrate various types of heterogeneous data such as pavement inspection vehicle data, camera data, temperature sensors, humidity sensors, and manual record files of high-grade asphalt pavements in a certain area, and construct a typical database containing data such as province, road section number, inspection year, DR, PCI, service life, annual rainfall, high temperature index, freezing index, existing maintenance number and maintenance thickness, AADTT, kESAL, surface layer thickness, base layer thickness, base layer, and sub-base layer type;

[0066] S102. Perform outlier screening and data splicing processing on the typical database, and construct a high-quality (meeting the decay trend, large quantity, and complete variety) asphalt pavement dataset for pavement performance prediction through data preprocessing means such as outlier screening, data patching, and data splicing, ensuring the generalization ability (applicability) of the network model at the data level. Among them, the data processing principles are that the road has not been maintained and the PCI value suddenly rises; the road has not been maintained and the PCI suddenly drops by more than 20 points; the long-time series data is spatially and temporally aligned and spliced.

[0067] Step S2 includes the following sub-steps:

[0068] S201. Preprocess the asphalt pavement dataset in Step S1 to determine 35 inputs and 1 output of the model,

[0069] Among them, the preprocessing includes one-hot encoding and Min-Max Scaling. The data for one-hot encoding are the maintenance measures, base course types, and sub-base course types in the asphalt pavement dataset. The maintenance measures correspond to 7 numbers, the base course types correspond to 8 numbers, and the sub-base course types correspond to 7 types (although there are only four sub-base course types, the material types corresponding to their values need to be consistent with the base course material types. There are a total of 8 base course material types, and the largest number corresponding to the sub-base course material is 6, so the sub-base course types correspond to 7 types). Finally, 22-dimensional data are obtained to represent these three types of characteristic data. The remaining data in the asphalt pavement dataset are numerical data, and these data are preprocessed by Min-Max Scaling. Min-Max Scaling preprocesses the data through linear transformation, so that the original values are mapped to the range of [0,1] through linear transformation, and at the same time, the correlation between features can be retained. The formula for Min-Max Scaling is:

[0070]

[0071] Among them, x′ is the feature value after min-max scaling, x is a certain feature value in the dataset, minA is the minimum value of this feature in the dataset, and maxA is the maximum value of this feature in the dataset;

[0072] S202. Build a block model (the basic model of the asphalt pavement service performance prediction model), add an LSTM layer and a fully connected layer to the model, specify the RMES loss function and the cost function optimizer of the model, use the random search module to perform hyperparameter search, and select the R 2 evaluation index as the evaluation index of the prediction model, and initially complete the construction of the asphalt pavement service performance prediction model.

[0073] Among them, the loss function is:

[0074]

[0075] Among them, LOSS is the model loss function, RMSE is the root mean square error, y i is the measured value of the i-th sample, is the predicted value of the i-th sample, and n is the total number of data points;

[0076] R 2 The expression of the evaluation index is:

[0077]

[0078] Among them, R 2 is the coefficient of determination, y i is the measured value of the i-th sample, is the average value of the samples, is the predicted value for the i-th sample;

[0079] The cost function optimizer transforms the machine learning problem into an optimization problem: minimizing the cost function J(θ) on the training set (assuming that the empirical distribution on the training set can represent the actual true empirical distribution), and optimizing for the cost function J(θ) can indirectly maximize the evaluation metric R 2 as large as possible. The cost function J(θ) is the average of the loss function on the training set, and its expression is:

[0080] J(θ) = E (x)～p L(f(x, θ), y),

[0081] where J(θ) is the cost function, E (x)～p L(θ) is the average of the loss function on the training set, f(x, θ) is the prediction function, i.e., the neural network model, and y is the actual value of the sample;

[0082] As Figure 2 shown, the random search of hyperparameter search using the random search module randomly extracts a certain number of hyperparameters from the specified hyperparameter space for training and evaluation, which is more efficient than grid search;

[0083] S203. Introduce the Self-Attention mechanism into the preliminarily constructed asphalt pavement service performance prediction model to improve the LSTM (Long Short-Term Memory) network prediction model, enabling the model to more completely record the characteristics of time series data. Its network structure is as Figure 3 shown;

[0084] The LSTM model can effectively alleviate the gradient vanishing problem of common RNNs, but it still needs to be improved in capturing the time series characteristics of all key data. Introducing the self-attention mechanism (Self-Attention Mechanism), through the context recognition vector, when predicting the next state each time, it will look back at all previous states, which can effectively capture the key information in time series data and make the pavement performance prediction more accurate, as shown in the following formula:

[0085] h t = f(W · [x t , c t-1 + b),

[0086] where h t is the hidden state at time step t, W is the weight matrix, x t is the input vector at time step t, c t-1 is the hidden state at time step t - 1, and b is the bias term.

[0087] c t-1 is the context recognition vector (ContextVector), which is the weighted average of all previous moments (including the vector h at this moment). Figure 3 is the network structure of LSTM + Self-Attention. According to Figure 3 calculate, where α t is h t-1 and the correlation coefficient of h at each previous moment t . The network update method with the context recognition vector is as shown in Figure 3 . Regarding α t , it represents the correlation between the current h t and the corresponding h at this moment and all previous moments i . The calculation is carried out according to the following process:

[0088] (i) Linear maps: k i = W k ·h t , for i = 1 to m; q0 = W Q ·h t ,

[0089] (ii) Inner product: for i = 1 to m,

[0090] (iii) Normalization:

[0091] It can be seen from Figure 3 that LSTM + Self-Attention can better capture the key information of all previous states. Figure 3 In, X1, X2, X3 represent the input parameters of the network structure. In this method, X contains 35 parameters. F represents the FCNN neural network, L represents LSTM, and AM represents the self-attention mechanism. The arrow indicates the direction of forward propagation of data. The core of this step is to improve the LSTM model by introducing the Self-Attention mechanism into the LSTM network to further improve the accuracy of the pavement performance prediction model.

[0092] Step S3 includes the following sub-steps:

[0093] S301. Collect the typical maintenance material parameters and transportation parameters of asphalt pavement, and determine the calculation boundary of carbon emissions, mainly including raw material acquisition, transportation, and construction stages. Among them, the parameters in the construction stage are mainly energy consumption;

[0094] S302. Determine the authoritative carbon emission factors for maintenance measures based on various authoritative databases, determine the main energy consumption data during the construction process through the "Highway Engineering Budget Quota" and the "Highway Engineering Machinery Shift Cost Quota", and consider greenhouse gas emissions to conduct a comprehensive cost evaluation (including carbon emission costs, delay costs, and construction costs) for three typical maintenance measures (micro-surfacing, overlay, and milling and overlay) to provide comprehensive cost indicators for the subsequent decision-making optimization model;

[0095] The calculation scope of carbon emissions from maintenance measures mainly includes the acquisition of raw materials and the corresponding carbon emissions from the energy consumption of mechanical equipment. The specific boundaries are as Figure 4 shown. The quota method is used to calculate the carbon emissions during the maintenance construction process, which can ensure the consistency and comparability of the corresponding carbon emission results. The specific carbon emission calculation process is as Figure 4 shown;

[0096] S303. In addition, the additional user costs caused by maintenance measures are also considered, which are mainly calculated in the following way: Delay time = (length of the work area / speed in the work area) - (length of the work area / speed of the passing vehicle); Delay cost = total delay time * annual average daily traffic volume * number of working days in the work area * time value;

[0097] Based on the above calculations, it is known that for the four maintenance measures of micro-surfacing, 4 cm AC overlay, 4 cm SMA overlay, and 4 cm milling and overlay, the comprehensive costs are 10 yuan / (m²), 50 yuan / (m²), 66 yuan / (m²), and 60 yuan / (m²) respectively. For overlay maintenance measures, the carbon emissions during the mixing period can reach more than 40%, and even exceed 50%. Regarding the carbon emission cost, its proportion in the total cost is less than 1%, and the impact on the comprehensive maintenance cost is very small.

[0098] The comprehensive cost indicators corresponding to each maintenance measure calculated in step S3 will also be used for the calculation of maintenance benefits in the pavement maintenance decision model in step S4.

[0099] Step S4 includes the following sub-steps:

[0100] S401. Design a policy gradient algorithm that can calculate the gradient of the expected reward, so as to optimize the policy function through gradient ascent;

[0101] The problem to be solved by the reinforcement learning framework is the finite Markov decision process. "Finite" means that the states, actions, and rewards of this process are finite. The Markov decision process adds a reward function and decision-making actions on the basis of the Markov process. Usually, this Markov decision process is defined as four parts: S, A, P, and R. Among them, S is the state set, A is the action set, P represents the state transition probability, and R represents the obtained return (reward). The goal of the Markov decision process is to find a strategy that can maximize the cumulative reward through the interaction between the agent and the environment. As Figure 6 shown, the Markov decision process can be represented by a four-parameter function p, and its expression is:

[0102] p(s′,r|s,a) = p(S t = s′,R t = r|S t-1 = s,A t-1 = a),

[0103] where p(s′,r|s,a) is the conditional probability, s is the current state, a is the action taken in the current state s, s′ is the next state, r is the obtained reward, S t is the state at time step t, R t is the reward obtained at time step t, S t-1 is the state at time step t - 1, A t-1 is the action taken at time step t - 1;

[0104] At the same time, the Proximal Policy Optimization (PPO) algorithm is adopted. A policy function Π(θ) (where θ is the parameter of the network) can be represented by a neural network. Input the current state into the policy function, and the action to be taken at the next moment will be output. In a round of interaction, all the combinations of the states output by our environment and the actions of the actor are a trajectory, that is:

[0105] τ = {s1,a1,s2,a2...s t ,a t}

[0106] where τ represents a complete trajectory, s1,s2,...s t represents the state at time step t, a1, a2,...a t represents the action taken at the corresponding time step t;

[0107] The probability of this trajectory occurring can be calculated using the following:

[0108]

[0109] where p θ (τ) represents the probability of generating trajectory τ under policy parameter θ, and p(s1) represents the probability of the initial state s1. denotes the product over all time steps from t = 1 to t = T, where T is the total length of the trajectory. p θ (a t |s t ) represents the conditional probability of choosing action a t under state s t according to policy θ, and p(s t+1 |s t , a t ) ↓ represents the conditional probability of transitioning to state s t after taking action a t under state s t+1 ;

[0110] Once the probability corresponding to each trajectory and the reward R(τ) corresponding to each trajectory are obtained, the expected reward of this policy Π(θ) can be calculated. The calculation formula is:

[0111]

[0112] where is the expected value of the random variable R under the distribution p θ (τ), R(τ) is the value of the random variable R at a specific value τ, and p θ (τ) is the probability distribution function. is the expected value of the random variable R under the distribution p θ (τ);

[0113] The Markov decision process needs to maximize the expected reward of this policy. This can be achieved by taking the gradient of this expected reward and using gradient ascent to maximize the expected reward, that is, by finding where only is related to θ. The gradient of the expected reward can ultimately be written as:

[0114]

[0115] where is the gradient of the expected reward with respect to the policy parameter θ, N is the number of samples, R(τ n ) is the total reward of the nth sample trajectory τ n , is the probability of choosing action under state according to policy θ, is the gradient of the log probability with respect to the policy parameter θ, i.e., the derivative with respect to θ,

[0116] where the specific calculation and derivation process is as follows:

[0117]

[0118] where is the gradient of the log probability of the trajectory τ with respect to the policy parameter θ;

[0119] In the above formula and are determined by the environment and are independent of θ. Therefore, the derivatives of these two terms with respect to θ are both 0. After obtaining the gradient of the policy function, the parameters θ of the policy network can be updated by gradient ascent;

[0120] By adding a baseline and correcting the rewards, the policy network is trained more reasonably. The gradient update formula of the expected reward after adjustment is as shown in the specific calculation and derivation process. The gradient formula of the expected reward after improvement is as follows:

[0121]

[0122] where is the gradient of the expected reward with respect to the policy parameter θ, R(τ n ) is the total reward of the nth sample trajectory τ n , b is the baseline, that is, the average value of the rewards of the sample trajectories, is the gradient of the log probability with respect to the policy parameter θ, i.e., the derivative with respect to θ,

[0123] b is the baseline. During training, usually the rewards obtained by a policy are saved, and then their average reward is calculated, and this average reward is used as b. Therefore, the gradient formula of the expected reward after improvement can also be written as:

[0124]

[0125] where γ t′-t is the discount factor, is the reward obtained at the t'-th time step of the nth sample;

[0126] is called the advantage function. In the subsequent Actor-Critic algorithm, this term can be estimated by the Actor network;

[0127] S402. Design a proximal policy optimization algorithm and perform proximal policy optimization with Clip clipping to ensure the similarity of importance sampling results. When using the PPO (Proximal Policy Optimization) algorithm for training with the off-policy idea, the similarity between the two consecutive policies can be guaranteed. After adding importance sampling, the gradient formula for the expected reward is as follows:

[0128]

[0129] Among them, is the gradient of the expected reward with respect to the policy parameter θ, represents the expected value, which is calculated for all possible trajectories τ sampled according to the policy θ, p θ (a t ∣s t ) is the probability of selecting the action a t in the state s t according to the policy θ, p θ′ (a t ∣s t ) is the probability of selecting the action a t in the state s t according to the old policy θ′, A θ′ (s t ,a t ) is the advantage function, which represents the additional value of taking the action a t in the state s t compared to the average case, is the gradient of the log probability of the trajectory τ with respect to the policy parameter θ,

[0130] Among them, the advantage function is:

[0131]

[0132] Among them, A θ′ (s t ,a t ) is the advantage function, which represents the additional value of taking the action a t in the state s t compared to the average case, γ t′-t is the discount factor, is the reward obtained at the t′-th time step of the n-th sample, and b is the baseline, that is, the average value of the rewards of the sample trajectories;

[0133] According to the above two formulas, the objective function (loss function) can be deduced as follows:

[0134]

[0135] Among them, represents the gradient of the function f(θ) with respect to the parameter θ; f(θ) represents the function with respect to the parameter θ; represents the gradient of logf(θ) with respect to the parameter θ,

[0136]

[0137] in, According to the strategy π θ All possible state-action pairs (s t ,a t )Calculate the expected value, p θ (a t ∣s t ) is in state s t Next, select action a according to strategy θ t The probability, p θ′ (a t ∣s t ): In state s t Next, select action a according to the old strategy θ′ t The probability of A θ′ (s t ,a t ):Advantage function, indicating that in state s t Next take action a t The additional value relative to the average situation;

[0138] The objective function of the proximal policy optimization pruning algorithm is:

[0139]

[0140] Among them, p θ (a t ∣s t ) is in state s t Next, select action a according to strategy θ t The probability, p θ′ (a t ∣s t ) is in state s t Next, select action a according to the old strategy θ′ t The probability of A θ′ (s t ,a t ) is the advantage function, that is, in state s t Next take action a t The additional value relative to the average case, It is a truncation function, which limits the value of x to between 1-ε and 1+ε, where ε is a hyperparameter;

[0141] In the objective function formula of the proximal policy optimization clip algorithm above, the operator min takes the smaller of the two terms; there is a clipping function in front of the second term. The clipping function means that for the three terms in the parentheses, if the first term is within the range (1 - ε, 1 + ε) formed by the second term and the third term, the first term is output; if it exceeds this range, the term closest to the first term is output; ε is a hyperparameter, usually 0.1 or 0.2.

[0142] Figure 5 Figure 4 is the iteration flowchart of the proximal policy optimization model with Clip clipping. Based on the ideological framework of the reinforcement learning Markov decision process, the algorithm structure of Actor-Critic is combined during training and update, and the advantage function and clipping operation are adopted to update the policy (Critic) network and the value (Actor) network.

[0143] The sub-steps of step S5 are as follows:

[0144] S501. Build the benefit function in reinforcement learning from the perspective of the benefits of maintenance measures, incorporate the carbon emission benefits, construction delay costs, maintenance measure economic costs and expected benefits in step S3 into the maintenance costs, and calculate the maintenance measure benefits according to the following expression:

[0145]

[0146] Among them, B i is the change in the performance of the pavement after the maintenance measure i, C i is the cost corresponding to the i-th maintenance measure, and S is the PCI value of the pavement output by the asphalt pavement service performance prediction model;

[0147] S502. Set the pavement life prediction period. By inputting the corresponding asphalt pavement service data and environmental parameters, predict the benefit-cost ratio of the section-level maintenance measures and the comprehensive costs considering carbon emissions, etc. within a certain period of the same pavement structure in a similar environmental scenario, and add two exponential coefficients to B and C in the maintenance measure benefit expression respectively. This operation is to make the benefit function pay more reasonable attention to the improvement of pavement performance or the reduction of costs. When the pavement performance index PCI is poor, the model should pay more attention to the improvement of the PCI index, and this exponential coefficient will amplify the corresponding benefits; when the pavement performance index PCI is better, the model should pay more attention to the cost C, and at this time, the cost should be reduced as much as possible, and this exponential coefficient will reduce the corresponding benefits. In addition, to ensure the rapid convergence of the agent and save the time for training the agent, in the action selection function of the PPO algorithm, set rules such as:

[0148] (i) When the PCI index is greater than 85 points, no maintenance is required;

[0149] (ii) When the PCI index is less than 60 points, maintenance measures must be taken, and the specified maintenance measures are milling and overlaying;

[0150] Finally, a proximal policy optimization model with a clipping operation was established, which improved the convergence speed of the proximal policy optimization model.

[0151] Finally, based on the actual situation, the simulation of step S6 was carried out. Specifically, what maintenance measures were adopted is as Figure 7 shown. According to the input parameters of the asphalt pavement of this section, the PCI of this section for the next 20 years was predicted, and based on the predicted PCI index, the maintenance measure with the best comprehensive benefit-cost ratio of the maintenance measures for this section in 20 years was selected. Figure 6 Among them, the PCI index of the road surface after applying the maintenance measures decided by this method is significantly better than that without applying the maintenance measures. It can be seen that the present invention is effective and feasible.

[0152] As mentioned above, the above are only the specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for predicting pavement performance and optimizing maintenance decisions based on machine learning, characterized in that, The method includes: Quantifying the carbon emission cost, traffic delay cost and economic cost of asphalt pavement maintenance, obtaining the predicted pavement life years, asphalt pavement service data and environmental parameters of the section to be measured, and obtaining the optimal maintenance measures for the section and their benefit-cost ratios considering carbon emissions and delay costs through the asphalt pavement service performance prediction model and the asphalt pavement maintenance decision model. The above process is cycled to update the maintenance measures, where The asphalt pavement service performance prediction model is an LSTM neural network model introducing the Attention mechanism. The asphalt pavement service performance prediction model is trained through the asphalt pavement dataset, and the asphalt pavement maintenance decision model is optimized through the proximal policy optimization algorithm clipped by Clip.

2. The pavement performance prediction and maintenance decision optimization method based on machine learning according to claim 1, characterized in that The construction process of the asphalt pavement dataset includes: Collecting historical asphalt pavement heterogeneous data, including section number, inspection year, DR, PCI value, service life, annual rainfall, high temperature index, freezing index, existing maintenance number and maintenance thickness, AADTT, kESAL, surface layer thickness, base layer thickness, base and sub-base types; Performing outlier screening and data splicing processing on the asphalt pavement heterogeneous data to construct an asphalt pavement dataset.

3. A pavement performance prediction and maintenance decision optimization method based on machine learning according to claim 1, characterized in that, The training process of the asphalt pavement service performance prediction model includes: Preprocessing the asphalt pavement dataset and dividing it into a training set and a test set; Constructing the asphalt pavement service performance prediction model; Training the asphalt pavement service performance prediction model using the training set; Validating the trained asphalt pavement service performance prediction model using the test set; Use R 2 Evaluate the prediction effect of the asphalt pavement service performance prediction model using the evaluation index. If the evaluation result is lower than the threshold, repeat the above training and prediction; otherwise, output the asphalt pavement service performance prediction model.

4. A method for predicting pavement performance and optimizing maintenance decisions based on machine learning according to claim 3, characterized in that The preprocessing includes: Dividing the data in the asphalt pavement dataset into maintenance measures, base types, sub-base types and numerical data, and normalizing the asphalt pavement dataset; Performing one-hot encoding on the maintenance measures, base types and sub-base types data, and performing min-max scaling on the numerical data to complete the preprocessing.

5. The pavement performance prediction and maintenance decision optimization method based on machine learning according to claim 3, characterized in that The construction process of the asphalt pavement service performance prediction model includes: Building the basic structure of the asphalt pavement service performance prediction model, including an input layer, an LSTM layer, a fully connected layer and a Self-Attention layer; Determining the input and output of the asphalt pavement service performance prediction model according to the preprocessed asphalt pavement dataset to complete the construction of the asphalt pavement service performance prediction model, where the output is the PCI value of the asphalt pavement.

6. The pavement performance prediction and maintenance decision optimization method based on machine learning according to claim 3, characterized in that The loss function in the training of the asphalt pavement service performance prediction model is: Among them, LOSS is the model loss function, RMSE is the root mean square error, y i is the measured value of the i-th sample, is the predicted value of the i-th sample, and n is the total number of data points.

7. A method for predicting pavement performance and optimizing maintenance decisions based on machine learning according to claim 3, characterized in that The said R 2 The expression of the evaluation index is: Among them, R 2 is the coefficient of determination, y i is the measured value of the i-th sample, is the average value of the samples, is the predicted value of the i-th sample.

8. A pavement performance prediction and maintenance decision optimization method based on machine learning according to claim 1, characterized in that The quantification process of the carbon emission cost, traffic delay cost and economic cost of asphalt pavement maintenance includes: Obtaining asphalt pavement maintenance measures and their typical asphalt pavement maintenance material parameters and transportation parameters, and determining the boundary range of the assumed parameters; Obtaining the data of the preset carbon emission factor library, and calculating the carbon emissions and their costs in the raw material acquisition, transportation and construction stages according to the boundary range of the assumed parameters; Calculating the traffic delay time and cost caused by maintenance construction based on the highway machinery work shift cost quota according to the boundary range of the assumed parameters; Integrating the carbon emission cost and the traffic delay cost to form a comprehensive cost index for each maintenance measure.

9. A pavement performance prediction and maintenance decision optimization method based on machine learning according to claim 8, characterized in that, The asphalt pavement maintenance decision-making model is constructed based on the Proximal Policy Optimization (PPO) algorithm. Its framework is a Markov decision process framework, and the benefit-cost ratio of carbon emissions and delay costs is calculated based on the comprehensive cost indicators of each maintenance measure. The calculation formula for the benefit-cost ratio is as follows: Among them, B i is the change in performance after the maintenance measure i for this section of the road, and C i is the cost corresponding to the i-th maintenance measure. S is the PCI value of the road surface output by the asphalt pavement service performance prediction model.

10. A method for predicting pavement performance and optimizing maintenance decisions based on machine learning according to claim 9, characterized in that The asphalt pavement maintenance decision-making model uses the Clip clipping technique to limit the update amplitude of the Proximal Policy Optimization (PPO) algorithm. The optimization objective function of the Proximal Policy Optimization (PPO) algorithm is: where p θ (a t ∣s t ) is the probability of selecting action a t under state s t according to policy θ, and p θ′ (a t ∣s t ) is the probability of selecting action a t under state s t according to the old policy θ′. A θ′ (s t ,a t ) is the advantage function, that is, the additional value of taking action a t under state s t relative to the average case. is a truncation function that limits the value of x between 1 - ε and 1 + ε, where ε is a hyperparameter.

Images