Airport pavement maintenance decision method and device, electronic equipment and storage medium

Abstract

The application provides an airport pavement maintenance decision method and device, electronic equipment and a storage medium, wherein the method comprises: acquiring the current state of a target airport pavement unit; inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model comprises a deep Q network (DQN) layer and a state prediction layer; the parameters of the state prediction layer are adaptively updated based on an extended Kalman filter; wherein the decision model is trained based on the sample state of a sample airport pavement unit and the target Q value corresponding to the sample state by using a WoLF-PHC algorithm. The application reduces the cost of model training, improves the robustness of the decision model, and is suitable for application scenarios with measurement errors.

Description

Technical Field

[0001] This invention relates to the field of airport pavement maintenance technology, and in particular to an airport pavement maintenance decision-making method, apparatus, electronic device and storage medium. Background Technology

[0002] Airport pavement maintenance primarily focuses on two issues: when to perform maintenance operations and what type of maintenance measures to adopt. Currently, pavement decision-making maintenance methods based on operations research suffer from instability and are very slow in solving large-scale planning problems. Artificial intelligence-based methods can solve large-scale optimization decision problems, but decision-making maintenance methods based on artificial intelligence and traditional machine learning algorithms rely on high-quality training sets, resulting in high model training costs and poor robustness when facing application scenarios with measurement errors. Summary of the Invention

[0003] This invention provides an airport pavement maintenance decision-making method, apparatus, electronic device, and storage medium to address the shortcomings of existing airport pavement maintenance decision-making methods based on artificial intelligence and traditional machine learning algorithms, which rely on high-quality training sets, have high model training costs, and exhibit poor robustness when facing application scenarios with measurement errors.

[0004] In a first aspect, the present invention provides an airport pavement maintenance decision-making method, comprising:

[0005] Obtain the current state of the target airport pavement unit;

[0006] The current state of the target airport pavement unit is input into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a deep Q-network (DQN) layer and a state prediction layer.

[0007] The parameters of the state prediction layer are adaptively updated based on the extended Kalman filter.

[0008] The decision model is trained using the WoLF-PHC algorithm, based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state.

[0009] In some embodiments, the training process of the decision model includes:

[0010] An initial decision model and an experience replay buffer are constructed. The initial decision model includes an initial DQN layer and a pre-trained state prediction layer. The initial DQN layer includes an initial online network and an initial target network.

[0011] Obtain the sample state of the sample airport pavement unit at time t. The sample state includes: sample pavement condition index, sample airport pavement strength index, sample slab bottom void rate, and sample flatness.

[0012] Based on the sample status at time t, search for the corresponding sample action, which includes the sample maintenance measures for the sample airport pavement unit;

[0013] The sample state at time t is input into the initial state prediction layer to obtain the sample state at time t+1 output by the initial state prediction layer.

[0014] The reward for the sample action is determined based on the result of the interaction between the sample action and the environment.

[0015] The sample state, sample action, reward of the sample action, and sample state at time t+1 are stored in the experience replay buffer.

[0016] Randomly sample sample states from the experience replay buffer, predict the Q value corresponding to the sample state based on the initial online network, and predict the target Q value corresponding to the sample state based on the initial target network.

[0017] Based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated. Starting from searching for the corresponding sample action according to the sample state at time t, the process is iteratively optimized to optimize the parameters of the initial online network and the initial target network until the iteration termination condition is met, thus obtaining the DQN layer.

[0018] In some embodiments, updating the parameters of the initial online network and the initial target network based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state includes:

[0019] Based on the sample state, multiple corresponding sample actions are obtained, and the average estimation strategy of the multiple sample actions is updated.

[0020] The strategy for each sample action is updated based on the Q-value corresponding to each sample action and the target Q-value.

[0021] In some embodiments, the update formula for the average estimation strategy is as follows:

[0022] ;

[0023] ;

[0024] in, This represents the updated average estimation strategy. This indicates the average estimation strategy. Indicates sample state The corresponding number Individual sample actions strategy, Indicates sample state Number of times it appears Indicates sample state Updated value for the number of occurrences;

[0025] The update formula for the strategy of each sample action is as follows:

[0026] ;

[0027] in, Indicates sample action The update volume of the strategy, Based on sample state The corresponding target Q value is determined.

[0028] In some embodiments, updating the parameters of the initial online network and the initial target network based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state includes:

[0029] Based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, calculate the loss function value;

[0030] Based on the loss function value, the parameters of the initial online network and the initial target network are updated.

[0031] In some embodiments, the loss function value is calculated using the following formula:

[0032] ;

[0033] in, The value of the loss function. express Sample state at time 1 The corresponding target Q value, express Sample state at time 1 The corresponding number Individual sample actions The reward refers to taking sample actions on the sample airport pavement unit. The overall cost-benefit ratio after corresponding maintenance measures express Sample state at time 1 Corresponding sample actions The maximum Q value, express Sample state at time 1 Corresponding sample actions Q value, This represents the discount rate.

[0034] In some embodiments, the adaptive update of the parameters of the state prediction layer based on the extended Kalman filter includes:

[0035] The observed value of the current state of the target airport pavement unit and the predicted value of the current state of the target airport pavement unit are determined. The predicted value of the current state is obtained by inputting the historical state of the target airport pavement unit into the state prediction layer for prediction.

[0036] Based on the prediction noise of the state prediction layer, the Kalman gain of the extended Kalman filter is determined;

[0037] The parameters of the state prediction layer are adaptively updated based on the observed value of the current state, the predicted value of the current state, and the Kalman gain.

[0038] Secondly, the present invention also provides an airport pavement maintenance decision-making device, comprising:

[0039] The acquisition unit is used to acquire the current state of the target airport pavement unit;

[0040] A decision unit is used to input the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a deep Q-network (DQN) layer and a state prediction layer;

[0041] The update unit is used to adaptively update the parameters of the state prediction layer based on the extended Kalman filter;

[0042] The decision model is trained using the WoLF-PHC algorithm, based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state.

[0043] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the airport pavement maintenance decision-making method as described above.

[0044] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the airport pavement maintenance decision-making method as described above.

[0045] The airport pavement maintenance decision-making method, device, electronic device, and storage medium provided by this invention obtain the current state of the target airport pavement unit, input the current state of the target airport pavement unit into a pre-trained decision model, and obtain the optimal maintenance measures for the target airport pavement unit output by the decision model. The decision model includes a deep Q-network (DQN) layer and a state prediction layer. Based on an extended Kalman filter, the parameters of the state prediction layer are adaptively updated, which reduces the cost of model training, improves the robustness of the decision model, and is suitable for application scenarios with measurement errors. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the airport pavement maintenance decision-making method provided in an embodiment of the present invention;

[0048] Figure 2 This is a flowchart illustrating the training process of the decision model provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the airport pavement maintenance decision-making device provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] Currently, airport pavement maintenance decision-making methods based on reinforcement learning mainly rely on neural network models for state prediction in terms of state updates. However, the model parameters are not dynamically updated. Especially when there are measurement errors or inaccurate predictions, the model can only be retrained, increasing training costs. In addition, existing improvement methods for Deep Q-Network (DQN) do not consider the update efficiency of the learning policy, which also increases the training cost of reinforcement learning methods.

[0053] To address this, embodiments of the present invention provide an airport pavement maintenance decision-making method, apparatus, electronic device, and storage medium. By acquiring the current state of a target airport pavement unit, the current state of the target airport pavement unit is input into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model. The decision model includes a Deep Q-Network (DQN) layer and a state prediction layer. Based on an Extended Kalman Filter (EKF), the parameters of the state prediction layer are adaptively updated. Embodiments of the present invention can reduce the cost of model training, improve the robustness of the decision model, and are suitable for application scenarios where measurement errors exist.

[0054] Figure 1 This is a flowchart illustrating the airport pavement maintenance decision-making method provided in an embodiment of the present invention. Figure 1 As shown, an airport pavement maintenance decision-making method is provided, including the following steps: step 110, step 120, and step 130. This method's steps are merely one possible implementation of the invention.

[0055] Step 110: Obtain the current state of the target airport pavement unit.

[0056] The current state describes the current structural state, performance state, and material information of the target airport pavement unit.

[0057] Optionally, the current status includes at least: the current pavement condition index, the current airport pavement strength index, the current under-slab clearance rate, and the current smoothness.

[0058] Optionally, the pavement of the target airport can be divided into multiple target airport pavement units.

[0059] The target airport pavement refers to the cement concrete pavement, and the target airport pavement unit includes multiple cement concrete slabs.

[0060] It should be noted that the zoning rules for the target airport pavement follow three levels: "location-area-unit".

[0061] Optionally, the target airport pavement can be divided into different parts based on the differences in the physical structure of the runway and the impact of aircraft loads, such as: runway, parallel taxiway, connecting taxiway, apron and asphalt transition section; the same part can be divided into different areas based on pavement structure and load level, and each area can be divided into multiple units.

[0062] Step 120: Input the current state of the target airport pavement unit into the pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a deep Q-network (DQN) layer and a state prediction layer.

[0063] The decision model is trained using the WoLF-PHC algorithm, based on the sample state of the airport pavement unit and the target Q value corresponding to the sample state.

[0064] DQN combines deep learning and reinforcement learning algorithms to solve complex problems with high-dimensional state spaces. The DQN layer consists of an online network and a target network. The online network generates a Q-value estimate of the current policy, while the target network calculates the target Q-value to stabilize the training process. The target network periodically copies weights from the online network to help reduce instability during training.

[0065] It should be noted that reinforcement learning is an important branch of machine learning. In reinforcement learning, an agent (such as a target airport pavement unit) takes different actions based on its different states in order to obtain the maximum reward from the environment.

[0066] Maintenance measures include, but are not limited to:

[0067] 1) Routine maintenance: The pavement is in good overall condition. Any damage can be repaired promptly without any additional measures.

[0068] 2) Routine maintenance and attention to void observation: The pavement surface is in good condition and there is no serious damage, but there is some void in the base layer or foundation. It is necessary to strengthen the observation of void under the pavement slab.

[0069] 3) Targeted repair: The pavement is damaged in many ways. In order to prevent the damage from developing, targeted repair work is needed for some of the damage, such as crack filling, shallow repair, partial thickness repair, full thickness repair, slab grinding, and precast hoisting and replacement of slabs.

[0070] 4) Replace the joint sealant and pay attention to void observation: The overall condition of the pavement is good, but the joint sealant is damaged in many places. The joint sealant needs to be replaced as a whole, and the void observation of the pavement slab bottom needs to be strengthened.

[0071] 5) Grouting repair: The pavement surface is in good condition, but the base layer or foundation has serious voids, so grouting repair is required for the pavement in the area.

[0072] 6) Targeted repair and grouting repair: The pavement is severely damaged and the voids are quite serious, so grouting repair is required, and targeted repairs should be carried out for some defects;

[0073] 7) Operation Restriction: The pavement structure has poor performance and its load-bearing capacity does not meet the usage requirements, so it is necessary to restrict the types of vehicles and the total load capacity in this area;

[0074] 8) Overlay: If the bearing capacity or flatness of the pavement does not meet the requirements, structural or functional overlay is required in the area to ensure normal operation.

[0075] It should be noted that the environment can be considered as the external environment in which the target airport pavement unit is located. The problem that reinforcement learning considers is the interaction between the agent and the environment. The role of the environment is to provide the state and immediate reward at the next moment. Therefore, in the maintenance decision problem, the environment can be abstracted into a state prediction layer.

[0076] Step 130: Adaptively update the parameters of the state prediction layer based on the extended Kalman filter.

[0077] The Extended Kalman Filter (EKF) is a state estimation method for nonlinear dynamic systems. EKF is an extension of the classic Kalman filter. EKF achieves state estimation by linearizing the nonlinear function.

[0078] In some embodiments, step 130 adaptively updates the parameters of the state prediction layer based on the extended Kalman filter, including:

[0079] Step 131: Determine the observed value of the current state of the target airport pavement unit and the predicted value of the current state of the target airport pavement unit. The predicted value of the current state is obtained by inputting the historical state of the target airport pavement unit into the state prediction layer for prediction.

[0080] Step 132: Determine the Kalman gain of the extended Kalman filter based on the prediction noise of the state prediction layer;

[0081] Step 133: Based on the observed value of the current state, the predicted value of the current state, and the Kalman gain, adaptively update the parameters of the state prediction layer.

[0082] Optionally, historical conditions include: historical pavement condition index, historical airport pavement strength index, historical slab undercarriage clearance rate, historical smoothness, historical traffic volume, historical structural strength, historical surface layer thickness, historical base layer type, historical environmental conditions, and historical material type.

[0083] Optionally, the predicted values ​​for the current state include: the predicted value for the current pavement condition index, the predicted value for the current airport pavement strength index, the predicted value for the current under-slab clearance rate, and the predicted value for the current smoothness.

[0084] Optionally, the adaptive correction formula for the parameters of the state prediction layer is as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] in, This represents the parameters after correction for the state prediction layer. Indicates the current state prediction layer The parameters at time, Indicates Kalman gain, Indicates the current The observed value of the state at time t. This represents the current output of the state prediction layer. The predicted value of the state at time t. This represents the input to the state prediction layer, i.e., the historical states. Represents a diagonal matrix consisting entirely of 1s. express The transpose of the matrix, for Prior covariance at time, for The posterior covariance at time 1, express The posterior covariance at time 1, For observation error, It is an identity matrix.

[0090] Optionally, the parameters of the state prediction layer can be iteratively updated until the stopping condition is met, if this method is not required to make maintenance decisions for the target airport pavement unit.

[0091] In this embodiment of the invention, the current state of the target airport pavement unit is obtained and input into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model. The decision model includes a Deep Q-Network (DQN) layer and a state prediction layer. Based on an Extended Kalman Filter (EKF), the parameters of the state prediction layer are adaptively updated. This embodiment of the invention can reduce the cost of model training, improve the robustness of the decision model, and is suitable for application scenarios where measurement errors exist.

[0092] Figure 2 This is a flowchart illustrating the training process of the decision model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, in some embodiments, the training process of the decision model includes:

[0093] Step 210: Construct an initial decision model and an experience replay buffer. The initial decision model includes an initial DQN layer and a pre-trained state prediction layer. The initial DQN layer includes an initial online network and an initial target network.

[0094] Step 220: Obtain the sample state of the sample airport pavement unit at time t. The sample state includes: sample pavement condition index, sample airport pavement strength index, sample slab bottom void rate, and sample flatness.

[0095] Step 230: Based on the sample status at time t, search for the corresponding sample action, which includes sample maintenance measures for the sample airport pavement unit.

[0096] Step 240: Input the sample state at time t into the initial state prediction layer to obtain the sample state at time t+1 output by the initial state prediction layer.

[0097] Step 250: Determine the reward for the sample action based on the result of the interaction between the sample action and the environment;

[0098] Step 260: Store the sample state, sample action, sample action reward, and sample state at time t+1 into the experience replay buffer.

[0099] Step 270: Randomly sample sample states from the experience replay buffer, predict the Q value corresponding to the sample state based on the initial online network, and predict the target Q value corresponding to the sample state based on the initial target network.

[0100] Step 280: Based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, update the parameters of the initial online network and the initial target network. Start by searching for the corresponding sample action based on the sample state at time t, and iterate to optimize the parameters of the initial online network and the initial target network until the iteration termination condition is met, and obtain the DQN layer.

[0101] Optionally, sample maintenance measures include, but are not limited to: routine maintenance, void observation, targeted repair, replacement of caulking material, grouting repair, operation restriction, and overlay.

[0102] The state prediction layer is trained based on historical states.

[0103] Optionally, the Q value of the sample airport pavement unit can be initialized to 0.

[0104] Optionally, an ε-greedy strategy can be used to search for the sample action corresponding to the sample state at time t.

[0105] Among them, the ε-greedy strategy is a commonly used reinforcement learning strategy used to balance exploration and exploitation. Its purpose is to maximize the benefit by utilizing the currently known information during the action selection process, while also exploring new possibilities to discover better strategies.

[0106] Optionally, the online network, the target network, and the state prediction layer all include a multilayer perceptron network.

[0107] In some embodiments, the parameters of the initial online network and the initial target network are updated based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, including:

[0108] Based on the sample state, obtain the corresponding multiple sample actions, and update the average estimation strategy of multiple sample actions;

[0109] The strategy for each sample action is updated based on the Q-value and target Q-value corresponding to each sample action.

[0110] Understandably, updating the average estimated policy using multiple sample actions helps improve the stability and accuracy of policy estimation and reduces bias caused by a single sample. Updating each sample action independently can better capture the complexity of the environment and improve the policy's performance under different states. The average estimate of multiple samples can make the learning process smoother, thereby accelerating policy convergence. Making full use of sample action data helps improve the efficiency and effectiveness of policy optimization.

[0111] In some embodiments, the update formula for the average estimation strategy is as follows:

[0112] ;

[0113] ;

[0114] in, This represents the updated average estimation strategy. This indicates the average estimation strategy. Indicates sample state The corresponding number Individual sample actions strategy, Indicates sample state Number of times it appears Indicates sample state Updated value for the number of occurrences;

[0115] The policy update formula for each sample action is as follows:

[0116] ;

[0117] in, Indicates sample action The update volume of the strategy, Based on sample state The corresponding target Q value is determined.

[0118] Optionally, The calculation formula is as follows:

[0119]

[0120]

[0121]

[0122] in, Indicates sample state The sample action corresponding to the maximum Q value. Indicates sample state Corresponding sample actions Q value, Indicates sample state The corresponding maximum Q value; Indicates sample action The corresponding adjustment amount of the strategy, Indicates sample action The corresponding adjustment amount of the strategy, Indicates sample action The corresponding strategy, ) indicates sample action The corresponding strategy, This indicates the adjustment amount of the strategy parameters. This indicates the number of sample actions.

[0123] Optionally, The calculation formula is as follows:

[0124]

[0125] in, and These are all adjustments to the strategy parameters. and , Indicates sample state The corresponding set of sample actions.

[0126] In some embodiments, the parameters of the initial online network and the initial target network are updated based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, including:

[0127] The loss function value is calculated based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state.

[0128] Based on the loss function value, the parameters of the initial online network and the initial target network are updated.

[0129] In some embodiments, the loss function value is calculated using the following formula:

[0130] ;

[0131] in, The value of the loss function. express Sample state at time 1 The corresponding target Q value, express Sample state at time 1 The corresponding number Individual sample actions The reward refers to taking sample actions on the sample airport pavement units. The overall cost-benefit ratio after corresponding maintenance measures express Sample state at time 1 Corresponding sample actions The maximum Q value, express Sample state at time 1 Corresponding sample actions Q value, This represents the discount rate.

[0132] Optionally, The calculation formula is as follows:

[0133]

[0134]

[0135]

[0136] in, Indicates sample state The total number of indicators, Indicates the first One indicator, Indicates the first Rewards corresponding to each indicator Represents sample airport pavement units According to the The maintenance benefits of each indicator at time t. Represents sample airport pavement units According to the The maintenance cost of each indicator at time t. Represents sample airport pavement units According to the Each indicator requires maintenance measures at time t. The resulting maintenance benefits Represents sample airport pavement units According to the The maintenance benefits of each indicator when no maintenance measures are taken at time t. This indicates the time since the last maintenance.

[0137] The airport pavement maintenance decision-making device provided in the embodiments of the present invention is described below. The airport pavement maintenance decision-making device described below and the airport pavement maintenance decision-making method described above can be referred to in correspondence.

[0138] Figure 3 This is a schematic diagram of the airport pavement maintenance decision-making device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the airport pavement maintenance decision-making device 300 includes:

[0139] Acquisition unit 310 is used to acquire the current state of the target airport pavement unit;

[0140] The decision unit 320 is used to input the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a deep Q-network (DQN) layer and a state prediction layer.

[0141] The update unit 330 is used to adaptively update the parameters of the state prediction layer based on the extended Kalman filter;

[0142] The decision model is trained using the WoLF-PHC algorithm, based on the sample state of the airport pavement unit and the target Q value corresponding to the sample state.

[0143] Optionally, the training process of the decision model includes:

[0144] Construct an initial decision model and an experience replay buffer. The initial decision model includes an initial DQN layer and a pre-trained state prediction layer. The initial DQN layer includes an initial online network and an initial target network.

[0145] The sample state of the airport pavement unit at time t is obtained. The sample state includes: sample pavement condition index, sample airport pavement strength index, sample slab bottom void rate, and sample smoothness.

[0146] Based on the sample status at time t, search for the corresponding sample actions, which include sample maintenance measures for the sample airport pavement unit.

[0147] The sample state at time t is input into the initial state prediction layer to obtain the sample state at time t+1 output by the initial state prediction layer.

[0148] The reward for a sample action is determined based on the result of its interaction with the environment.

[0149] Store the sample state, sample action, sample action reward, and sample state at time t+1 in the experience replay buffer.

[0150] Randomly sample state from the experience replay buffer, predict the Q value corresponding to the sample state based on the initial online network, and predict the target Q value corresponding to the sample state based on the initial target network.

[0151] Based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated. Starting from searching for the corresponding sample action based on the sample state at time t, the loop iterates and optimizes the parameters of the initial online network and the initial target network until the iteration termination condition is met, thus obtaining the DQN layer.

[0152] Optionally, based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated, including:

[0153] Based on the sample state, obtain the corresponding multiple sample actions, and update the average estimation strategy of multiple sample actions;

[0154] The strategy for each sample action is updated based on the Q-value and target Q-value corresponding to each sample action.

[0155] Optionally, the update formula for the average estimation strategy is as follows:

[0156] ;

[0157] ;

[0158] in, This represents the updated average estimation strategy. This indicates the average estimation strategy. Indicates sample state The corresponding number Individual sample actions strategy, Indicates sample state Number of times it appears Indicates sample state Updated value for the number of occurrences;

[0159] The policy update formula for each sample action is as follows:

[0160] ;

[0161] in, Indicates sample action The update volume of the strategy, Based on sample state The corresponding target Q value is determined.

[0162] Optionally, based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated, including:

[0163] The loss function value is calculated based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state.

[0164] Based on the loss function value, the parameters of the initial online network and the initial target network are updated.

[0165] Alternatively, the formula for calculating the loss function value is as follows:

[0166] ;

[0167] in, The value of the loss function. express Sample state at time 1 The corresponding target Q value, express Sample state at time 1 The corresponding number Individual sample actions The reward refers to taking sample actions on the sample airport pavement units. The overall cost-benefit ratio after corresponding maintenance measures express Sample state at time 1 Corresponding sample actions The maximum Q value, express Sample state at time 1 Corresponding sample actions Q value, This represents the discount rate.

[0168] Optionally, based on the extended Kalman filter, the parameters of the state prediction layer are adaptively updated, including:

[0169] The observed value of the current state of the target airport pavement unit and the predicted value of the current state of the target airport pavement unit are determined. The predicted value of the current state is obtained by inputting the historical state of the target airport pavement unit into the state prediction layer for prediction.

[0170] The Kalman gain of the extended Kalman filter is determined based on the prediction noise of the state prediction layer.

[0171] The parameters of the state prediction layer are adaptively updated based on the observed values ​​of the current state, the predicted values ​​of the current state, and the Kalman gain.

[0172] It should be noted that the airport pavement maintenance decision-making device provided in this embodiment of the invention can realize all the method steps implemented in the above-mentioned airport pavement maintenance decision-making method embodiment, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0173] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, communication interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an airport pavement maintenance decision-making method. This method includes: acquiring the current state of the target airport pavement unit; inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a Deep Q-Network (DQN) layer and a state prediction layer; and adaptively updating the parameters of the state prediction layer based on an Extended Kalman Filter (EKF). The decision model is trained using the WoLF-PHC algorithm based on the sample states of sample airport pavement units and the target Q-values ​​corresponding to the sample states.

[0174] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the airport pavement maintenance decision-making method provided by the above methods. The method includes: obtaining the current state of a target airport pavement unit; inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a deep Q-network (DQN) layer and a state prediction layer; and adaptively updating the parameters of the state prediction layer based on an extended Kalman filter; wherein the decision model is trained using the WoLF-PHC algorithm based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A decision-making method for airport pavement maintenance, characterized in that, include: Obtain the current state of the target airport pavement unit; The current state of the target airport pavement unit is input into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a deep Q-network (DQN) layer and a state prediction layer. The parameters of the state prediction layer are adaptively updated based on the extended Kalman filter. The decision model is trained using the WoLF-PHC algorithm, based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state. The training process of the decision model includes: An initial decision model and an experience replay buffer are constructed. The initial decision model includes an initial DQN layer and a pre-trained state prediction layer. The initial DQN layer includes an initial online network and an initial target network. Obtain the sample state of the sample airport pavement unit at time t. The sample state includes: sample pavement condition index, sample airport pavement strength index, sample slab bottom void rate, and sample flatness. Based on the sample status at time t, search for the corresponding sample action, which includes the sample maintenance measures for the sample airport pavement unit; The sample state at time t is input into the state prediction layer to obtain the sample state at time t+1 output by the state prediction layer. The reward for the sample action is determined based on the result of the interaction between the sample action and the environment. The sample state, sample action, reward of the sample action, and sample state at time t+1 are stored in the experience replay buffer. Randomly sample sample states from the experience replay buffer, predict the Q value corresponding to the sample state based on the initial online network, and predict the target Q value corresponding to the sample state based on the initial target network. Based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated. Starting from searching for the corresponding sample action according to the sample state at time t, the loop is performed to iteratively optimize the parameters of the initial online network and the initial target network until the iteration termination condition is met, and the DQN layer is obtained. The step of updating the parameters of the initial online network and the initial target network based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state includes: Based on the sample state, multiple corresponding sample actions are obtained, and the average estimation strategy of the multiple sample actions is updated. The strategy for each sample action is updated based on the Q-value corresponding to each sample action and the target Q-value.

2. The airport pavement maintenance decision-making method according to claim 1, characterized in that, The update formula for the average estimation strategy is as follows: ； ； in, This represents the updated average estimation strategy. This represents the average estimation strategy. Indicates sample state The corresponding number Individual sample actions strategy, Indicates sample state Number of times it appears Indicates sample state Updated value for the number of occurrences; The update formula for the strategy of each sample action is as follows: ； in, Indicates sample action The update volume of the strategy, Based on sample state The corresponding target Q value is determined.

3. The airport pavement maintenance decision-making method according to claim 1, characterized in that, The step of updating the parameters of the initial online network and the initial target network based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state includes: Based on the Q-value corresponding to the sample state and the target Q-value corresponding to the sample state, calculate the loss function value; Based on the loss function value, the parameters of the initial online network and the initial target network are updated.

4. The airport pavement maintenance decision-making method according to claim 3, characterized in that, The formula for calculating the value of the loss function is as follows: ； in, The value of the loss function. express Sample state at time 1 The corresponding target Q value, express Sample state at time 1 The corresponding number Individual sample actions The reward refers to taking sample actions on the sample airport pavement unit. The overall cost-benefit ratio after corresponding maintenance measures express Sample state at time 1 Corresponding sample actions The maximum Q value, express Sample state at time 1 Corresponding sample actions Q value, This represents the discount rate.

5. The airport pavement maintenance decision-making method according to any one of claims 1-4, characterized in that, The adaptive update of the parameters of the state prediction layer based on the extended Kalman filter includes: The observed value of the current state of the target airport pavement unit and the predicted value of the current state of the target airport pavement unit are determined. The predicted value of the current state is obtained by inputting the historical state of the target airport pavement unit into the state prediction layer for prediction. Based on the prediction noise of the state prediction layer, the Kalman gain of the extended Kalman filter is determined; The parameters of the state prediction layer are adaptively updated based on the observed value of the current state, the predicted value of the current state, and the Kalman gain.

6. An airport pavement maintenance decision-making device, the device being used to execute the airport pavement maintenance decision-making method as described in any one of claims 1-5, characterized in that, include: The acquisition unit is used to acquire the current state of the target airport pavement unit; A decision unit is used to input the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures for the target airport pavement unit output by the decision model; the decision model includes a deep Q-network (DQN) layer and a state prediction layer; The update unit is used to adaptively update the parameters of the state prediction layer based on the extended Kalman filter; The decision model is trained using the WoLF-PHC algorithm, based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the airport pavement maintenance decision-making method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the airport pavement maintenance decision-making method as described in any one of claims 1 to 5.

Images