Airfield pavement maintenance decision-making method and device, electronic equipment and storage medium

Abstract

The invention provides an airport pavement maintenance decision-making method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a current state of a target airport pavement unit; inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain an optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model comprises a deep Q network DQN layer and a state prediction layer; based on an extended Kalman filter, adaptively updating the parameters of the state prediction layer; wherein the decision model is obtained by training based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state by adopting a WoLF-PHC algorithm. According to the method, the model training cost is reduced, the robustness of the decision model is improved, and the method is suitable for application scenes with measurement errors.

Description

Technical Field

[0001] The invention relates to the technical field of airport pavement maintenance, and in particular to an airport pavement maintenance decision-making method, device, electronic equipment and storage medium. Background Art

[0002] Airport pavement maintenance mainly focuses on two issues: when to perform maintenance operations; and what type of maintenance measures to adopt. At present, the pavement decision-making maintenance method based on operations research is unstable and is very slow in solving large-scale planning problems. The method based on artificial intelligence can solve large-scale optimization decision-making problems, but the decision-making maintenance method based on artificial intelligence and traditional machine learning algorithms relies on high-quality training sets, and the model training cost is high. When faced with application scenarios with measurement errors, the model robustness is poor. Summary of the invention

[0003] The present invention provides an airport pavement maintenance decision-making method, device, electronic device and storage medium, which are used to solve the defects of the airport pavement maintenance decision-making method based on artificial intelligence and traditional machine learning algorithms in the prior art, which relies on high-quality training sets, has high model training costs, and has poor model robustness when facing application scenarios with measurement errors.

[0004] In a first aspect, the present invention provides an airport pavement maintenance decision-making method, comprising: Get the current status of the target airport pavement unit; Inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model includes a deep Q network DQN layer and a state prediction layer; Based on the extended Kalman filter, adaptively updating the parameters of the state prediction layer; The decision model is obtained by training using the WoLF-PHC algorithm based on the sample states of the sample airport pavement units and the target Q values ​​corresponding to the sample states.

[0005] In some embodiments, the training process of the decision model includes: Constructing an initial decision model and an experience replay buffer, wherein the initial decision model includes an initial DQN layer and a pre-trained state prediction layer, and the initial DQN layer includes an initial online network and an initial target network; Obtaining the sample state of the sample airport pavement unit at time t, wherein the sample state includes: a sample pavement condition index, a sample airport pavement strength index, a sample plate bottom void rate, and a sample flatness; According to the sample state at time t, searching for a corresponding sample action, wherein the sample action includes a sample maintenance measure of the sample airport pavement unit; Input the sample state at time t into the initial state prediction layer to obtain the sample state at time t+1 output by the initial state prediction layer; Determining a reward for the sample action according to a result of the interaction between the sample action and the environment; The sample state at time t, the sample action, the reward of the sample action and the sample state at time t+1 are stored in the experience replay buffer; Randomly extracting a sample state from the experience replay buffer, predicting a Q value corresponding to the sample state based on the initial online network, and predicting a target Q value corresponding to the sample state based on the initial target network; Based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated, and a loop is started by searching for the corresponding sample action according to the sample state at time t, and the parameters of the initial online network and the initial target network are iteratively optimized until the iteration termination condition is met to obtain the DQN layer.

[0006] In some embodiments, updating the parameters of the initial online network and the initial target network based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state includes: Based on the sample state, obtain a corresponding plurality of sample actions, and update an average estimation strategy for the plurality of sample actions; Based on the Q value corresponding to each sample action and the target Q value, the strategy of each sample action is updated.

[0007] In some embodiments, the update formula of the average estimation strategy is as follows: ; ; in, represents the updated average estimation strategy, represents the average estimation strategy, Indicates the sample status The corresponding Sample actions strategy, Indicates the sample status The number of occurrences, Indicates the sample status Updated value of the number of occurrences; The update formula of the strategy for each sample action is as follows: ; in, Indicates sample actions The update amount of the strategy, Based on the sample status The corresponding target Q value is determined.

[0008] In some embodiments, updating the parameters of the initial online network and the initial target network based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state includes: Calculate a loss function value based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state; Based on the loss function value, parameters of the initial online network and the initial target network are updated.

[0009] In some embodiments, the loss function value is calculated as follows: ; in, is the loss function value, express Sample status at time The corresponding target Q value is, express Sample status at time The corresponding Sample actions The reward is to take a sample action on the sample airport pavement unit The comprehensive benefit-cost ratio after the corresponding maintenance measures, express Sample status at time Corresponding sample actions The maximum Q value, express Sample status at time Corresponding sample actions The Q value, is the discount rate.

[0010] In some embodiments, the adaptively updating the parameters of the state prediction layer based on the extended Kalman filter includes: Determine an observed value of a current state of the target airport pavement unit and a predicted value of the current state of the target airport pavement unit, wherein the predicted value of the current state is obtained by inputting a historical state of the target airport pavement unit into the state prediction layer for prediction; Determining a Kalman gain of an extended Kalman filter based on the prediction noise of the state prediction layer; Based on the observed value of the current state, the predicted value of the current state and the Kalman gain, the parameters of the state prediction layer are adaptively updated.

[0011] In a second aspect, the present invention further provides an airport pavement maintenance decision-making device, comprising: An acquisition unit, used to acquire the current state of the target airport pavement unit; A decision unit, used to input the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model includes a deep Q network DQN layer and a state prediction layer; An updating unit, configured to adaptively update the parameters of the state prediction layer based on an extended Kalman filter; The decision model is obtained by training using the WoLF-PHC algorithm based on the sample states of the sample airport pavement units and the target Q values ​​corresponding to the sample states.

[0012] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-mentioned airport pavement maintenance decision-making methods.

[0013] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described airport pavement maintenance decision-making methods.

[0014] The airport pavement maintenance decision method, device, electronic device and storage medium provided by the present invention obtain the current state of the target airport pavement unit, input the current state of the target airport pavement unit into a pre-trained decision model, and obtain the optimal maintenance measures of the target airport pavement unit output by the decision model. The decision model includes a deep Q network DQN layer and a state prediction layer. Based on the extended Kalman filter, the parameters of the state prediction layer are adaptively updated, which reduces the cost of model training and improves the robustness of the decision model. It is suitable for application scenarios with measurement errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1It is a flowchart of an airport pavement maintenance decision-making method provided by an embodiment of the present invention; Figure 2 It is a flowchart of the training process of the decision model provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of an airport pavement maintenance decision-making device provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] At present, the airport pavement maintenance decision-making method based on reinforcement learning mainly uses the neural network model for state prediction in terms of state update, but the model parameters will not be updated dynamically, especially when there are errors in measurement or inaccurate predictions, the model can only be retrained, which increases the training cost; in addition, the existing improvement method for the Deep Q-Network (DQN) does not consider the update efficiency of the learning strategy, which also increases the training cost of the reinforcement learning method.

[0019] To this end, the embodiments of the present invention provide an airport pavement maintenance decision method, device, electronic device and storage medium, which obtains the current state of the target airport pavement unit and inputs the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures of the target airport pavement unit output by the decision model. The decision model includes a deep Q network DQN layer and a state prediction layer. Based on the extended Kalman filter, the parameters of the state prediction layer are adaptively updated. The embodiments of the present invention can reduce the cost of model training, improve the robustness of the decision model, and are suitable for application scenarios with measurement errors.

[0020] Figure 1 The following is a flow chart of the airport pavement maintenance decision-making method provided by the embodiment of the present invention. Figure 1 As shown, a method for decision-making on airport pavement maintenance is provided, comprising the following steps: step 110, step 120, and step 130. The steps of the method flow are only a possible implementation of the present invention.

[0021] Step 110: Obtain the current status of the target airport pavement unit.

[0022] The current state is used to describe the current structural state, performance state, and material information of the target airport pavement unit.

[0023] Optionally, the current state includes at least: a current pavement condition index, a current airport pavement strength index, a current bottom plate void rate, and a current flatness.

[0024] Optionally, the target airport pavement is divided to obtain a plurality of target airport pavement units.

[0025] The target airport pavement refers to a cement concrete pavement, and the target airport pavement unit includes a plurality of cement concrete slabs.

[0026] It should be noted that the zoning rules for the target airport pavement follow the three levels of “location-area-unit”.

[0027] Optionally, the target airport pavement is divided into different parts according to the physical structure differences of the runway and the influence of aircraft loads, such as: runway, parallel taxiway, connecting road, apron and asphalt transition section; the same part is divided into different areas based on the pavement structure and load level, and each area is divided into multiple units.

[0028] Step 120: Input the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures of the target airport pavement unit output by the decision model; the decision model includes a deep Q network DQN layer and a state prediction layer.

[0029] The decision model is trained by using the WoLF-PHC algorithm based on the sample states of the sample airport pavement units and the target Q values ​​corresponding to the sample states.

[0030] Among them, DQN combines deep learning and reinforcement learning algorithms to solve complex problems with high-dimensional state space; the DQN layer includes an online network and a target network. The online network is used to generate a Q-value estimate of the current strategy, while the target network is used to calculate the target Q-value to stabilize the training process. The target network periodically copies weights from the online network to help reduce instability during training.

[0031] It should be noted that reinforcement learning is an important branch of machine learning. In reinforcement learning, an intelligent agent (such as a target airport pavement unit) takes different actions according to its own different states in the hope of obtaining maximum rewards from the environment.

[0032] Maintenance measures include but are not limited to: 1) Daily maintenance: The overall condition of the pavement is good. If damage is found, it can be repaired in time without taking any measures; 2) Daily maintenance, attention to void observation: The surface condition of the pavement is good, without serious damage, but there is some void in the base or foundation, and it is necessary to strengthen the void observation under the pavement panel; 3) Targeted repair: The road surface is more damaged. In order to prevent the development of the disease, targeted repair work is needed for some diseases, such as crack filling, shallow repair, partial thickness repair, full thickness repair, plate grinding, prefabricated hoisting and plate replacement, etc.; 4) Replace the caulking material and pay attention to the observation of voids: The overall condition of the pavement is good, but the caulking material is quite damaged. The caulking material needs to be replaced as a whole, and the observation of voids under the pavement slab needs to be strengthened; 5) Grouting repair: The surface condition of the pavement is good, but the base or foundation is seriously hollowed out, so grouting repair is required for the pavement in the area; 6) Targeted repair and grouting repair: The road surface is more damaged and the voids are more serious, so grouting repair is required, and targeted repair is taken for some diseases; 7) Restricted operation: The pavement structure performance is poor and the load-bearing capacity does not meet the use requirements. It is necessary to limit the operating models and total load in this area; 8) Overlay: If the pavement bearing capacity or flatness does not meet the requirements, structural or functional overlay is required in the area to ensure normal operation.

[0033] It should be noted that the environment can be considered as the external environment of the target airport pavement unit. The problem considered by reinforcement learning is the task of interaction between the intelligent agent and the environment. The role of the environment is to give the state and immediate reward at the next moment. Therefore, in the maintenance decision-making problem, the environment can be abstracted into a state prediction layer.

[0034] Step 130: Based on the extended Kalman filter, the parameters of the state prediction layer are adaptively updated.

[0035] Among them, the Extended Kalman Filter (EKF) is a state estimation method for nonlinear dynamic systems. EKF is an extension of the classic Kalman filter. EKF estimates the state by linearizing nonlinear functions.

[0036] In some embodiments, step 130 adaptively updates the parameters of the state prediction layer based on the extended Kalman filter, including: Step 131, determining an observed value of the current state of the target airport pavement unit and a predicted value of the current state of the target airport pavement unit, wherein the predicted value of the current state is obtained by inputting the historical state of the target airport pavement unit into the state prediction layer for prediction; Step 132: determining the Kalman gain of the extended Kalman filter based on the prediction noise of the state prediction layer; Step 133: Based on the observed value of the current state, the predicted value of the current state and the Kalman gain, the parameters of the state prediction layer are adaptively updated.

[0037] Optionally, the historical status includes: historical pavement condition index, historical airport pavement strength index, historical slab bottom void rate, historical flatness, historical traffic volume, historical structural strength, historical surface thickness, historical base type, historical environmental conditions and historical material type.

[0038] Optionally, the predicted value of the current state includes: a current pavement condition index predicted value, a current airport pavement strength index predicted value, a current bottom plate void rate predicted value and a current flatness predicted value.

[0039] Optionally, the parameter adaptive correction formula of the state prediction layer is as follows: ; ; ; ; in, represents the corrected parameters of the state prediction layer, Indicates the current state prediction layer The parameters of the moment, represents the Kalman gain, Indicates the current The observed value of the state at time , Represents the current state prediction layer output The predicted value of the state at time represents the input of the state prediction layer, that is, the historical state, represents an all-one diagonal matrix, express The transposed matrix of for The prior covariance of time, for The posterior covariance at time , express The posterior covariance at time , is the observation error, is the identity matrix.

[0040] Optionally, the parameters of the state prediction layer are iteratively updated until a stop condition is met, such as there is no need to use this method to make maintenance decisions for the target airport pavement unit.

[0041] In an embodiment of the present invention, by acquiring the current state of the target airport pavement unit, the current state of the target airport pavement unit is input into a pre-trained decision model to obtain the optimal maintenance measures of the target airport pavement unit output by the decision model. The decision model includes a deep Q network DQN layer and a state prediction layer. Based on the extended Kalman filter, the parameters of the state prediction layer are adaptively updated. The embodiment of the present invention can reduce the cost of model training, improve the robustness of the decision model, and is suitable for application scenarios with measurement errors.

[0042] Figure 2 A flowchart of the training process of the decision model provided by the embodiment of the present invention is shown in FIG. Figure 2 As shown, in some embodiments, the training process of the decision model includes: Step 210: construct an initial decision model and an experience replay buffer, the initial decision model includes an initial DQN layer and a pre-trained state prediction layer, and the initial DQN layer includes an initial online network and an initial target network; Step 220, obtaining the sample state of the sample airport pavement unit at time t, the sample state including: sample pavement condition index, sample airport pavement strength index, sample plate bottom void rate and sample flatness; Step 230: searching for corresponding sample actions according to the sample state at time t, where the sample actions include sample maintenance measures for the sample airport pavement unit; Step 240: input the sample state at time t to the initial state prediction layer to obtain the sample state at time t+1 output by the initial state prediction layer; Step 250: Determine a reward for the sample action based on the result of the interaction between the sample action and the environment; Step 260: store the sample state at time t, the sample action, the reward of the sample action, and the sample state at time t+1 in the experience playback buffer; Step 270: randomly extract a sample state from the experience replay buffer, predict the Q value corresponding to the sample state based on the initial online network, and predict the target Q value corresponding to the sample state based on the initial target network; Step 280: Based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, update the parameters of the initial online network and the initial target network, start the loop by searching for the corresponding sample action according to the sample state at time t, and iteratively optimize the parameters of the initial online network and the initial target network until the iteration termination condition is met to obtain the DQN layer.

[0043] Optionally, sample maintenance measures include, but are not limited to: daily maintenance, void observation, targeted repair, replacement of caulking materials, grouting repair, restricted operation, and additional paving.

[0044] Among them, the state prediction layer is trained based on the historical state.

[0045] Optionally, the Q value of the sample airport pavement unit is initialized to 0.

[0046] Optionally, an ε-greedy strategy is used to search for the sample action corresponding to the sample state at time t.

[0047] Among them, the ε-greedy strategy is a commonly used reinforcement learning strategy for balancing exploration and exploitation. Its purpose is to utilize the currently known information to obtain maximum benefits during the action selection process, while exploring new possibilities to discover better strategies.

[0048] Optionally, the online network, the target network and the state prediction layer all include multi-layer perceptron networks.

[0049] In some embodiments, based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, updating the parameters of the initial online network and the initial target network includes: Based on the sample state, obtain the corresponding multiple sample actions, and update the average estimation strategy of the multiple sample actions; Based on the Q value corresponding to each sample action and the target Q value, the strategy of each sample action is updated.

[0050] It can be understood that by using multiple sample actions to update the average estimated strategy, it helps to improve the stability and accuracy of the strategy estimation and reduce the deviation caused by a single sample; by independently updating each sample action, the complexity of the environment can be better captured and the performance of the strategy in different states can be improved; the average estimate of multiple samples can make the learning process smoother, thereby accelerating strategy convergence; and by making full use of sample action data, it helps to improve the efficiency and effectiveness of strategy optimization.

[0051] In some embodiments, the update formula of the average estimation strategy is as follows: ; ; in, represents the updated average estimation strategy, represents the average estimation strategy, Indicates the sample status The corresponding Sample actions strategy, Indicates the sample status The number of occurrences, Indicates the sample status Updated value of the number of occurrences; The update formula of the strategy for each sample action is as follows: ; in, Indicates sample actions The update amount of the strategy, Based on the sample status The corresponding target Q value is determined.

[0052] Optionally, The calculation formula is as follows:

[0053] in, Indicates the sample status The sample action corresponding to the maximum Q value is Indicates the sample status Corresponding sample actions The Q value, Indicates the sample status The corresponding maximum Q value; Indicates sample actions The corresponding strategy adjustment amount, Indicates sample actions The corresponding adjustment amount of the strategy, Indicates sample actions The corresponding strategy, ) indicates a sample action The corresponding strategy, represents the adjustment amount of the strategy parameter, Indicates the number of sample actions.

[0054] Optionally, The calculation formula is as follows:

[0055] in, and are the adjustment amounts of strategy parameters. and , Indicates the sample status The corresponding sample action set.

[0056] In some embodiments, based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, updating the parameters of the initial online network and the initial target network includes: Calculate the loss function value based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state; Based on the loss function value, the parameters of the initial online network and the initial target network are updated.

[0057] In some embodiments, the loss function value is calculated as follows: ; in, is the loss function value, express Sample status at time The corresponding target Q value is, express Sample status at time The corresponding Sample actions The reward is to take sample actions for the sample airport pavement unit The comprehensive benefit-cost ratio after the corresponding maintenance measures, express Sample status at time Corresponding sample actions The maximum Q value, express Sample status at time Corresponding sample actions The Q value, is the discount rate.

[0058] Optionally, The calculation formula is as follows:

[0059] in, Indicates the sample status The total number of indicators, Indicates indicators, Indicates The reward corresponding to the indicator is Represents the sample airport pavement unit According to The maintenance benefit of the indicator maintenance at time t is: Represents the sample airport pavement unit According to The maintenance cost of the indicator at time t, Represents the sample airport pavement unit According to Indicators take maintenance measures at time t The maintenance benefits generated Represents the sample airport pavement unit According to The maintenance benefit of an indicator when no maintenance measures are taken at time t, Indicates the last maintenance time.

[0060] The following is a description of an airport pavement maintenance decision device provided by an embodiment of the present invention. The airport pavement maintenance decision device described below and the airport pavement maintenance decision method described above can be referenced to each other.

[0061] Figure 3 A schematic diagram of the structure of the airport pavement maintenance decision-making device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the airport pavement maintenance decision device 300 includes: An acquisition unit 310 is used to acquire the current state of the target airport pavement unit; A decision unit 320 is used to input the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model includes a deep Q network DQN layer and a state prediction layer; An updating unit 330, configured to adaptively update parameters of the state prediction layer based on an extended Kalman filter; The decision model is trained by using the WoLF-PHC algorithm based on the sample states of the sample airport pavement units and the target Q values ​​corresponding to the sample states.

[0062] Optionally, the training process of the decision model includes: Construct an initial decision model and experience replay buffer. The initial decision model includes an initial DQN layer and a pre-trained state prediction layer. The initial DQN layer includes an initial online network and an initial target network. Obtain the sample status of the sample airport pavement unit at time t, the sample status includes: sample pavement condition index, sample airport pavement strength index, sample plate bottom void rate and sample flatness; According to the sample state at time t, the corresponding sample action is searched, and the sample action includes the sample maintenance measures of the sample airport pavement unit; Input the sample state at time t into the initial state prediction layer, and obtain the sample state at time t+1 output by the initial state prediction layer; Determine the reward for the sample action based on the result of the interaction between the sample action and the environment; Store the sample state at time t, sample action, reward of sample action, and sample state at time t+1 into the experience replay buffer; Randomly extract a sample state from the experience replay buffer, predict the Q value corresponding to the sample state based on the initial online network, and predict the target Q value corresponding to the sample state based on the initial target network; Based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated, and the cycle starts from searching for the corresponding sample action according to the sample state at time t, and the parameters of the initial online network and the initial target network are iteratively optimized until the iteration termination condition is met to obtain the DQN layer.

[0063] Optionally, based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, updating the parameters of the initial online network and the initial target network includes: Based on the sample state, obtain the corresponding multiple sample actions, and update the average estimation strategy of the multiple sample actions; Based on the Q value corresponding to each sample action and the target Q value, the strategy of each sample action is updated.

[0064] Optionally, the update formula of the average estimation strategy is as follows: ; ; in, represents the updated average estimation strategy, represents the average estimation strategy, Indicates the sample status The corresponding Sample actions strategy, Indicates the sample status The number of occurrences, Indicates the sample status Updated value of the number of occurrences; The update formula of the strategy for each sample action is as follows: ; in, Indicates sample actions The update amount of the strategy, Based on the sample status The corresponding target Q value is determined.

[0065] Optionally, based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, updating the parameters of the initial online network and the initial target network includes: Calculate the loss function value based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state; Based on the loss function value, the parameters of the initial online network and the initial target network are updated.

[0066] Optionally, the loss function value is calculated as follows: ; in, is the loss function value, express Sample status at time The corresponding target Q value is, express Sample status at time The corresponding Sample actions The reward is to take sample actions for the sample airport pavement unit The comprehensive benefit-cost ratio after the corresponding maintenance measures, express Sample status at time Corresponding sample actions The maximum Q value, express Sample status at time Corresponding sample actions The Q value, is the discount rate.

[0067] Optionally, based on the extended Kalman filter, the parameters of the state prediction layer are adaptively updated, including: Determine the observed value of the current state of the target airport pavement unit and the predicted value of the current state of the target airport pavement unit, where the predicted value of the current state is obtained by inputting the historical state of the target airport pavement unit into the state prediction layer for prediction; Determining a Kalman gain of an extended Kalman filter based on prediction noise of a state prediction layer; Based on the observed value of the current state, the predicted value of the current state and the Kalman gain, the parameters of the state prediction layer are adaptively updated.

[0068] It should be noted here that the airport pavement maintenance decision-making device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned airport pavement maintenance decision-making method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be specifically described here.

[0069] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the airport pavement maintenance decision method, which includes: obtaining the current state of the target airport pavement unit; inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model includes a deep Q network DQN layer and a state prediction layer; based on the extended Kalman filter, the parameters of the state prediction layer are adaptively updated; wherein the decision model is trained using the WoLF-PHC algorithm based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state.

[0070] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0071] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the airport pavement maintenance decision method provided by the above-mentioned methods, the method comprising: obtaining the current state of the target airport pavement unit; inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measures of the target airport pavement unit output by the decision model; the decision model comprises a deep Q network DQN layer and a state prediction layer; based on an extended Kalman filter, the parameters of the state prediction layer are adaptively updated; wherein the decision model is trained using the WoLF-PHC algorithm based on the sample state of the sample airport pavement unit and the target Q value corresponding to the sample state.

[0072] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for decision-making on airport pavement maintenance, characterized in that: include: Get the current status of the target airport pavement unit; Inputting the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model includes a deep Q network DQN layer and a state prediction layer; Based on the extended Kalman filter, adaptively updating the parameters of the state prediction layer; The decision model is obtained by training using the WoLF-PHC algorithm based on the sample states of the sample airport pavement units and the target Q values ​​corresponding to the sample states.

2. The airport pavement maintenance decision-making method according to claim 1 is characterized in that: The training process of the decision model includes: Constructing an initial decision model and an experience replay buffer, wherein the initial decision model includes an initial DQN layer and a pre-trained state prediction layer, and the initial DQN layer includes an initial online network and an initial target network; Obtaining the sample state of the sample airport pavement unit at time t, wherein the sample state includes: a sample pavement condition index, a sample airport pavement strength index, a sample plate bottom void rate, and a sample flatness; According to the sample state at time t, searching for a corresponding sample action, wherein the sample action includes a sample maintenance measure of the sample airport pavement unit; Input the sample state at time t into the initial state prediction layer to obtain the sample state at time t+1 output by the initial state prediction layer; Determining a reward for the sample action according to a result of the interaction between the sample action and the environment; The sample state at time t, the sample action, the reward of the sample action and the sample state at time t+1 are stored in the experience replay buffer; Randomly extracting a sample state from the experience replay buffer, predicting a Q value corresponding to the sample state based on the initial online network, and predicting a target Q value corresponding to the sample state based on the initial target network; Based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state, the parameters of the initial online network and the initial target network are updated, and a loop is started by searching for the corresponding sample action according to the sample state at time t, and the parameters of the initial online network and the initial target network are iteratively optimized until the iteration termination condition is met to obtain the DQN layer.

3. The airport pavement maintenance decision-making method according to claim 2 is characterized in that: The updating of parameters of the initial online network and the initial target network based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state includes: Based on the sample state, obtain a corresponding plurality of sample actions, and update an average estimation strategy for the plurality of sample actions; Based on the Q value corresponding to each sample action and the target Q value, the strategy of each sample action is updated.

4. The airport pavement maintenance decision-making method according to claim 3 is characterized in that: The update formula of the average estimation strategy is as follows: ； ； in, represents the updated average estimation strategy, represents the average estimation strategy, Indicates the sample status The corresponding Sample actions strategy, Indicates the sample status The number of occurrences, Indicates the sample status Updated value of the number of occurrences; The update formula of the strategy for each sample action is as follows: ； in, Indicates sample actions The update amount of the strategy, Based on the sample status The corresponding target Q value is determined.

5. The airport pavement maintenance decision-making method according to claim 2 is characterized in that: The updating of parameters of the initial online network and the initial target network based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state includes: Calculate a loss function value based on the Q value corresponding to the sample state and the target Q value corresponding to the sample state; Based on the loss function value, parameters of the initial online network and the initial target network are updated.

6. The airport pavement maintenance decision-making method according to claim 5 is characterized in that: The calculation formula of the loss function value is as follows: ； in, is the loss function value, express Sample status at time The corresponding target Q value is, express Sample status at time The corresponding Sample actions The reward is to take a sample action on the sample airport pavement unit The comprehensive benefit-cost ratio after the corresponding maintenance measures, express Sample status at time Corresponding sample actions The maximum Q value, express Sample status at time Corresponding sample actions The Q value, is the discount rate.

7. The airport pavement maintenance decision-making method according to any one of claims 2 to 6, characterized in that: The method of adaptively updating the parameters of the state prediction layer based on the extended Kalman filter includes: Determine an observed value of a current state of the target airport pavement unit and a predicted value of the current state of the target airport pavement unit, wherein the predicted value of the current state is obtained by inputting a historical state of the target airport pavement unit into the state prediction layer for prediction; Determining a Kalman gain of an extended Kalman filter based on the prediction noise of the state prediction layer; Based on the observed value of the current state, the predicted value of the current state and the Kalman gain, the parameters of the state prediction layer are adaptively updated.

8. An airport pavement maintenance decision-making device, characterized in that: include: An acquisition unit, used to acquire the current state of the target airport pavement unit; A decision unit, used to input the current state of the target airport pavement unit into a pre-trained decision model to obtain the optimal maintenance measure of the target airport pavement unit output by the decision model; the decision model includes a deep Q network DQN layer and a state prediction layer; An updating unit, configured to adaptively update the parameters of the state prediction layer based on an extended Kalman filter; The decision model is obtained by training using the WoLF-PHC algorithm based on the sample states of the sample airport pavement units and the target Q values ​​corresponding to the sample states.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the airport pavement maintenance decision-making method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the airport pavement maintenance decision method as described in any one of claims 1 to 7 is implemented.

Images