Geological disaster emergency response path planning method based on reinforcement learning

Abstract

The invention discloses a geological disaster emergency response path planning method based on reinforcement learning. The geological disaster emergency response path planning method comprises the following steps: S1, fusing and processing multi-source data; s2, constructing a four-dimensional space-time model based on the fused data; s3, multi-agent collaborative decision making is carried out, and a mixed agent system based on a Safe-PPO improved algorithm is deployed; a multi-objective optimization algorithm is adopted to balance the rescue efficiency and safety; deciding and outputting a leading path and a plurality of alternative paths; s4, performing conflict detection on the action of the path planning agent; s5, carrying out preferential selection on the paths subjected to conflict detection and global optimization; s6, issuing the path to a terminal for rescuing the action personnel; s7, dynamically monitoring the environment; s8, judging whether a path needs to be re-planned or not according to environment dynamic monitoring data; and S9, completing the task. According to the method, through reinforcement learning and multi-agent collaborative architecture, the risk penalty term is embedded in the reward function through the improved Safe-PPO algorithm, and the accuracy of path planning is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of geological disasters, and particularly relates to a geological disaster emergency response path planning method based on reinforcement learning. BACKGROUND

[0002] Geological disasters (such as landslides and debris flows) have the characteristics of strong suddenness and complex environmental evolution. Traditional emergency path planning mainly relies on static geographic information systems (GIS) and artificial experience decision-making. Existing models are mostly constructed based on pre-disaster geological data, and cannot real-time fuse multi-source heterogeneous data such as satellites and sensors, resulting in that path planning lags behind disaster evolution; resource scheduling, risk assessment and path planning are processed separately, lacking a multi-objective collaborative optimization mechanism, and being prone to imbalance between rescue efficiency and safety. SUMMARY

[0003] Therefore, the present application provides a geological disaster emergency response path planning method based on reinforcement learning to solve the problems in the prior art.

[0004] In order to achieve the above purpose, the present application provides the following technical scheme: The geological disaster emergency response path planning method based on reinforcement learning comprises the following steps: S1. Fusion and processing of multi-source data, key elements of the fused data including three-dimensional geological structure, secondary disaster risk distribution, and road passability; S2. Construction of a four-dimensional space-time model based on the fused data, depicting geological interfaces, introducing an expert knowledge base, and correcting uncertain parameters in the model; deploying an incremental modeling engine to update the model when environmental changes are monitored; S3. Multi-agent collaborative decision-making, deploying a hybrid agent system based on the improved Safe-PPO algorithm, the hybrid agent system including a path planning agent responsible for spatial search, a resource scheduling agent managing rescue supplies, and a risk assessment agent predicting secondary disasters; the hybrid agent system adopts a multi-objective optimization algorithm to balance rescue efficiency and safety; the decision output is a dominant path and several alternative paths; S4. Conflict detection of the actions of the path planning agent; global optimization of the output alternative paths; S5. Optimal selection of the paths after conflict detection and global optimization; S6. Path delivery to the terminal of rescue personnel, recording data generated during the rescue process, and feeding back to the rescue action execution and the hybrid agent system; S7. Dynamic monitoring of the environment; S8. According to the data of environmental dynamic monitoring, it is judged whether the path needs to be re-planned; when it is determined that the path needs to be re-planned, the four-dimensional space-time model incremental update is triggered; when it is determined that the path does not need to be re-planned, the rescue task is continued to be executed until the task is completed. S9. The task is completed.

[0005] Further: the multi-source data includes satellite remote sensing image data, ground sensor network data, unmanned aerial vehicle inspection data, historical disaster database and real-time reported information data of on-site emergency team.

[0006] Further: in S2, the landslide range predicted by the model is compared with the actual disaster situation through historical disaster scene backtracking verification, and the model parameters are adjusted to make the prediction error ≤15%.

[0007] Further: in S3, the optimization objectives of the multi-objective optimization algorithm include: maximizing the covered population, minimizing the cost, and minimizing the response time, and the dominant path confidence needs to be > 85%.

[0008] Further: the improvement derivation process of the Safe-PPO algorithm is as follows: The calculation formula of the original PPO loss function is ; Where θ is the current parameter vector of the policy network (Policy Network); πθ represents the new policy, and πold represents the old policy. represents the probability of selecting action a under state s based on the policy with parameters θ, represents the probability of selecting action a under state s based on the old policy (the policy after the last update, πold). π (s,a) is the advantage function, which represents the difference between the actual return of action a in state s and the benchmark return. is the clipping coefficient, which controls the policy update step size. represents the clipping function that limits x in the interval; E is the book sequence expectation, which takes the average value of all state-action pairs. The calculation formula of the advantage function A π (s,a) is: ; Where Q π (s,a) is the action value function, which represents the expected total reward of following the policy π after performing action a in state s. π (s): state value function, which represents the expected total reward of following the policy π in state s. The action value function Q π(s, a) can be expanded by Bellman equation, that is: . where r(s, a) is the reward function, γ is the discount factor (typical value 0.95), s' is the next state after performing action a; The formula for calculating the reward function r(s, a) is: . where w1, w2, w3 are weight coefficients; In order to ensure the correctness of the calculation, the formula for calculating the modified reward function r'(s, a) after adding risk penalty is: . where λ is the risk weight factor (typical value 0.15), which controls the influence strength of risk on the total reward; wi is the weight coefficient of the i-th risk; Risk i (s, a) represents the probability of occurrence of the i-th risk when action a is performed in state s.

[0009] Further: the conflict detection adopts a rule-based checking system to ensure that the path planning does not violate physical limitations; and the digital twin platform is used to simulate the execution of actions to verify whether the path avoids dangerous areas.

[0010] Further: the path selection process of S5 is as follows: A four-dimensional evaluation matrix containing path length, passability, secondary disaster risk, and rescue point coverage is constructed; the entropy weight method is introduced to determine the weight of each index, and the closeness degree of each scheme to the ideal solution is calculated; a threshold τ is set, and only the paths with closeness degree > τ are retained as candidates; In addition, according to real-time monitoring data, the path weight is recalculated every 10-30 minutes to ensure that the optimization result adapts to environmental changes.

[0011] Further: the re-planning threshold is set to three levels, the first level requires immediate re-planning, the second level requires re-planning within 5-20 minutes; and the third level requires evaluation within 20-60 minutes; where the threshold determination conditions for the third level of re-planning are: the first level, loss of main path passability or discovery of SOS signal; the second level, secondary disaster probability > 60% or resource consumption exceeding budget by 20%; the third level, environmental factor change rate > 30% or execution efficiency continuously below the benchmark value of 85%.

[0012] Further: the re-planning algorithm adopts a rolling horizon control strategy, which only optimizes the path within 1-3 hours in the future each time; and historical effective path segments are introduced to accelerate the generation of new schemes and shorten the re-planning time.

[0013] Further: the termination condition of the task completion is that all the marked rescue points complete the disposal, the coverage is > 95%, the monitoring shows that the disaster spread stops for more than 48 hours, and the field command department confirms the end of the task.

[0014] The application has the following advantages: the application improves the Safe-PPO algorithm by embedding a risk penalty term in the reward function through reinforcement learning and a multi-agent collaborative architecture, thereby improving the accuracy of path output; the reinforcement learning algorithm is deeply coupled with real-time environmental data, thereby realizing the intelligentization of the whole process from disaster perception to action execution.

[0015] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes, structures shown in the drawings should not be regarded as limiting conditions in the implementation of the present application; for example, based on the technical concepts disclosed in the present application and the exemplary drawings, those skilled in the art can easily make routine adjustments or further optimizations to some units (components) in terms of increase / decrease / attribute division, specific shape, positional relationship, connection mode, size ratio relationship, etc.

[0017] Figure 1 An implementation flowchart of the reinforcement learning-based geological disaster emergency response path planning method provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0018] The embodiments of the present application are described below by specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosed content. Obviously, the described embodiments are part of the embodiments of the present application, not all. It should be understood that these embodiments are only for further illustration of the present application, and cannot be understood as a limitation on the protection scope of the present application. The technical engineers in the field can make some non-essential improvements and adjustments to the present application according to the above-mentioned content of the present application; based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0019] Please refer to Figure 1 The reinforcement learning-based geological disaster emergency response path planning method comprises the following steps: When a geological disaster event occurs, automatically perceive the disaster signal through the Internet of Things sensor network deployed in the disaster area, which includes seismometers, tiltmeters, water level sensors, etc.; when the monitoring data exceeds the preset threshold, immediately start the emergency response process; for example, when the ground acceleration > 0.5g, the water level rising rate > 5cm / h, start the emergency response process.

[0020] The initialization module synchronously loads the pre-trained disaster scene database, which contains historical disaster data, geological structure information, climate characteristics, etc. structured data, providing a benchmark reference for subsequent processing.

[0021] S1. Multi-source data fusion processing Integrate satellite remote sensing image data, ground sensor network data, unmanned aerial vehicle inspection data, historical disaster database, and real-time reported information data of the on-site emergency team.

[0022] Use Kalman filtering to suppress noise in sensor data, use wavelet transform to extract disaster features in satellite images, such as crack length and landslide volume; then introduce MapGIS product data integration specifications to establish a unified data exchange format, ensuring semantic interoperability of multi-source heterogeneous data.

[0023] Deploy a deep belief network (DBN) for feature-level fusion, extract spatial-temporal features of disaster elements layer by layer through a restricted Boltzmann machine (RBM); the fused data contains key elements such as three-dimensional geological structure, secondary disaster risk distribution, and road passability.

[0024] S2. Environment dynamic modeling Based on the fusion data, construct a four-dimensional spatio-temporal model, the four dimensions in the four-dimensional spatio-temporal model are three-dimensional space + time dimension, use NURBS surface fitting technology to depict geological interfaces, including faults, landslide surfaces, etc. geological interfaces; introduce expert knowledge base, correct the uncertainty parameters in the model through Bayesian network; uncertainty parameters include rock mass friction angle, permeability coefficient, etc.

[0025] Deploy an incremental modeling engine, when environmental changes are detected, use local grid refinement technology (LOD) for model updating; environmental changes include new cracks, rising water levels, etc.

[0026] Through historical disaster scene backtracking verification, compare the landslide range predicted by the model with the actual disaster situation, adjust the model parameters to make the prediction error ≤15%.

[0027] S3. Multi-agent collaborative decision-making Deploy a hybrid agent system based on the PPO algorithm. The hybrid agent system includes a path planning agent responsible for spatial search, a resource scheduling agent for managing relief supplies, and a risk assessment agent for predicting secondary disasters. Communication between multiple agents is achieved through a blackboard system to share information.

[0028] The actions of the path planning agent include continuous actions such as moving forward, turning, and obstacle avoidance, while the actions of the resource scheduling agent include discrete actions such as resource allocation and team deployment. The Safe-PPO algorithm is introduced to improve the reward function by adding a risk factor term, such as risk weights for water areas and fault zones.

[0029] In addition, a multi-objective optimization algorithm (NSGA-III) is used to balance rescue efficiency and safety. The optimization objectives include: maximizing the coverage population, minimizing cost, and minimizing response time. In this embodiment, the weights of the above three optimization objectives are: maximizing the coverage population weight 0.4, minimizing cost weight 0.3, and minimizing response time weight 0.3. The decision output includes the dominant path and three alternative paths, and the confidence of the dominant path must be >85%.

[0030] The specific process of improving the Safe-PPO algorithm is as follows: The original formula for calculating the PPO loss function is as follows: ; Where θ is the current parameter vector of the Policy Network; πθ represents the new policy, and πold represents the old policy; Let θ represent the probability that a policy based on parameter θ will choose action a in state s. Let A represent the probability that the old policy (the policy updated in the previous round) will choose action a in state s; π (s,a) is the advantage function, which represents the difference between the actual benefit of action a in state s and the benchmark benefit. The pruning factor is the control strategy update step size; This means restricting x to The pruning function within the interval; E is the expected value of the book sequence, which is the average value over all state-action relationships.

[0031] The formula for calculating the dominance function is: ; Among them, Q π (s,a) is the action value function, representing the expected total reward for following policy π after performing action a in state s; V π(s): State value function, representing the expected total reward for following policy π in state s.

[0032] Action value function Q π (s,a) can be expanded using the Bellman equation, i.e.: ; Where r(s,a) is the reward function, γ is the discount factor (typically 0.95), and s′ is the next state after performing action a.

[0033] The formula for calculating the reward function r(s,a) is: ; Where w1, w2, and w3 are weighting coefficients.

[0034] To ensure the accuracy of the calculation, the formula for the modified reward function r′(s,a) after adding risk penalty is as follows: ; Wherein, λ is the risk weighting factor (typical value 0.15), which controls the strength of the impact of risk on the total reward; wi is the weighting coefficient of the i-th type of risk; Risk i (s,a) represents the probability of the i-th type of risk occurring when action a is performed in state s.

[0035] Suppose there are two candidate actions in state s: Action 1: High return, high risk (r=10, risk value 0.8); Action 2: Low return, low risk (r=5, risk value 0.1).

[0036] Original PPO (No Risk Penalty): Action 1's Q π The higher the value, the larger the advantage function Aπ, and the more likely the policy update will choose action 1.

[0037] Safe-PPO (with risk penalty): Corrected reward for action 1 r′=10-0.15•(0.8)=9.88; Corrected reward for action 2 r′=5-0.15•(0.1)=4.985; Advantage function A for action 1 π If the price drops, the strategy update may shift to action 2.

[0038] In addition, in this embodiment, a scenario is set where the disaster area is a 5km×5km region, the initial path is A→B→C (total length 12km, passability score 0.75), and the secondary disaster risk is a mudslide probability of 45% (weight 0.35).

[0039] Then the path agent generates candidate path 1 and path 2; Among them, route 1, A→D→C, is 14km long and has a throughput of 0.68; route 2, A→B→E→C, is 16km long and has a throughput of 0.82.

[0040] At this point, the risk agent evaluates the two paths: Path 1, which crosses the high-risk debris flow area, has a risk value of 0.72; Path 2, which avoids the risk area, has a risk value of 0.28.

[0041] The NSGA-III algorithm was used for optimization to generate a Pareto front solution set (covering ≥90% of the population, cost ≤150,000, and time ≤2h); therefore, path 2 was selected as the dominant path (confidence 88%).

[0042] S4. Joint Action Generation and Optimization Conflict detection is required for the actions of the path planning agent. A rule-based inspection system should be used to ensure that the path planning does not violate physical constraints, such as the maximum climbing angle of 30° for vehicles.

[0043] Meanwhile, the ant colony optimization (ACO) algorithm is used to globally optimize the candidate solutions output by PPO.

[0044] In this embodiment, the pheromone evaporation coefficient ρ is set to 0.7, and the number of iterations N is set to 100. The optimized path length is shortened by 12%, and the throughput is improved by 25%.

[0045] In addition, actions are simulated using a digital twin platform to verify whether the path avoids dangerous areas, such as mudslide gullies.

[0046] S5. Dynamic Optimal Path Selection A four-dimensional evaluation matrix is ​​constructed, which includes path length, passability, secondary disaster risk, and rescue point coverage. The entropy weight method is introduced to determine the weight of each indicator. In this embodiment, the weight of secondary disaster risk can be set to 0.35.

[0047] In addition, the TOPSIS method is used to rank the schemes and calculate the closeness of each scheme to the ideal solution; a threshold τ is set to retain only the paths with a closeness greater than τ as candidates.

[0048] Based on real-time monitoring data, the path weights are recalculated every 10-30 minutes to ensure that the selection results adapt to environmental changes; in this embodiment, the path weights can be recalculated every 15 minutes.

[0049] S6. Rescue Operation Execution and Feedback Using an AR-based navigation terminal, the optimal route is overlaid in the field of vision of rescuers, and warning information is marked on key nodes, including landslide sections and water impact sections.

[0050] Record the transit time, resource consumption, and emergency handling process for all nodes in the path, including situations such as temporary route changes.

[0051] In addition, an execution performance heatmap is generated and fed back to the decision-making system every 5-20 minutes.

[0052] S7. Environmental Dynamics Monitoring and Model Correction Dynamic environmental monitoring is achieved through a monitoring network consisting of several deployed multimodal monitoring devices, including ground vibration sensors, UAV thermal imaging systems, and high-resolution satellite multispectral images.

[0053] The monitoring data stream is processed in real time, and the data buffer window is set to 10-60 seconds to ensure that no critical events are lost, including secondary landslides.

[0054] S8. Intelligent Replanning Determination Mechanism Based on the data from dynamic environmental monitoring, it is determined whether replanning is needed; if replanning is determined to be needed, an incremental update of the four-dimensional spatiotemporal model is triggered; the dynamic modeling technology of MapGIS is introduced, so that the local update time of the model is less than 2 minutes; if it is determined that replanning is not needed, the rescue mission continues until the mission is completed.

[0055] In this embodiment, the replanning threshold is set to three levels: the first level requires immediate replanning, the second level requires replanning within 15 minutes, and the third level requires evaluation within 30 minutes.

[0056] The threshold criteria for the three-level replanning are as follows: Level 1 threshold: loss of accessibility of the main route or discovery of an SOS signal; Level 2 threshold: probability of secondary disasters > 60% or resource consumption exceeds the budget by 20%; Level 3 threshold: rate of change of environmental factors > 30% or execution efficiency is consistently lower than the baseline value by 85%.

[0057] The replanning algorithm employs a rolling time control (RHC) strategy, optimizing only the path within the next 1-3 hours for each replanning iteration; and it incorporates historical effective path segments to accelerate the generation of new solutions and shorten the replanning time.

[0058] S9. Task Completion and Knowledge Accumulation The mission is terminated when all marked rescue points have been dealt with (coverage > 95%), monitoring shows that the spread of the disaster has stopped for more than 48 hours, and the on-site command confirms that the mission is over.

[0059] Once the task is completed, a case library is built, generating structured case entries that include disaster type, decision trajectory, execution log, and performance evaluation; and ontology (OWL) is used to describe case elements to support semantic retrieval.

[0060] Meanwhile, the disaster data was added to the training set, and the reinforcement learning model was updated using transfer learning techniques.

[0061] This method achieves intelligent management of the entire process from disaster perception to action execution by deeply coupling reinforcement learning algorithms with real-time environmental data.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for geological disaster emergency response path planning based on reinforcement learning, characterized in that, The method comprises the following steps: S1. Fusion and processing of multi-source data, key elements of the fused data including three-dimensional geological structure, secondary disaster risk distribution, road trafficability; S2. Construction of a four-dimensional space-time model based on the fused data, depiction of geological interfaces, introduction of an expert knowledge base, and correction of uncertain parameters in the model; S3. Multi-agent collaborative decision-making, deployment of a hybrid agent system based on the improved Safe-PPO algorithm, the hybrid agent system including a path planning agent responsible for spatial search, a resource scheduling agent managing rescue supplies, and a risk assessment agent predicting secondary disasters; the hybrid agent system uses a multi-objective optimization algorithm to balance rescue efficiency and safety; the decision output is a main path and several alternative paths; S4. Conflict detection of the actions of the path planning agent; global optimization of the output alternative paths; S5. Selection of the paths after conflict detection and global optimization; S6. The paths are sent to the terminals of rescue personnel, the data generated during the rescue process are recorded, and the data are fed back to the rescue operation and to the hybrid agent system; S7. Dynamic monitoring of the environment; S8. Determination of whether path re-planning is needed according to the data of the dynamic monitoring of the environment; when it is determined that path re-planning is needed, triggering of incremental update of the four-dimensional space-time model; when it is determined that path re-planning is not needed, continuing the rescue task until the task is completed; S9. Task completion.

2. The method of claim 1, wherein, The multi-source data include satellite remote sensing image data, ground sensor network data, unmanned aerial vehicle inspection data, historical disaster database, and information data reported by on-site emergency teams in real time. 3.The method of claim 1, wherein, In S2, the model parameters are adjusted to make the prediction error ≤ 15% by comparing the landslide range predicted by the model with the actual disaster situation through historical disaster scene backtracking verification.

4. The geological disaster emergency response path planning method based on reinforcement learning according to claim 1, characterized in that In S3, the optimization objectives of the multi-objective optimization algorithm include maximizing the covered population, minimizing the cost, and minimizing the response time, and the confidence of the main path needs to be > 85%. 5.The method of claim 1, wherein, The improvement derivation process of the Safe-PPO algorithm is as follows: The calculation formula of the original PPO loss function is where θ is the current parameter vector of the policy network (Policy Network); πθ represents the new policy, and πold represents the old policy; π θ (a|s) represents the probability of the policy based on the parameters θ selecting action a in state s, and π old (a|s) represents the probability of the old policy (the policy after the last update) selecting action a in state s; A π (s,a) is the advantage function, which represents the difference between the actual return of action a in state s and the baseline return; ∈ is the clipping coefficient, which controls the policy update step size; \clip(1-∈,1+∈,x) represents the clipping function that limits x to the interval [1-∈,1+∈]; E is the book order expectation, which takes the average value of all state-actions; Advantage function A π The formula for computing (s, a) is: A π (s,a) = Q π (s,a) - V π (s); where Q π (s, a) is the action-value function, representing the expected total reward of following policy p after performing action a in state s; V π (s) is the state-value function, representing the expected total reward of following policy p in state s. Action-value function Q π (s, a) can be expanded by the Bellman equation, i.e.: Q π (s,a) = E[r(s,a) + γ · V π (s′)]; Wherein, r(s,a) is the reward function, γ is the discount factor (typical value 0.95), s' is the next state after performing action a; The calculation formula of the reward function r(s,a) is: r(s,a) = w1·covered population gain(s,a) - w2·rescue cost(s,a) - w3·response time(s,a); Wherein, w1, w2, w3 are weight coefficients; In order to ensure the correctness of the calculation, the calculation formula of the modified reward function r'(s,a) after adding risk penalty is: ； where λ is the risk weight factor (typical value 0.15) that controls the strength of the impact of risk on the total reward; wiis the weight coefficient of the ith risk; Risk i (s, a) represents the probability of occurrence of the ith risk when action a is performed in state s. 6.The method of claim 1, wherein, The conflict detection uses a rule-based checking system to ensure that the path planning does not violate physical limitations; it also simulates the execution of actions through a digital twin platform to verify whether the path avoids dangerous areas. 7.The method of claim 1, wherein, The path selection process of S5 is as follows: A four-dimensional evaluation matrix is constructed, including path length, passability, secondary disaster risk, and rescue point coverage. The entropy weight method is introduced to determine the weight of each index, and the closeness degree of each scheme to the ideal solution is calculated. A threshold τ is set, and only the paths with closeness degree > τ are reserved as candidates. In addition, according to the real-time monitoring data, the path weight is recalculated every 10-30 minutes to ensure that the optimization result adapts to the environmental changes. 8.The method of claim 1, wherein, The re-planning threshold is set to three levels, the first level requires immediate re-planning, the second level requires re-planning within 5-20 minutes, and the third level requires evaluation within 20-60 minutes. The threshold determination conditions for the third level re-planning are: the first level threshold, the main path passability is lost or the SOS signal is found; the second level threshold, the secondary disaster probability > 60% or the resource consumption exceeds the budget by 20%; the third level threshold, the environmental factor change rate > 30% or the execution efficiency continuously below the benchmark value of 85%.

9. The method of claim 8, wherein, The re-planning algorithm adopts a rolling horizon control strategy, which only optimizes the path within 1-3 hours in the future each time; and introduces historical effective path segments to accelerate the generation of new schemes and shorten the re-planning time. 10.The method of claim 1, wherein, The termination condition of the task completion is that all the marked rescue points are completed and disposed, the coverage rate is > 95%, the monitoring shows that the disaster spread has stopped for more than 48 hours, and the field command department confirms the task completion.

Images