Reinforcement learning-driven edge defense emergency treatment dynamic line selection method

Abstract

The invention provides a reinforcement learning-driven dynamic line selection method for frontier defense emergency treatment. The method comprises the following steps of data acquisition and processing, road network topology modeling, recommended route optimization and dynamic line selection. According to the line selection method, the action route can be recommended according to the emergency event and the superior intention, the problem of line adaptive selection under the condition that different emergency events such as border public health, natural disasters and enemy sudden attack occur is solved, and a reliable guarantee is provided for border management and control support. The self-adaptive dynamic programming algorithm in reinforcement learning is adopted, environment changes are sensed in real time, better adaptability is achieved, and the method is suitable for different scenes and emergencies. According to the method, the route recommendation results of the reinforcement learning algorithm under different emergency events can be utilized, and resource allocation is optimized.

Description

Technical Field

[0001] This invention relates to the field of emergency response, specifically to a reinforcement learning-driven dynamic route selection method for border defense emergency response. Background Technology

[0002] Due to the complex geographical environment and diverse climate in border areas, in the event of an emergency, it is necessary to quickly locate nearby supplies based on the emergency and the intentions of higher authorities, and to dynamically select action routes based on the situation, weather changes, road conditions, and supply availability to ensure the effective acquisition of supplies and reduce maneuver time.

[0003] Traditional route selection methods typically start by optimizing the distance of the action route, using linear programming, neural network algorithms, and genetic algorithms to provide route recommendations in relatively static environments. However, when faced with the dynamic changes in road conditions, road types, and the environment in large-scale road networks, existing methods lack flexibility and fail to consider the impact of resource availability on the route selection results. Therefore, there is an urgent need for a method that can flexibly and dynamically select routes in complex and ever-changing environments and situations. Summary of the Invention

[0004] The purpose of this invention is to provide a reinforcement learning-driven dynamic route selection method for border defense and emergency response, which can recommend action routes based on emergency events and superior intentions, solving the problem of adaptive route selection under different emergency events such as border public health, natural disasters, and enemy attacks, and providing reliable support for border control.

[0005] The method of the present invention includes the following steps:

[0006] Step 1: Data collection and processing: Obtain the latitude and longitude data of the road network, road types (highways, urban roads, rural roads, forest roads, etc.), road damage status, weather forecast information and real-time weather information, location of sudden severe events, and historical emergency data (including event type, location, scale, and the types and quantities of materials required for disposal).

[0007] Step 2: Road network topology modeling: Calculate the comprehensive impedance of the route selection. The comprehensive impedance of the route selection consists of road impedance, risk impedance and material impedance, taking into account the impact of route travel time, possible severe weather, sudden severe events and the availability of materials.

[0008] Step 3: Optimize recommended routes: Optimize recommended routes based on the reinforcement learning algorithm to provide route references for the military in handling emergencies.

[0009] Step 4: Dynamically adjust the route: Adjust the action route online based on the reinforcement learning algorithm.

[0010] Compared with the prior art, the beneficial effects of this invention are as follows:

[0011] (1) Better adaptability: This method adopts the adaptive dynamic programming algorithm in reinforcement learning. Through continuous interaction and learning between the agent and the environment, it can perceive environmental changes in real time and has better adaptability. It is suitable for different scenarios and emergencies. It does not rely on the model and dynamically selects the action route according to real-time environmental information, such as terrain, weather, personnel and material distribution, to adapt to the needs of different scenarios.

[0012] (2) Optimize resource allocation: This method can optimize resource allocation by utilizing the route recommendation results of reinforcement learning algorithms under different emergency events, prioritizing the allocation of general materials and equipment to the high-frequency action routes; and allocating special materials and equipment to the vicinity of high-frequency special event locations, flexibly allocating limited border resources so that they can be used more efficiently.

[0013] (3) Data-driven optimization: In the actual operation of the method, the command data, environmental data, event data, and route selection data at each time are stored in the historical database. With the accumulation of data, the reinforcement learning-driven dynamic route selection method for border defense and emergency response will be continuously optimized. By learning from the historical database, the accuracy of offline route selection will be improved, the time for online optimization will be shortened, and the method can respond to complex environments more quickly. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the optimization framework for the adaptive dynamic programming algorithm.

[0015] Figure 2 Workflow diagram of adaptive dynamic programming algorithm for optimizing route selection. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] The implementation of the present invention will be described in detail below with reference to specific embodiments.

[0018] Step 1: Data Acquisition

[0019] Acquire latitude and longitude data of the road network, road types (highways, urban roads, rural roads, forest roads, etc.), and road damage status; calculate road distances between traffic nodes and label road attributes between nodes to construct an original version of the road network data model; acquire weather forecast information and the latest weather information, including meteorological data such as rainfall, snowfall, fog and haze warnings, and gale warnings; acquire real-time location data of sudden severe events; acquire historical emergency event data, including the type, location, scale of the emergency, and the types and quantities of emergency supplies required, and establish a database.

[0020] Step 2: Road Network Modeling

[0021] The overall impedance of the route selection indicates the difficulty of traversing a road when troops are advancing, and it consists of road impedance, risk impedance, and material impedance.

[0022] The overall impedance of the route from the current node to the i-th node for

[0023] (1);

[0024] In the formula, The total impedance for the selected route from the current node to the i-th node; Let be the road impedance of the i-th node; Let the risk impedance be that of the i-th node; Let be the impedance for acquiring materials at the i-th node.

[0025] Road impedance This reflects the impact of road attributes such as road distance between nodes, road type, and damage condition on the alignment results. The expression for road impedance is:

[0026] (2);

[0027] In the formula, i represents the next node to be arrived; The impedance of the path to the i-th node; Let be the road type coefficient leading to the i-th node, where is the road type when it is a highway. =1.0; When the road type is urban road, =1.1; When the road type is rural road, =1.2; When the road type is a forest road, =1.3; for other road types, =1.4. This represents the road damage status leading to the i-th node. When the road is in good condition... =1.0; When road conditions are average, =1.1; When road conditions are poor, =1.3; This represents the distance from the current node to the i-th node; The risk coefficient measures the priority of considering road condition factors.

[0028] Risk impedance reflects the impact of potential severe weather and unforeseen adverse events on route selection. The expression for risk impedance is:

[0029] (3);

[0030] In the formula, i represents the next node to be arrived; The risk impedance to the i-th node; Let be the smog warning coefficient for the road segment leading to the i-th node. ; This is the gale warning coefficient for the road segment leading to the i-th node. When the wind force is below level 6, the impact on traffic is relatively small. When the wind force is 6 or above, ,in, Wind force forecast level; Let be the average predicted rainfall for the road segment leading to the i-th node; Let be the average predicted snowfall amount for the road segment leading to the i-th node; The distance of the sudden and severe event from node i; The risk coefficient measures the priority of considering factors that pose risks during the journey.

[0031] Material acquisition impedance reflects the current material acquisition status at the node and is a key indicator affecting the route selection result;

[0032] (4);

[0033] In the formula, This represents the total amount of resources obtained when the action reaches the i-th node; The total quantity of supplies needed to handle emergencies; Let be the quantity of materials at the i-th node; A value of 0 indicates that sufficient resources have been obtained, and the degree of resource acquisition no longer affects the overall impedance of the selected line. The cumulative time to reach the i-th node. The larger the value, the closer the emergency response mission is to the later stages, the more urgent the need for supplies, and the greater the resistance at nodes lacking supplies. This is the material coefficient, which measures the priority of consideration for the availability of materials for disposal.

[0034] The comprehensive impedance of each node is calculated according to equations (1)-(4), and a comprehensive impedance model of the road network is established.

[0035] Step 3: Optimize route selection:

[0036] Based on the integrated impedance model of the road network established in step 2, the route selection problem is modeled as a multi-stage decision problem, and the adaptive dynamic programming algorithm in reinforcement learning is used to optimize the route selection strategy. Adaptive dynamic programming introduces neural networks into the reinforcement learning dynamic programming algorithm, using the neural network as a function approximation structure to approximate the solution of the value function, which can solve multi-stage approximate optimal decision problems.

[0037] The adaptive dynamic framework comprises an evaluation network, an execution network, and a dynamic system. The evaluation and execution networks are constructed using neural networks, whose parameters are updated according to the Bellman optimality principle. The dynamic system is a dynamic route selection model composed of the road network integrated impedance model constructed in step 2. The actions selected by the action network affect the dynamic system, and the dynamic system then provides a reward / penalty, which is fed back to the evaluation network.

[0038] Step 3.1: Evaluate the network

[0039] The evaluation network uses a backpropagation (BP) neural network, consisting of an input layer, hidden layers, and an output layer. The sigmoid function is chosen as the activation function, and the network is evaluated by fitting the system's state variables. With action variables With cost function The nonlinear relationship between the parameters is used to evaluate the quality of the network's actions. When the network training converges, the fitted cost function converges to the true system value. In the k-th iteration, we have:

[0040] (5)

[0041] In the formula, The total cost accumulated by the system from the start of the route selection action in stage i until the end of the route selection is reached; The cost function for performing the line selection action in the i-th stage is set as the comprehensive impedance Z of line i in this method. i In other words, the optimization direction is to select lines with lower overall impedance. is the discount factor, representing the discount coefficient for the cost of the method occurring in the future stage.

[0042] The error between the true value and the fitted value is:

[0043] (6)

[0044] In the formula, The cost function is used to evaluate the network fit.

[0045] The evaluation network updates the parameters of the neural network by minimizing the error function:

[0046] (7)

[0047] Step 3.2: Execute the network

[0048] The execution network employs a backpropagation (BP) neural network, consisting of an input layer, hidden layers, and an output layer. The sigmoid function is chosen as the activation function. Based on the evaluation network, the action that minimizes the evaluation network's value function is fitted, thus outputting the optimal control method. The input to the execution network is the system state variable. The output is an action variable. In the (k+1)th iteration, the goal of the execution network is to minimize the system cost function:

[0049] (8)

[0050] The error function of an action network is defined as:

[0051] (9)

[0052] The weight updates in the execution network are similar to those in the evaluation network, both using the gradient descent method.

[0053] In the adaptive dynamic programming algorithm's route selection optimization process, each stage is based on the Bellman optimality principle. When the system state changes, the evaluation network and the execution network subsequently output a "better" control method. The system executes the control method, changes the system state, and then trains the evaluation network and the execution network based on the new state until the performance requirements are met. Figure 2 As shown.

[0054] The execution network, based on the evaluation network, fits the action that minimizes the value function of the evaluation network, thus outputting the optimal control method. The input to the execution network is the system state variables. The output is an action variable. In the (k+1)th iteration, the goal of the execution network is to minimize the system cost function, such as... Figure 2 As shown.

[0055] Step 4: Dynamic Route Selection: The reinforcement learning model is trained offline based on the constructed road network topology model and typical events. The trained model from Step 3 is used for route recommendation. Simultaneously, based on environmental changes, the comprehensive impedance of the route from the current node to the i-th node is calculated in real time, achieving dynamic adaptive route selection.

[0056] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A reinforcement learning-driven dynamic route selection method for border defense and emergency response, characterized in that, The method includes: Step 1: Data Acquisition and Processing: Acquire latitude and longitude data of the road network, road types, road damage status, weather forecast information, real-time weather information, location of sudden severe events, and historical data of sudden events. Calculate the road distances between traffic nodes and label the road attributes between nodes to construct the original road network data model; acquire real-time location data of sudden severe events and establish a database. Step 2: Establish a comprehensive road network impedance model: Calculate the comprehensive impedance of the route selection based on the impact of route travel time, possible severe weather, sudden severe events, and the availability of supplies; the comprehensive impedance of the route selection is based on the difficulty of the roads traversed by the troops during their march, including road impedance, risk impedance, and material impedance; calculate the comprehensive impedance of each node to establish a comprehensive road network impedance model. Step 3: Optimize the recommended route: Based on the road network integrated impedance model established in Step 2, the adaptive dynamic programming algorithm in reinforcement learning is used to optimize the route selection and construct an adaptive dynamic framework, including an evaluation network, an execution network, and a dynamic system. The evaluation network and the execution network are constructed through neural networks, and the parameters in the neural networks are updated according to the Bellman optimality principle. The dynamic system is a dynamic route selection model composed of the road network integrated impedance model constructed in Step 2. Step 4: Dynamic route selection: Based on the constructed road network topology model and typical events, the reinforcement learning model dynamically selects the action route online.

2. The reinforcement learning-driven dynamic route selection method for border defense and emergency response as described in claim 1, characterized in that, In step 1, road types include highways, urban roads, rural roads, and forest roads; Historical emergency data includes the type, location, scale of the event, and the types and quantities of supplies required for its handling. Weather forecasts and real-time weather information include rainfall, snowfall, fog and haze warnings, and gale warnings.

3. The reinforcement learning-driven dynamic route selection method for border defense and emergency response as described in claim 1, characterized in that, In step 2, the comprehensive impedance of the line selection from the current node to the i-th node is calculated. for: （1）； In the formula, The total impedance for the selected route from the current node to the i-th node; Let be the road impedance of the i-th node; Let the risk impedance be that of the i-th node; Let be the impedance for acquiring materials at the i-th node.

4. The reinforcement learning-driven dynamic route selection method for border defense and emergency response as described in claim 3, characterized in that, The expression for road impedance is: （2）； In the formula, i represents the next node to be arrived; The impedance of the path to the i-th node; Let be the road type coefficient leading to the i-th node, where is the road type when it is a highway. =1.0; When the road type is urban road, =1.1; When the road type is rural road, =1.2; When the road type is a forest road, =1.3; for other road types, =1.4; This represents the road damage status leading to the i-th node. When the road is in good condition... =1.0; When road conditions are average, =1.1; When road conditions are poor, =1.3; This represents the distance from the current node to the i-th node; The risk coefficient measures the priority of considering road condition factors.

5. The reinforcement learning-driven dynamic route selection method for border defense and emergency response according to claim 3, characterized in that, The expression for risk impedance is: （3）； In the formula, i represents the next node to be arrived; The risk impedance to the i-th node; Let be the smog warning coefficient for the road segment leading to the i-th node. ; This is the gale warning coefficient for the road segment leading to the i-th node. When the wind force is below level 6, the impact on traffic is relatively small. ; When the wind force is 6 or above ,in, Wind force forecast level; Let be the average predicted rainfall for the road segment leading to the i-th node; Let be the average predicted snowfall amount for the road segment leading to the i-th node; The distance of the sudden and severe event from node i; The risk coefficient measures the priority of considering factors that pose risks during the journey.

6. The reinforcement learning-driven dynamic route selection method for border defense and emergency response according to claim 3, characterized in that, The expression is: （4）； In the formula, This represents the total amount of resources obtained when the action reaches the i-th node; The total quantity of supplies needed to handle emergencies; Let be the quantity of materials at the i-th node; A value of 0 indicates that sufficient resources have been obtained, and the degree of resource acquisition no longer affects the overall impedance of the selected line. The cumulative time to reach the i-th node. The larger the value, the closer the emergency response mission is to the later stages, the more urgent the need for supplies, and the greater the resistance at nodes lacking supplies. This is the material coefficient, which measures the priority of consideration for the availability of materials for disposal.

7. The reinforcement learning-driven dynamic route selection method for border defense and emergency response according to claim 1, characterized in that, In step 3, the evaluation network fits the system's state variables. With action variables With cost function The nonlinear relationship between the network and the system is used to evaluate the network's actions. When the network training converges, the fitted cost function converges to the true value of the system. In the k-th iteration, we have: （5）； In the formula, The total cost accumulated by the system from the start of the route selection action in stage i until the end of the route selection is reached; The cost function for performing the line selection action in the i-th stage is set as the combined impedance Z of line i. i In other words, the optimization direction is to select lines with lower overall impedance. The discount factor represents the discount coefficient used to calculate the costs incurred by the method in future stages; The error between the true value and the fitted value is: （6）； In the formula, The cost function is used to evaluate the network fit.

8. The reinforcement learning-driven dynamic route selection method for border defense and emergency response according to claim 7, characterized in that, The evaluation network described above updates the parameters of the neural network by minimizing the error function. （7）。 9. The reinforcement learning-driven dynamic route selection method for border defense and emergency response according to claim 1, characterized in that, The execution network, based on the evaluation network, fits the action that minimizes the value function of the evaluation network, thus outputting the optimal control method; the input to the execution network is the system state variables. The output is an action variable. In the (k+1)th iteration, the goal of the execution network is to minimize the system cost function: （8）； The error function of an action network is defined as: （9）； The weight updates in the execution network are similar to those in the evaluation network, both using the gradient descent method.

10. The reinforcement learning-driven dynamic route selection method for border defense and emergency response according to claim 1, characterized in that, In step 4, during dynamic route selection, the reinforcement learning model is trained offline based on the constructed road network topology model and typical events. The model trained in step 3 is used to recommend routes. At the same time, the comprehensive impedance of the route from the current node to the i-th node is calculated in a timely manner according to environmental changes, so as to achieve dynamic adaptive route selection.

Images