Emergency rescue scheme generation method, system and device based on reinforcement learning

Abstract

The invention relates to the technical field of intelligent emergency rescue, in particular to an emergency rescue scheme generation method, system and device based on reinforcement learning, and the method comprises the steps: S1, obtaining all wounded data and disposable rescue vehicle data of a disaster site; s2, inputting the wounded data and the disposable rescue vehicle data as rescue states into a reinforcement learning model, constructing a reward function based on the survival probability of the wounded, and performing iterative training by taking maximization of the value of the reward function as a target to obtain a trained reinforcement learning model; and S3, based on the trained reinforcement learning model, inputting a current rescue state, and obtaining an action with a maximum reward value, namely an optimal wounded person rescue scheme. According to the method, the rescue efficiency can be improved, casualties can be reduced, resource allocation can be optimized, the workload of rescue workers can be relieved, and the method is of great significance in improving the overall level of emergency rescue.

Description

Technical Field

[0001] The present invention relates to the field of intelligent emergency rescue technology, and in particular to a method, system and device for generating an emergency rescue plan based on reinforcement learning. Background Art

[0002] With the frequent occurrence of natural disasters and social emergencies, how to conduct emergency rescue efficiently has become an important issue of global concern. Traditional emergency rescue methods often rely on manual experience and command and dispatch. When faced with complex and changing disaster scenarios, this method may have problems such as slow response speed and unreasonable resource allocation. Especially in the early stages of a disaster, how to quickly and effectively allocate limited rescue resources to maximize the saving of lives is an issue that needs to be solved.

[0003] In recent years, with the development of artificial intelligence technology, especially the advancement of reinforcement learning (RL) technology, people have begun to explore its application in the field of emergency rescue. Reinforcement learning is a method that enables intelligent agents to learn optimal behavior strategies through interaction with the environment. It can adapt to dynamically changing environments and does not require a large amount of historical data as a training prerequisite, which makes it very suitable for unstructured and highly uncertain scenarios such as emergency rescue.

[0004] In emergency rescue scenarios, reinforcement learning can automatically learn a set of optimal rescue plans by simulating various possible situations to improve rescue efficiency and effectiveness. For example, it can automatically identify which wounded people are most in need of emergency assistance and plan the most effective rescue path. However, there are still some challenges in practical applications, such as how to accurately assess the survival probability of the wounded, how to effectively transform complex real-world situations into state spaces and action spaces that reinforcement learning models can handle, and how to design reasonable reward functions to guide learning.

[0005] Therefore, it is particularly important to propose a method and technical solution for emergency rescue plan generation based on reinforcement learning. This method not only needs to consider how to obtain effective data from the disaster site, but also needs to design a reasonable model training process and rescue plan calculation mechanism in order to play a positive role in actual rescue operations. In addition, in order to ensure the smooth implementation of this technical solution, it is also necessary to develop corresponding system modules and storage media to support the operation of the entire system. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method for generating an emergency rescue plan based on reinforcement learning, comprising the following steps:

[0007] S1: Obtain data on all casualties and available rescue vehicles at the disaster site;

[0008] S2: inputting the injured person data and the available rescue vehicle data into the reinforcement learning model as rescue states, constructing a reward function based on the survival probability of the injured person, and performing iterative training with the goal of maximizing the value of the reward function to obtain a trained reinforcement learning model;

[0009] S3: Based on the trained reinforcement learning model, the current rescue state is input to obtain the action with the maximum reward value, that is, the best rescue plan for the wounded.

[0010] In one embodiment of the present invention, the injured person data includes the injured person's location data and injury time, and the available rescue vehicle data includes the number and location data of currently available rescue vehicles at the disaster site.

[0011] In one embodiment of the present invention, in S2, the method for obtaining the trained reinforcement learning model is as follows:

[0012] S21: Define the state space S, where s t ∈S represents the data of the injured and available rescue vehicles at the current time t, including the position of the injured (x i ,y i ), casualty number i, injury time t i , the number of vehicles num and the vehicle position (x v ,y v ), and the action space A, where a i ∈A represents the action of selecting a wounded person for rescue;

[0013] S22: Initialize the neural network parameters θ, the experience replay pool β, set the discount factor γ, the advantage estimation coefficient λ, and the batch size N for multi-step updates;

[0014] S23: The neural network is based on the current state s t Output action distribution π θ (·|s t ), sample action a in action space A t , perform action a in the environment t Generate instant reward r t+1 and the next decision point state s t+1 , the sequence (s t ,a t ,r t+1 ,s t+1 ) is stored in the experience replay pool β;

[0015] S24: Sample m sequences (s t ,a t ,r t+1 ,s t+1) form a set D t The advantage function A of each sequence is calculated using the generalized advantage estimate t ;

[0016] S25: Based on advantage function A t Construct the loss function L CLIP (θ), through the loss function L CLIP (θ) Update the parameter θ using the gradient ascent method: Where α is the learning rate;

[0017] S26: Determine whether the maximum number of iterations has been reached or the network has converged:

[0018] If not, return to step S23;

[0019] If so, output the current iterative training parameters, load the current iterative training parameters into the reinforcement learning model, and obtain the trained reinforcement learning model.

[0020] In one embodiment of the present invention, the advantage function A t The expression is:

[0021]

[0022] in, represents the estimated action a at time t The advantage function value of πθ(a t |s t ) represents the strategy of executing action a in state s at time t under parameter θ; It represents the strategy estimation of executing action a in state s at time t; represents the estimated value of the advantage function at time t;

[0023] In one embodiment of the present invention, the loss function L CLIP The expression of (θ) is as follows:

[0024]

[0025] in, Express expectations, min means taking the smaller value of the first and second terms; the clip function ensures that the ratio of the new and old strategies does not exceed the range of [1-ε,1+ε], which means that for the three terms in the brackets, if the first term is less than the second term, then 1-ε is output; if the first term is greater than the third term, then 1+ε is output.

[0026] In one embodiment of the present invention, in S2, the method for constructing a reward function according to the survival probability of the wounded is as follows:

[0027] According to the transportation time t1 of any injured person i to the hospital and the injury time t2 of any injured person i, the waiting time t2 of any injured person i is obtained. i ;

[0028] Based on the waiting time t i , we can get the survival probability P(t i );

[0029] Based on the survival probability P(t i ), construct the reward function.

[0030] In one embodiment of the present invention, the survival probability P(t i ) is calculated as:

[0031]

[0032] Among them, α and γ are hyperparameters. b0 and b1 are fitting parameters.

[0033] In one embodiment of the present invention, the reward function R(s,a) is expressed as:

[0034]

[0035] Among them, I represents the set of injured persons, s is the current rescue state, and a is the action corresponding to the current rescue state s.

[0036] The present invention also provides an emergency rescue plan generation system based on reinforcement learning, which is used to implement the steps of the above-mentioned emergency rescue plan generation method based on reinforcement learning, and includes the following modules:

[0037] Data collection module, used to obtain data on all injured persons and available rescue vehicles at the disaster site;

[0038] A model training module, used to input the injured person data and the available rescue vehicle data as rescue states into a reinforcement learning model, construct a reward function based on the injured person's survival probability, perform iterative training with the goal of maximizing the value of the reward function, and obtain a trained reinforcement learning model;

[0039] The rescue plan generation module is used to input the current rescue state based on the trained reinforcement learning model to obtain the action with the maximum reward value, that is, the best rescue plan for the wounded.

[0040] The present invention also provides an emergency rescue plan generation device based on reinforcement learning, which includes the above-mentioned emergency rescue plan generation system based on reinforcement learning, a real-time data interface module, a dynamic resource scheduling module and a human-computer interaction interface;

[0041] Among them, the real-time data interface module obtains the latest injured data and rescue vehicle data from the external data source acquisition device in real time; the dynamic resource scheduling module plans the optimal path according to the rescue plan generated by the system, and automatically dispatches the rescue vehicle to the best rescue location; the human-computer interaction interface includes an operable graphical interface for inputting instructions, checking the rescue status and receiving the best rescue plan.

[0042] The above technical solution of the present invention has the following advantages compared with the prior art:

[0043] The present invention realizes the intelligent generation of emergency rescue plans, can dynamically adjust the rescue strategy according to the real-time data of the injured and the rescue vehicle data, maximize the survival probability of the injured, thereby improving the rescue efficiency and resource utilization, and at the same time has high flexibility, real-time and accuracy, effectively reducing the uncertainty in human decision-making, and providing a scientific and efficient solution for emergency rescue. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0045] Figure 1 is a flow chart of a method for generating an emergency rescue plan based on reinforcement learning provided in the first embodiment of the present invention;

[0046] Figure 2 is a flow chart of a method for obtaining a trained reinforcement learning model provided in the first embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of the structure of an emergency rescue plan generation system based on reinforcement learning provided in the second embodiment of the present invention;

[0048] Description of the accompanying drawings in the specification: 100, data acquisition module; 200, model training module; 300, rescue plan generation module. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0050] Embodiment 1

[0051] Reference Figure 1 As shown, the present invention provides a method for generating an emergency rescue plan based on reinforcement learning, comprising the following steps:

[0052] S1: Acquire all the injured data and available rescue vehicle data at the disaster site; wherein the injured data include the longitude and latitude location data and the time of injury of the injured, and the available rescue vehicle data include the number and location data of the rescue vehicles currently available at the disaster site;

[0053] S2: inputting the injured person data and the available rescue vehicle data into the reinforcement learning model as rescue states, constructing a reward function based on the survival probability of the injured person, and performing iterative training with the goal of maximizing the value of the reward function to obtain a trained reinforcement learning model;

[0054] S3: Based on the trained reinforcement learning model, the current rescue state is input to obtain the action with the maximum reward value, that is, the best rescue plan for the wounded.

[0055] It can be seen from the above technical solutions that the present invention can dynamically adjust the rescue strategy based on the real-time acquired casualty data (such as location, injury time) and rescue vehicle data (such as number, location), while enhancing the ability to respond to emergencies, reasonably allocating rescue resources, avoiding waste of resources, and improving rescue efficiency.

[0056] like Figure 2 As shown, in S2, the injured person data and the available rescue vehicle data are input into the reinforcement learning model, i.e., the proximal policy optimization algorithm (PPO), as the rescue state, and a reward function is constructed based on the survival probability of the injured person. Iterative training is performed with the goal of maximizing the value of the reward function to obtain a trained reinforcement learning model. The specific steps of the method are as follows:

[0057] S21: Define the state space S, where s t ∈S represents the data of the injured and available rescue vehicles at the current time t, including the position of the injured (x i ,y i ), casualty number i, injury time t i , the number of vehicles num and the vehicle position (x v ,y v ), and the action space A, where a i ∈A represents the action of selecting a wounded person for rescue;

[0058] S22: Initialize the parameters θ and the experience replay pool β of the neural network, set the discount factor γ, the advantage estimation coefficient λ, and the batch size N of the multi-step update; specifically, the discount factor γ is used to balance the impact of immediate rewards and future rewards, and the advantage estimation coefficient λ is used to adjust the impact of the generalized advantage estimation (GAE) on long-term and short-term rewards;

[0059] S23: The neural network is based on the current state s t Output action distribution πθ (·|s t ), sample action a in action space A t , perform action a in the environment t Generate instant reward r t+1 and the next decision point state s t+1 , the sequence (s t ,a t ,r t+1 ,s t+1 ) is stored in the experience replay pool β;

[0060] S24: Sample m sequences (s t ,a t ,r t+1 ,s t+1 ) form a set D t Using generalized advantage estimation to estimate the advantage of each action helps the algorithm better evaluate the quality of the action, that is, calculating the advantage function A of each sequence t :

[0061]

[0062] in, represents the estimated action a at time t The advantage function value of θ (a t |s t ) represents the strategy of executing action a in state s at time t under parameter θ; It represents the strategy estimation of executing action a in state s at time t; represents the estimated value of the advantage function at time t;

[0063] S25: Based on advantage function A t Construct the loss function L CLIP (θ), through the loss function L CLIP (θ) Update the parameter θ using the gradient ascent method: Where α is the learning rate;

[0064] S26: Determine whether the maximum number of iterations has been reached or the network has converged:

[0065] If not, return to step S23;

[0066] If so, output the current iterative training parameters, load the current iterative training parameters into the reinforcement learning model, and obtain the trained reinforcement learning model.

[0067] Furthermore, in step S25, the loss function L CLIP The expression of (θ) is as follows:

[0068]

[0069] in, Express expectations, min means taking the smaller value of the first and second terms; the clip function ensures that the ratio of the new and old strategies does not exceed the range of [1-ε,1+ε], which means that for the three terms in the brackets, if the first term is less than the second term, then 1-ε is output; if the first term is greater than the third term, then 1+ε is output.

[0070] Furthermore, in S2, the method of constructing the reward function according to the survival probability of the wounded is as follows:

[0071] According to the transportation time t1 of any injured person i to the hospital and the injury time t2 of any injured person i, the waiting time t2 of any injured person i is obtained. i ;

[0072] Based on the waiting time t i After data fitting, the survival probability P(t i ):

[0073]

[0074] Among them, α and γ are hyperparameters. b0=4.368 and b1=0.713 are both fitting parameters;

[0075] Based on the survival probability P(t i ), construct the reward function R(s,a):

[0076]

[0077] Where I represents the set of injured persons, s is the current rescue state, and a is the action corresponding to the current rescue state s. The instant reward r in step S23 is calculated according to the above formula t+1 .

[0078] Embodiment 2

[0079] The present invention also provides a system for generating an emergency rescue plan based on reinforcement learning, which is used to implement the steps of the method for generating an emergency rescue plan based on reinforcement learning provided in the first embodiment. Figure 3 As shown, the system includes the following modules:

[0080] The data collection module 100 is used to obtain data of all injured persons and available rescue vehicles at the disaster site;

[0081] A model training module 200, for inputting the injured person data and the available rescue vehicle data as rescue states into a reinforcement learning model, constructing a reward function based on the injured person's survival probability, and performing iterative training with the goal of maximizing the value of the reward function to obtain a trained reinforcement learning model;

[0082] The rescue plan generation module 300 is used to input the current rescue state based on the trained reinforcement learning model to obtain the action with the maximum reward value, that is, the best rescue plan for the injured.

[0083] The present embodiment proposes a reinforcement learning-based emergency rescue plan generation system, which is used to implement the aforementioned reinforcement learning-based emergency rescue plan generation method. Therefore, the specific implementation method of the reinforcement learning-based emergency rescue plan generation system can be seen in the aforementioned reinforcement learning-based emergency rescue plan generation method embodiment part. For example, the data acquisition module 100, the model training module 200 and the rescue plan generation module 300 are respectively used to implement the steps S1, S2 and S3 in the reinforcement learning-based emergency rescue plan generation method in embodiment one. Therefore, its specific implementation method can refer to the description of the corresponding parts of the embodiments. In order to avoid redundancy, it will not be repeated here.

[0084] Embodiment 3

[0085] The present invention also provides an emergency rescue plan generation device based on reinforcement learning, which includes the emergency rescue plan generation system based on reinforcement learning described in Example 2, a real-time data interface module, a dynamic resource scheduling module and a human-computer interaction interface.

[0086] The real-time data interface module integrates multiple data interface protocols (such as HTTP, MQTT, etc.) to obtain the latest casualty data and rescue vehicle data in real time from external data source acquisition devices (such as drones, satellite images, field sensors, etc.). These data will be updated to the system in real time to ensure that the model can make decisions based on the latest information.

[0087] The dynamic resource scheduling module integrates a path planning algorithm (such as A*, Dijkstra or a path planning method based on deep learning), which can plan the optimal path for the rescue vehicle according to the current rescue status and the rescue plan obtained by the trained reinforcement learning model, and automatically schedule the rescue vehicle to the best rescue location.

[0088] The human-computer interaction interface integrates a highly specialized graphical user interface (GUI), which mainly includes functional modules such as map display, status monitoring, and solution display, providing rescuers with an intuitive operation interface for inputting commands, viewing rescue status, and receiving the best rescue solution. At the same time, the interface can also display real-time data and charts during the rescue process to help rescuers better understand the rescue situation.

[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0093] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A method for generating an emergency rescue plan based on reinforcement learning, characterized in that: The following steps are involved: S1: Obtain data on all casualties and available rescue vehicles at the disaster site; S2: inputting the injured person data and the available rescue vehicle data into the reinforcement learning model as rescue states, constructing a reward function based on the survival probability of the injured person, and performing iterative training with the goal of maximizing the value of the reward function to obtain a trained reinforcement learning model; S3: Based on the trained reinforcement learning model, the current rescue state is input to obtain the action with the maximum reward value, that is, the best rescue plan for the wounded.

2. The method for generating an emergency rescue plan based on reinforcement learning according to claim 1, characterized in that: The injured person data includes the injured person's location data and injury time, and the available rescue vehicle data includes the number and location data of the rescue vehicles currently available at the disaster site.

3. The method for generating an emergency rescue plan based on reinforcement learning according to claim 2, characterized in that: In S2, the method for obtaining the trained reinforcement learning model is as follows: S21: Define the state space S, where s t ∈S represents the data of the injured and available rescue vehicles at the current time t, including the position of the injured (x i ,y i ), casualty number i, injury time t i , the number of vehicles num and the vehicle position (x v ,y v ), and the action space A, where a i ∈A represents the action of selecting a wounded person for rescue; S22: Initialize the neural network parameters θ, the experience replay pool β, set the discount factor γ, the advantage estimation coefficient λ, and the batch size N for multi-step updates; S23: The neural network is based on the current state s t Output action distribution π θ (·|s t ), sample action a in action space A t , perform action a in the environment t Generate instant reward r t+1 and the next decision point state s t+1 , the sequence (s t ,a t ,r t+1 ,s t+1 ) is stored in the experience replay pool β; S24: Sample m sequences (s t ,a t ,r t+1 ,s t+1 ) form a set D t The advantage function A of each sequence is calculated using the generalized advantage estimate t ; S25: Based on advantage function A t Construct the loss function L CLIP (θ), through the loss function L CLIP (θ) Update the parameter θ using the gradient ascent method: Where α is the learning rate; S26: Determine whether the maximum number of iterations has been reached or the network has converged: If not, return to step S23; If so, output the current iterative training parameters, load the current iterative training parameters into the reinforcement learning model, and obtain the trained reinforcement learning model.

4. The method for generating an emergency rescue plan based on reinforcement learning according to claim 3, characterized in that: The advantage function A t The expression is: in, represents the estimated action a at time t The advantage function value of θ (a t |s t ) represents the strategy of executing action a in state s at time t under parameter θ; It represents the strategy estimation of executing action a in state s at time t; represents the estimated value of the advantage function at time t.

5. The method for generating an emergency rescue plan based on reinforcement learning according to claim 4, characterized in that: The loss function L CLIP The expression of (θ) is as follows: in, Express expectations, min means taking the smaller value of the first and second terms; the clip function ensures that the ratio of the new and old strategies does not exceed the range of [1-ε,1+ε], which means that for the three terms in the brackets, if the first term is less than the second term, then 1-ε is output; if the first term is greater than the third term, then 1+ε is output.

6. The method for generating an emergency rescue plan based on reinforcement learning according to claim 1, characterized in that: In S2, the method of constructing the reward function based on the survival probability of the wounded is as follows: According to the transportation time t1 of any injured person i to the hospital and the injury time t2 of any injured person i, the waiting time t2 of any injured person i is obtained. i ; Based on the waiting time t i , we can get the survival probability P(t i ); Based on the survival probability P(t i ), construct the reward function.

7. The method for generating an emergency rescue plan based on reinforcement learning according to claim 6, characterized in that: The survival probability P(t i ) is calculated as: Among them, α and γ are hyperparameters. b0 and b1 are fitting parameters.

8. The method for generating an emergency rescue plan based on reinforcement learning according to claim 7, characterized in that: The expression of the reward function R(s,a) is: Among them, I represents the set of injured persons, s is the current rescue state, and a is the action corresponding to the current rescue state s.

9. An emergency rescue plan generation system based on reinforcement learning, characterized in that: The steps for implementing the method for generating an emergency rescue plan based on reinforcement learning as described in any one of claims 1 to 8 include the following modules: Data collection module, used to obtain data on all injured persons and available rescue vehicles at the disaster site; A model training module, used to input the injured person data and the available rescue vehicle data as rescue states into a reinforcement learning model, construct a reward function based on the injured person's survival probability, perform iterative training with the goal of maximizing the value of the reward function, and obtain a trained reinforcement learning model; The rescue plan generation module is used to input the current rescue state based on the trained reinforcement learning model to obtain the action with the maximum reward value, that is, the best rescue plan for the wounded.

10. An emergency rescue plan generation device based on reinforcement learning, characterized in that: It includes the emergency rescue plan generation system based on reinforcement learning as claimed in claim 9, a real-time data interface module, a dynamic resource scheduling module and a human-computer interaction interface; Among them, the real-time data interface module obtains the latest injured data and rescue vehicle data from the external data source acquisition device in real time; the dynamic resource scheduling module plans the optimal path according to the rescue plan generated by the system, and automatically dispatches the rescue vehicle to the best rescue location; the human-computer interaction interface includes an operable graphical interface for inputting instructions, checking the rescue status and receiving the best rescue plan.

Images