Emergency decision generation method, device and equipment based on deep reinforcement learning

Abstract

The application provides an emergency decision generation method and device based on deep reinforcement learning and an equipment, to solve the technical problem that the generalization of an emergency decision model based on deep reinforcement learning is poor and is difficult to reuse in the related art, the method comprising: analyzing a scene requiring emergency decision reinforcement learning to obtain scene factors of the scene; determining a plurality of behavior instructions for making the emergency decision in the scene based on the scene factors; determining state information for representing a state of a sudden event of the scene based on the scene factors; generating an auxiliary reward function according to the plurality of behavior instructions and the state information, and training an emergency decision generation model through the auxiliary reward function; and generating an emergency decision by using the trained emergency decision generation model.

Description

Technical Field

[0001] This invention relates to the field of deep reinforcement learning, and in particular to an emergency decision generation method, apparatus, and device based on deep reinforcement learning. Background Technology

[0002] How to make emergency decisions after a sudden incident is a core issue in the emergency rescue process. Research on methods for generating emergency decision-making plans is a key focus in the field of emergency decision-making.

[0003] While deep reinforcement learning technology shows great promise for emergency response decision generation, it has not yet been widely adopted due to poor generalization ability of trained models and difficulty in reusing them in different environments. This poor generalization stems from the lack of a standardized reward function during reinforcement learning training for emergency response decision generation. Inconsistencies in reward function formulation among different personnel result in incompatible reinforcement learning decision models trained with different reward functions, significantly hindering the standardization and application of deep reinforcement learning technology in emergency response decision generation. Summary of the Invention

[0004] This invention provides an emergency decision generation method, apparatus, and device based on deep reinforcement learning, which solves the technical problem that emergency decision-making models based on deep reinforcement learning have poor generalization and are difficult to reuse in related technologies.

[0005] In a first aspect, embodiments of the present invention provide an emergency decision generation method based on deep reinforcement learning, the method comprising:

[0006] Analyze scenarios that require reinforcement learning for emergency decision-making to identify scenario factors;

[0007] Based on scenario factors, determine multiple behavioral instructions for emergency decision-making in the scenario;

[0008] Determine the state information used to characterize the state of sudden events in the scenario based on scenario factors;

[0009] An auxiliary reward function is generated based on multiple behavioral instructions and status information, and an emergency decision generation model is trained using the auxiliary reward function.

[0010] Emergency decisions are generated using a pre-trained emergency decision generation model.

[0011] In one possible implementation, the method provided in this embodiment of the invention analyzes the scenario requiring reinforcement learning for emergency decision-making to obtain scenario factors, including:

[0012] Analyze the scene to identify its scene factors;

[0013] Based on scenario factors, we obtain the following: emergency event factors that characterize the damage caused to the scenario, disaster-bearing carrier factors that characterize the objects affected by the emergency, and emergency management factors that characterize the emergency measures to respond to the emergency.

[0014] In one possible implementation, the method provided by this embodiment of the invention determines state information characterizing the state of a sudden event in the scene based on scene factors, including:

[0015] Determine the status information of an emergency based on the factors of the emergency;

[0016] Based on disaster-bearing carrier factors, determine the disaster-bearing carrier status information that characterizes the carrier status;

[0017] Emergency management status information, which characterizes the status of emergency measures, is determined based on emergency management factors.

[0018] In one possible implementation, the method provided in this embodiment of the invention generates an auxiliary reward function based on multiple behavioral instructions and state information, including:

[0019] Based on multiple action instructions, disaster-bearing carrier status information, and emergency management status information, determine the first loss value caused by multiple action instructions when responding to emergencies;

[0020] Determine the second loss value caused by failure to respond to emergencies based on the status information of the disaster-bearing carrier;

[0021] The auxiliary reward function is determined based on the first loss value and the second loss value.

[0022] In one possible implementation, the method provided by this embodiment of the invention determines the initial reward function of the auxiliary reward function using the following formula:

[0023]

[0024] Where l represents the number of damaged disaster-bearing structures, m represents the number of damaged rescue forces, n represents the number of rescue forces deployed, and loss Bi For damage to the disaster-bearing carrier, loss Ej Due to the damage to emergency response capabilities, R each_step This is the initial reward function.

[0025] In one possible implementation, the method provided by the embodiments of the present invention determines the first loss value and the second loss value using the following formula:

[0026]

[0027]

[0028] The final reward function of the auxiliary reward function is as follows:

[0029]

[0030] Where m is the number of damaged disaster-bearing structures in the entire scenario after the rescue, and n is the number of disaster-bearing structures that may ultimately be damaged in the scenario. This refers to the price of the j-th damaged disaster-bearing structure. This refers to the price of the i-th damaged disaster-bearing carrier. The degree of damage to the j-th damaged disaster-bearing structure, loss action The first loss value is called loss. max R is the second loss value. final_step This is the final reward function.

[0031] In one possible implementation, the method provided by this embodiment of the invention determines the auxiliary reward function using the following formula:

[0032]

[0033] Where l represents the number of damaged disaster-bearing structures, m represents the number of damaged rescue forces, n represents the number of rescue forces deployed, and loss Bi For damage to the disaster-bearing carrier, loss Ej Due to the loss of emergency response capabilities, action The first loss value is called loss. max R is the second loss value. step This is an auxiliary reward function.

[0034] Secondly, embodiments of the present invention provide an emergency decision generation device based on deep reinforcement learning, characterized in that it includes:

[0035] The analysis unit is used to analyze scenarios that require reinforcement learning for emergency decision-making and to obtain the scenario factors.

[0036] The first determining unit is used to determine multiple behavioral instructions for emergency decision-making in the scenario based on scenario factors;

[0037] The second determining unit is used to determine the state information that characterizes the state of a sudden event in the scene based on scene factors;

[0038] The processing unit is used to generate an auxiliary reward function based on multiple behavioral instructions and status information, and to train an emergency decision generation model using the auxiliary reward function.

[0039] The generation unit is used to generate emergency decisions using a trained emergency decision generation model.

[0040] In one possible implementation, the analysis unit in the apparatus provided by the embodiments of the present invention is specifically used for:

[0041] Analyze the scene to identify its scene factors;

[0042] Based on scenario factors, we obtain the following: emergency event factors that characterize the damage caused to the scenario, disaster-bearing carrier factors that characterize the objects affected by the emergency, and emergency management factors that characterize the emergency measures to respond to the emergency.

[0043] In one possible implementation, the second determining unit in the apparatus provided by the embodiments of the present invention is specifically used for:

[0044] Determine the status information of an emergency based on the factors of the emergency;

[0045] Based on disaster-bearing carrier factors, determine the disaster-bearing carrier status information that characterizes the carrier status;

[0046] Emergency management status information, which characterizes the status of emergency measures, is determined based on emergency management factors.

[0047] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention is specifically used for:

[0048] Based on multiple action instructions, disaster-bearing carrier status information, and emergency management status information, determine the first loss value caused by multiple action instructions when responding to emergencies;

[0049] Determine the second loss value caused by failure to respond to emergencies based on the status information of the disaster-bearing carrier;

[0050] The auxiliary reward function is determined based on the first loss value and the second loss value.

[0051] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention specifically determines the initial reward function using the following formula:

[0052]

[0053] Where l represents the number of damaged disaster-bearing structures, m represents the number of damaged rescue forces, n represents the number of rescue forces deployed, and loss Bi For damage to the disaster-bearing carrier, loss Ej Due to the damage to emergency response capabilities, R each_step This is the initial reward function.

[0054] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention specifically determines the first loss value and the second loss value using the following formula:

[0055]

[0056]

[0057] The final reward function of the auxiliary reward function is as follows:

[0058]

[0059] Where m is the number of damaged disaster-bearing structures in the entire scenario after the rescue, and n is the number of disaster-bearing structures that may ultimately be damaged in the scenario. This refers to the price of the j-th damaged disaster-bearing structure. This refers to the price of the i-th damaged disaster-bearing carrier. The degree of damage to the j-th damaged disaster-bearing structure, loss action The first loss value is called loss. max R is the second loss value. final_step This is the final reward function.

[0060] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention specifically determines the auxiliary reward function using the following formula:

[0061]

[0062] Where l represents the number of damaged disaster-bearing structures, m represents the number of damaged rescue forces, n represents the number of rescue forces deployed, and loss B For damage to the disaster-bearing carrier, loss E Due to the loss of emergency response capabilities, action The first loss value is called loss. max R is the second loss value. step This is an auxiliary reward function.

[0063] Thirdly, embodiments of the present invention provide an electronic device, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method provided in the first aspect of the present invention.

[0064] Fourthly, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method provided in the first aspect of the present invention.

[0065] In this embodiment of the invention, the scenario requiring reinforcement learning for emergency decision-making is first analyzed to obtain scenario factors. Then, based on these scenario factors, multiple behavioral instructions for emergency decision-making are determined. Next, state information characterizing the state of a sudden event in the scenario is determined based on these scenario factors. Then, an auxiliary reward function is generated based on the multiple behavioral instructions and state information, and the emergency decision-making generation model is trained using this auxiliary reward function. Finally, the trained emergency decision-making generation model is used to generate emergency decisions. Compared with related technologies, this invention solves the problems of poor generalization and difficulty in reusing emergency decision-making models based on deep reinforcement learning. By analyzing the scenario, a reward function is constructed for the emergency decision-making model, and this reward function is used to train the model, solving the problem of the lack of a unified standard for reward function construction and improving the generalization of the emergency decision-making model. Attached Figure Description

[0066] Figure 1 A flowchart illustrating an emergency decision generation method based on deep reinforcement learning, provided in an embodiment of the present invention;

[0067] Figure 2 A schematic diagram illustrating the specific process of an emergency decision generation method based on deep reinforcement learning, provided in an embodiment of the present invention;

[0068] Figure 3 A schematic diagram of an emergency decision generation device based on deep reinforcement learning provided in an embodiment of the present invention;

[0069] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0071] The following are explanations of some of the words that appear in the text:

[0072] 1. In the embodiments of this invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0073] How to make emergency decisions after a sudden incident is a core issue in the emergency rescue process. Research on methods for generating emergency decision-making plans is a key focus in the field of emergency decision-making.

[0074] While deep reinforcement learning technology shows great promise for emergency response decision generation, it has not yet been widely adopted due to poor generalization ability of trained models and difficulty in reusing them in different environments. This poor generalization stems from the lack of a standardized reward function during reinforcement learning training for emergency response decision generation. Inconsistencies in reward function formulation among different personnel result in incompatible reinforcement learning decision models trained with different reward functions, significantly hindering the standardization and application of deep reinforcement learning technology in emergency response decision generation.

[0075] This technical solution provides an emergency decision generation method based on deep reinforcement learning. The following description, in conjunction with the accompanying drawings and embodiments, provides a more detailed explanation of the emergency decision generation method, apparatus, and equipment based on deep reinforcement learning provided by this invention.

[0076] This invention provides an emergency decision generation method based on deep reinforcement learning, such as... Figure 1 As shown, it includes:

[0077] Step S101: Analyze the scenarios that require reinforcement learning for emergency decision-making to obtain the scenario factors.

[0078] In practice, the scenario is analyzed to obtain the scenario factors. Then, based on the scenario factors, emergency event factors that cause damage to the scenario, disaster-bearing carrier factors that represent the objects affected by the emergency, and emergency management factors that represent the emergency measures to respond to the emergency are obtained.

[0079] The term "emergency factors" refers to events that may cause catastrophic damage to people, property, or social systems.

[0080] The so-called disaster-bearing factors refer to the objects affected by emergencies, which generally include three aspects: people, things, and systems (the socio-economic operating system composed of people, things, and their functions).

[0081] Emergency management factors, in the context of emergency rescue, refer to the process of mitigating the intensity of sudden events and reducing damage to affected infrastructure by implementing appropriate, timely, and reasonably sized emergency measures. It's important to note that emergency rescue operations consume relevant emergency resources; that is, the emergency rescue process itself incurs costs. Furthermore, rescue capabilities are always limited during actual rescue operations.

[0082] Step S102: Determine multiple behavioral instructions for emergency decision-making in the scenario based on scenario factors.

[0083] In practice, multiple behavioral instructions for emergency decision-making in a scenario are determined based on scenario factors. These multiple behavioral instructions, or behavioral space, refer to the set of all behavioral instructions that the decision-making agent can issue. In deep reinforcement learning, the behavioral space of an agent is mainly the complete set of behaviors related to the dispatch of available rescue forces.

[0084] Step S103: Determine the state information used to characterize the state of sudden events in the scene based on scene factors.

[0085] In practical implementation, the state in deep reinforcement learning refers to the information that the agent can obtain from the environment during its interaction with the environment, while the design of the state space defines the range of information that the agent can obtain. In this step, we determine the emergency event state information representing the state of the emergency event based on the factors of the emergency event, determine the disaster-bearing carrier state information representing the carrier state based on the factors of the disaster-bearing carrier, and determine the emergency management state information representing the state of emergency measures based on the factors of emergency management.

[0086] Step S104: Generate an auxiliary reward function based on multiple behavioral instructions and status information, and train the emergency decision generation model using the auxiliary reward function.

[0087] In practice, the first loss value caused by multiple behavioral instructions in response to emergencies is determined based on multiple behavioral instructions, disaster carrier status information and emergency management status information. Then, the second loss value caused by not responding to emergencies is determined based on the disaster carrier status information. Finally, the auxiliary reward function is determined based on the first loss value and the second loss value.

[0088] Step S105: Generate emergency decisions using the trained emergency decision generation model.

[0089] like Figure 2 As shown, taking a forest fire spread and rescue scenario as an example, the emergency decision generation method based on deep reinforcement learning provided in this embodiment of the invention will be specifically explained:

[0090] Step S201: Analyze the scene to obtain the scene factors.

[0091] In practice, the scenario to be simulated is first analyzed to identify its factors. Since emergencies occur in the real world, and the number and types of real objects that may be affected are numerous and diverse, only a limited number of factors can be considered in real emergency response decision-making. Therefore, when conducting reinforcement learning for emergency decision-making, the scenario must first be analyzed to identify the factors that need to be considered.

[0092] Step S202: Based on scenario factors, obtain the factors of the sudden event, the disaster-bearing carrier, and the emergency management factors.

[0093] In practical implementation, this invention proposes to use the public safety "triangle" framework as theoretical support to analyze the elements of forest fire spread and emergency rescue decision-making scenarios. The so-called public safety "triangle" framework refers to the analysis and sorting of elements of the entire scenario through three dimensions: "emergency," "disaster-bearing carrier," and "emergency management" in the handling of emergencies.

[0094] An emergency (H) is an event that may cause catastrophic damage to people, property, or social systems. Examples include primary events like the spread of forest fires, secondary events such as the death of animals and plants in the forest, damage to power substations leading to power grid failures, and coupled events such as forest fires causing explosions at nearby gas stations, which in turn promote the spread of the forest fire. The occurrence and development of emergencies are the starting point and cause of the entire emergency response process. However, the occurrence and development of forest fires can cause many events, but not all of these need to be considered in the reinforcement learning training for emergency decision generation. For example, forest fires can disrupt the short-term ecological balance in forests, but these events are not considered in the emergency response process. This involves defining the scope of disaster-bearing carriers.

[0095] The term "disaster-bearing carrier" (B) refers to the target of a sudden event, generally including three aspects: people, objects, and systems (the socio-economic operating system composed of people, objects, and their functions). It can be argued that the reason we conduct emergency rescue for sudden events is because these events can cause some kind of harm to the people, objects, and systems we consider disaster-bearing carriers. If a sudden event occurs in an uninhabited natural environment, we would not conduct emergency rescue. Furthermore, in general, during forest fire emergency rescue, the primary disaster-bearing carriers we consider include densely populated areas such as cities, villages, and chemical industrial parks that may be burned by the fire, as well as key facilities that can affect human production and daily life. Only then do we consider factors such as the economic value of trees, but we generally do not give much consideration to the flora, fauna, and ecosystems within the forest.

[0096] Emergency management (E), in the context of emergency response, refers to the process of mitigating the intensity of sudden events and reducing damage to affected infrastructure by implementing appropriate, timely, and reasonably sized emergency measures. It's important to note that emergency response to sudden events consumes relevant emergency resources; that is, the process itself incurs costs. Furthermore, rescue forces are always limited. Specifically, in the context of forest fire spread and rescue, common rescue forces include forest fire brigades, fire trucks, and firefighting aircraft, utilizing tools such as fire hydrants, ventilation fans, water, and chemical reagents. These are all resources and methods that can be used in emergency response.

[0097] Step S203: Determine multiple behavioral instructions for emergency decision-making in the scenario based on scenario factors.

[0098] In practical implementation, the behavior space and state space are crucial for the realization of deep reinforcement learning. The behavior space refers to the set of all behavioral instructions that the decision-making agent can issue. For emergency decision-making deep reinforcement learning applications, the design of the behavior space and state space requires a certain understanding of the business requirements; only on this basis can a reasonable behavior space and state space be designed.

[0099] In forest fire spread and rescue scenarios, the behavioral space of an agent in deep reinforcement learning primarily consists of the entire set of actions related to the deployment of available rescue forces. For example, considering an emergency rescue decision-making scenario involving a forest fire near Beijing, the available rescue forces include four forest fire rescue teams, three forest fire trucks, and two forest fire helicopters. Each rescue force can perform different operations: forest fire rescue teams can move, use fire extinguishing tools (number 2), use ventilation fans, retreat, and standby; forest fire trucks can move, extinguish fire, replenish water, and standby; and forest fire helicopters can take off, land, move, extinguish fire, collect water, and refuel. It's important to note that different rescue forces may perform the same operations—for example, each force can extinguish fires—but the changes they bring to the environment (primarily the changes in the fire intensity at the point of impact) are different. These different rescue capabilities are determined by an evolutionary model within the environment.

[0100] The above instructions can be expressed as a function as follows:

[0101] a fire_team =(i,(t) xi ,t yi )|i

[0102] =1. Move; 2. Use tool #2 to extinguish the fire; 3. Use the fan to extinguish the fire; 4. Retreat; 5. Standby)

[0103] afire_truck =(j,(t) x ,t y (j = 1, Move; 2, Extinguish; 3, Replenish Water; 4, Standby)

[0104] a fire_heli =(k,(t) x ,t y (k = 1, takeoff; 2, landing; 3, movement; 4, firefighting; 5, water collection; 6, refueling)

[0105] above formula a fire_team a fire_truck a fire_heli These refer to the behavioral expressions of a single fire brigade, fire truck, and fire helicopter, respectively. i, j, and k represent the behavioral numbers selected by the three rescue forces, respectively. xi ,t yi The coordinates (latitude and longitude) represent the target point of the rescue force's actions. In other scenarios, the behavior function can be adjusted based on the actual situation of the rescue force.

[0106] During deep reinforcement learning training, the behavioral instructions sent to each emergency rescue unit can be combined to obtain the behavioral function that the agent sends to the environment at any given time. In this example, the behavioral functions for the 4 rescue teams, 3 fire trucks, and 2 firefighting aircraft are shown below:

[0107]

[0108] The range of values ​​for the above behavioral function is:

[0109]

[0110] The two equations above together constitute the behavioral space of this forest fire spread and rescue scenario. It is important to note that when determining the range of values ​​for the behavioral function, it is necessary to restrict the behavior of certain elements based on the actual situation. For example, the target points for the movement of fire brigades and fire trucks cannot be located on water, and the target points for the movement of fire trucks cannot be located outside the road network, etc.

[0111] Step S204: Determine the state information used to characterize the state of sudden events in the scene based on scene factors.

[0112] In practical implementation, the state in deep reinforcement learning represents the information that the agent can obtain from the environment during its interaction with it. The design of the state space defines the range of information the agent can acquire. For the agent, the state space is closely related to the behavior space and the reward function. After selecting an action from the behavior space, the agent needs to know the impact of that action on the environment and the reward value resulting from those impacts in order to evaluate the quality of the action selection and optimize the strategy through learning.

[0113] Based on the analysis using the public safety "triangle" framework from the previous step, the elements in the scenario were organized according to three dimensions: emergencies, disaster-bearing carriers, and emergency management. The state space design was also based on these three dimensions, and the overall state space can be expressed as follows:

[0114] s=(s H ,s B ,s E )

[0115] Where s represents the overall set of states, s H ,s B ,s E These represent the sets of states of emergencies, disaster-bearing entities, and emergency management-related elements, respectively.

[0116] Taking the scenario of forest fire spread and rescue as an example, this paper explains the three-dimensional state set.

[0117] (1) Emergency state space s H

[0118] Emergencies are a key objective in rescue and relief efforts; therefore, crucial attributes of relevant emergency elements need to be transmitted as state information to the agent. In a forest fire scenario, relevant emergency state information includes, but is not limited to, the extent of forest fire spread, the historical process of forest fire spread, the distribution of forest fire intensity, and meteorological information such as wind speed and direction.

[0119] (2) Disaster-bearing carrier state space s B

[0120] Disaster-bearing vehicles are key targets for protection in rescue operations; therefore, some key attributes of disaster-bearing vehicle-related elements need to be transmitted as state information to the agent. In a forest fire scenario, relevant disaster-bearing vehicle state information includes, but is not limited to, the location, shortest distance from the fire line, and current status of towns, villages, critical equipment and facilities, and key protection targets around the forest fire.

[0121] (3) Emergency management state space s E

[0122] Emergency management is the main process by which intelligent agents make decisions and implement rescue actions. Therefore, the key attributes of all elements in emergency management need to be transmitted to the intelligent agent as a state. In a forest fire scenario, the relevant state information of emergency management elements includes, but is not limited to, the type and location of rescue forces, current instructions, fatigue value, health value, capacity, remaining resources, etc.

[0123] Step S205: Generate an auxiliary reward function based on multiple behavioral instructions and status information, and train the emergency decision generation model using the auxiliary reward function.

[0124] In practical implementation, for deep reinforcement learning, the reward function is key to guiding the agent's policy learning. For emergency rescue work, success is marked by the smooth resolution of the emergency and the successful completion of related rescue efforts. If these goals are achieved through decision-making, a good result can be considered achieved, resulting in a large overall reward upon completion.

[0125] However, a single final reward is insufficient for an agent to successfully learn decision-making abilities. Therefore, process rewards need to be set at each step to guide the agent towards the final goal. Since the final reward is the primary task, while the process rewards play a supporting role, they are respectively referred to as the final reward and the auxiliary reward. The design of the reward function is crucial for deep reinforcement learning aimed at emergency decision-making; the design methods for the two reward functions will be explained separately.

[0126] First, the auxiliary reward function is designed. This involves determining the types of elements involved in the auxiliary reward, then determining the positive or negative sign of the reward for each element type, and finally determining the magnitude of the reward value for each element type.

[0127] (1) Determine the scope of auxiliary rewards

[0128] The analysis considers three dimensions: emergencies, disaster-bearing carriers, and emergency management. In forest fire spread and rescue scenarios, forest fires are beyond human control; therefore, emergencies should not be considered in the supplementary reward process. Disaster-bearing carriers are objects that emergency rescue work needs to protect; if they are damaged during the rescue process, this should be considered in the reward system. The number of emergency management elements is limited, and unreasonable deployment can lead to casualties among rescue forces; therefore, they need to be considered in the supplementary reward system.

[0129] (2) Determine the size of each supplementary reward

[0130] For auxiliary rewards, the rewards are generally negative, meaning they are usually penalties. This is mainly considered from two perspectives: Firstly, if the auxiliary reward were positive, the agent might learn to hesitate and struggle for rewards on the way to the final goal, leading to training failure; secondly, in emergency rescue scenarios, damage to the disaster-bearing vehicle... B Loss of emergency response capabilities E Both are losses, and the deployment of emergency forces is a consumption of resources. B From a business logic perspective, these should all be negative rewards and should be avoided as much as possible. Therefore, the auxiliary reward function R... each_step for:

[0131]

[0132] Where l, m, and n represent the number of damaged disaster-bearing structures, the number of damaged rescue forces, and the number of rescue forces deployed, respectively. Since losses and losses are calculated as positive numbers, a negative sign is added before summation. The magnitude of these auxiliary reward values ​​needs to be considered in conjunction with real-world factor prices and costs.

[0133] The final reward refers to the reward obtained at the end of a single training process. Generally, the final reward is obtained under two conditions: one is that the rescue mission is completed, meeting the end condition of a training session, at which point the computation contacts and the final reward is obtained; the other is that a pre-set number of training steps is reached, triggering contact and obtaining the final reward. In summary, the design of this final reward should be related to whether the mission is completed or not, and the distance to the mission objective if not completed. Therefore, the final reward function is obtained in two steps.

[0134] (1) Disaster loss assessment without rescue

[0135] First, we assess the potential disaster losses under an extreme scenario of forest fire spread and rescue—where no rescue orders are issued. This primarily evaluates the maximum extent of the forest fire and the maximum losses that may occur during this process, including possible secondary and derivative disasters. This can be considered the worst-case scenario, denoted as loss. max :

[0136]

[0137] In the above formula, n is the number of disaster-bearing carriers that may eventually be damaged in the scenario, while It refers to the price of the i-th damaged disaster-bearing carrier, used to quantitatively calculate the maximum loss.

[0138] (2) Calculation of the final reward function

[0139] When actually calculating the final reward function, we obtain the disaster loss assessment after emergency decision-making. Usually, the extent of loss is smaller if rescue is carried out than if no rescue is carried out. Based on this idea, the final reward function is defined as the disaster loss assessment at the end minus the disaster loss assessment without rescue. The final reward of the rescue process can be regarded as a positive externality manifested by the rescue action.

[0140] The disaster loss assessment results of emergency rescue actions are recorded as loss. action :

[0141]

[0142] In the above formula, m is the number of damaged disaster-bearing structures in the entire scenario after the rescue, while This refers to the price of the j-th damaged disaster-bearing structure. This refers to the degree of damage to the j-th damaged disaster-bearing carrier, used to quantitatively calculate the scene loss after rescue operations.

[0143] Therefore, the final reward function R final_step :

[0144]

[0145] Finally, the reward function R is used for forest fire spread and rescue scenarios. step :

[0146]

[0147] Step S206: Generate emergency decisions using the trained emergency decision generation model.

[0148] like Figure 3 As shown, based on the same inventive concept as the emergency decision generation method based on deep reinforcement learning, the present invention also provides an emergency decision generation device based on deep reinforcement learning, comprising:

[0149] Analysis unit 301 is used to analyze scenarios that require reinforcement learning for emergency decision-making and obtain scenario factors.

[0150] The first determining unit 302 is used to determine multiple behavioral instructions for emergency decision-making in a scenario based on scenario factors;

[0151] The second determining unit 303 is used to determine state information that characterizes the state of a sudden event in the scene based on scene factors;

[0152] Processing unit 304 is used to generate an auxiliary reward function based on multiple behavioral instructions and state information, and to train an emergency decision generation model through the auxiliary reward function;

[0153] The generation unit 305 is used to generate emergency decisions using a trained emergency decision generation model.

[0154] In one possible implementation, the analysis unit 301 in the apparatus provided by the embodiments of the present invention is specifically used for:

[0155] Analyze the scene to identify its scene factors;

[0156] Based on scenario factors, we obtain the following: emergency event factors that characterize the damage caused to the scenario, disaster-bearing carrier factors that characterize the objects affected by the emergency, and emergency management factors that characterize the emergency measures to respond to the emergency.

[0157] In one possible implementation, the second determining unit 303 in the apparatus provided by the embodiments of the present invention is specifically used for:

[0158] Determine the status information of an emergency based on the factors of the emergency;

[0159] Based on disaster-bearing carrier factors, determine the disaster-bearing carrier status information that characterizes the carrier status;

[0160] Emergency management status information, which characterizes the status of emergency measures, is determined based on emergency management factors.

[0161] In one possible implementation, the processing unit 304 in the apparatus provided by the embodiments of the present invention is specifically used for:

[0162] Based on multiple action instructions, disaster-bearing carrier status information, and emergency management status information, determine the first loss value caused by multiple action instructions when responding to emergencies;

[0163] Determine the second loss value caused by failure to respond to emergencies based on the status information of the disaster-bearing carrier;

[0164] The auxiliary reward function is determined based on the first loss value and the second loss value.

[0165] In one possible implementation, in the apparatus provided by the embodiments of the present invention, the processing unit 304 specifically determines the initial reward function using the following formula:

[0166]

[0167] Where l represents the number of damaged disaster-bearing structures, m represents the number of damaged rescue forces, n represents the number of rescue forces deployed, and loss Bi For damage to the disaster-bearing carrier, loss Ej Due to the damage to emergency response capabilities, R each_step This is the initial reward function.

[0168] In one possible implementation, the processing unit 304 in the apparatus provided by the embodiments of the present invention specifically determines the first loss value and the second loss value using the following formula:

[0169]

[0170]

[0171] The final reward function of the auxiliary reward function is as follows:

[0172]

[0173] Where m is the number of damaged disaster-bearing structures in the entire scenario after the rescue, and n is the number of disaster-bearing structures that may ultimately be damaged in the scenario. This refers to the price of the j-th damaged disaster-bearing structure. This refers to the price of the i-th damaged disaster-bearing carrier. The degree of damage to the j-th damaged disaster-bearing structure, loss action The first loss value is called loss. max R is the second loss value. final_step This is the final reward function.

[0174] In one possible implementation, in the apparatus provided by the embodiments of the present invention, the processing unit 304 specifically determines the auxiliary reward function using the following formula:

[0175]

[0176] Where l represents the number of damaged disaster-bearing structures, m represents the number of damaged rescue forces, n represents the number of rescue forces deployed, and loss B For damage to the disaster-bearing carrier, loss E Due to the loss of emergency response capabilities, action The first loss value is called loss. max R is the second loss value. step This is an auxiliary reward function.

[0177] In addition, combined Figures 1-3 The emergency decision generation method and apparatus based on deep reinforcement learning described in the embodiments of the present invention can be implemented by electronic devices. See below for details. Figure 4 It shows a schematic diagram of a structure suitable for implementing the electronic device 400 in the embodiments of this disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0178] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403 to implement the emergency decision generation method based on deep reinforcement learning as described in the embodiments of this disclosure. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0179] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0180] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the voice control method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0181] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0182] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0183] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0184] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0185] Analyze scenarios that require reinforcement learning for emergency decision-making to identify scenario factors;

[0186] Based on scenario factors, determine multiple behavioral instructions for emergency decision-making in the scenario;

[0187] Determine the state information used to characterize the state of sudden events in the scenario based on scenario factors;

[0188] An auxiliary reward function is generated based on multiple behavioral instructions and status information, and an emergency decision generation model is trained using the auxiliary reward function.

[0189] Emergency decisions are generated using a pre-trained emergency decision generation model.

[0190] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0191] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0193] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0194] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0195] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0196] In this embodiment of the invention, the scenario requiring reinforcement learning for emergency decision-making is first analyzed to obtain scenario factors. Then, based on these scenario factors, multiple behavioral instructions for emergency decision-making are determined. Next, state information characterizing the state of a sudden event in the scenario is determined based on these scenario factors. Then, an auxiliary reward function is generated based on the multiple behavioral instructions and state information, and the emergency decision-making generation model is trained using this auxiliary reward function. Finally, the trained emergency decision-making generation model is used to generate emergency decisions. Compared with related technologies, this invention solves the problems of poor generalization and difficulty in reusing emergency decision-making models based on deep reinforcement learning. By analyzing the scenario, a reward function is constructed for the emergency decision-making model, and this reward function is used to train the model, solving the problem of the lack of a unified standard for reward function construction and improving the generalization of the emergency decision-making model.

[0197] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0202] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An emergency decision generation method based on deep reinforcement learning, characterized in that, include: Analyze scenarios that require reinforcement learning for emergency decision-making to obtain scenario factors, which include sudden event factors, disaster-bearing carrier factors, and emergency management factors. Based on the scenario factors, multiple behavioral instructions are determined for making the emergency decision in the scenario; Based on the aforementioned scenario factors, status information is determined to characterize the state of the emergency in the scenario, wherein the status information includes emergency status information, disaster-bearing carrier status information, and emergency management status information; An auxiliary reward function is generated based on the multiple behavioral instructions and the state information, and an emergency decision generation model is trained using the auxiliary reward function. Generating the auxiliary reward function includes: Based on the multiple behavioral instructions, the disaster-bearing carrier status information, and the emergency management status information, determine the first loss value caused by the multiple behavioral instructions in response to the emergency; Based on the disaster-bearing carrier status information, determine the second loss value that should not be caused by the sudden event; The auxiliary reward function is determined based on the first loss value and the second loss value; Generate emergency decisions using a pre-trained emergency decision generation model; The initial reward function of the auxiliary reward function is determined by the following formula: in, The number of damaged structures. The number of rescue personnel injured, The number of rescue forces deployed. Damage to the disaster-bearing structure Due to the damage to emergency response capabilities, The initial reward function; The first loss value and the second loss value are determined using the following formula: The final reward function of the auxiliary reward function is as follows: in, It refers to the number of damaged disaster-bearing structures in the entire scene after the rescue operation. It refers to the number of disaster-bearing structures that may ultimately be damaged in the scenario. It refers to the first The price of a damaged disaster-bearing structure It refers to the first The price of a damaged disaster-bearing structure Refers to the first The extent of damage to each damaged disaster-bearing structure, The first loss value, This is the second loss value. The final reward function; The auxiliary reward function is determined using the following formula: in, The number of damaged structures. The number of rescue personnel injured, The number of rescue forces deployed. Damage to the disaster-bearing structure Due to the damage to emergency response capabilities, The first loss value, This is the second loss value. This is an auxiliary reward function.

2. The emergency decision generation method based on deep reinforcement learning according to claim 1, characterized in that, The analysis of scenarios requiring reinforcement learning for emergency decision-making yields scenario factors, including: The scenario is analyzed to obtain the scenario factors. Based on the scenario factors, we obtain the following: the sudden event factors that cause damage to the scenario, the disaster-bearing carrier factors that characterize the objects affected by the sudden event, and the emergency management factors that characterize the emergency measures to respond to the sudden event.

3. The emergency decision generation method based on deep reinforcement learning according to claim 2, characterized in that, The determination of state information to characterize the state of the sudden event in the scenario based on the scenario factors includes: Based on the aforementioned factors of the emergency, determine the emergency status information that characterizes the state of the emergency; Based on the aforementioned disaster-bearing carrier factors, determine the disaster-bearing carrier status information that characterizes the carrier's status; Emergency management status information, which characterizes the status of emergency measures, is determined based on the aforementioned emergency management factors.

4. An emergency decision generation device based on deep reinforcement learning, characterized in that, include: The analysis unit is used to analyze scenarios that require reinforcement learning for emergency decision-making and obtain the scenario factors of the scenario, wherein the scenario factors include sudden event factors, disaster-bearing carrier factors and emergency management factors. The first determining unit is used to determine multiple behavioral instructions for making emergency decisions in the scenario based on the scenario factors; The second determining unit is used to determine state information characterizing the state of the emergency event in the scenario based on the scenario factors, wherein the state information includes emergency event state information, disaster-bearing carrier state information and emergency management state information; The processing unit is configured to generate an auxiliary reward function based on the plurality of behavioral instructions and the state information, and to train an emergency decision generation model using the auxiliary reward function, wherein generating the auxiliary reward function includes: Based on the multiple behavioral instructions, the disaster-bearing carrier status information, and the emergency management status information, determine the first loss value caused by the multiple behavioral instructions in response to the emergency; Based on the disaster-bearing carrier status information, determine the second loss value that should not be caused by the sudden event; The auxiliary reward function is determined based on the first loss value and the second loss value; The generation unit is used to generate emergency decisions using a pre-trained emergency decision generation model. Specifically, the processing unit is used for: The initial reward function of the auxiliary reward function is determined by the following formula: in, The number of damaged structures. The number of rescue personnel injured, The number of rescue forces deployed. Damage to the disaster-bearing structure Due to the damage to emergency response capabilities, The initial reward function; The first loss value and the second loss value are determined using the following formula: The final reward function of the auxiliary reward function is as follows: in, It refers to the number of damaged disaster-bearing structures in the entire scene after the rescue operation. It refers to the number of disaster-bearing structures that may ultimately be damaged in the scenario. It refers to the first The price of a damaged disaster-bearing structure It refers to the first The price of a damaged disaster-bearing structure Refers to the first The extent of damage to each damaged disaster-bearing structure, The first loss value, This is the second loss value. The final reward function; The auxiliary reward function is determined using the following formula: in, The number of damaged structures. The number of rescue personnel injured, The number of rescue forces deployed. Damage to the disaster-bearing structure Due to the damage to emergency response capabilities, The first loss value, This is the second loss value. This is an auxiliary reward function.

5. An electronic device, characterized in that, include: At least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by a processor, the method as described in any one of claims 1-3 is implemented.

Images