Non-steady-state crowd evacuation path planning method and device based on deep reinforcement learning

Abstract

The application relates to a non-steady crowd evacuation path planning method and device based on deep reinforcement learning. The method comprises the following steps: simplifying a non-steady evacuation environment into a two-dimensional region by using a discrete grid method. A dangerous source dynamic diffusion model is established, and the fire source position and range are updated in real time according to the model, so that the dynamic development of the fire is simulated. People are regarded as agents, and move according to the decision rule of the potential field guidance. A non-steady crowd evacuation path planning model is established by taking the maximum number of evacuees and the minimum evacuation time as the objective function, combining the actual situation of a shopping mall and the movement rule of the agents, and determining the constraint conditions. An APF-MADDPG algorithm is used to convert the path planning into a Markov decision process, a state feature is designed, a deep reinforcement learning framework is established by combining the potential field, and the model is used to realize non-steady crowd evacuation path planning after being trained. The method can improve the practicability and real-time performance of crowd evacuation path planning.

Description

Technical Field

[0001] The present application relates to the field of crowd evacuation safety technology, and in particular to a non-steady-state crowd evacuation path planning method and device based on deep reinforcement learning. Background Art

[0002] With the acceleration of urbanization, crowd gatherings are increasing. Crowd evacuation in non-steady-state environments (such as the dynamic diffusion of hazardous sources during a fire or fluctuating accessibility and obstruction at entrances and exits) presents significant challenges. Traditional crowd evacuation path planning algorithms, such as heuristic path search algorithms based on A-star, mostly rely on static environmental modeling and are unable to adapt to dynamic changes in non-steady-state environments, making it difficult to provide real-time, reasonable evacuation paths. Current crowd guidance methods based on reinforcement learning mostly focus on optimizing individual decisions, ignoring the significant impact of interactions between individuals within a group and the dynamic changes in group behavior on evacuation efficiency. Furthermore, they lack understanding of non-steady-state environmental modeling and the mechanisms of group interaction. Therefore, an efficient method is urgently needed to solve the problem of crowd evacuation path planning in non-steady-state environments to ensure safety and improve evacuation efficiency.

[0003] As early as the end of the last century, several evacuation route planning methods based on traditional optimization theory were proposed. These methods often relied on static environmental assumptions, such as precalculating fixed routes and allowing people to evacuate along predetermined routes. Traditional methods planned routes based on static information such as building layout and crowd flow distribution. The principles were relatively straightforward, with the classic Floyd-Warshall algorithm used for path optimization differing only in details such as weight settings. However, the performance of these traditional methods relied on stable environmental conditions. In real-world scenarios such as fires and earthquakes, the environment changes dynamically over time, and smoke spread and building damage constantly alter feasible routes, significantly reducing the practicality of these methods.

[0004] Evacuation planning methods based on model predictive control attempt to address dynamic environments. They model the evacuation process as a dynamic system, using predictive models to anticipate environmental changes and adjust the route in advance. These methods leverage system state prediction and rolling optimization to adapt to environmental dynamics to a certain extent. For example, they can adjust crowd evacuation directions by predicting the range of smoke diffusion. However, model predictive control requires extremely high model accuracy. In complex, non-steady-state environments, accurately modeling smoke diffusion and crowd movement is difficult to construct, and computational complexity increases dramatically with prediction time. Consequently, in practical applications, evacuation decisions are delayed, making it difficult to ensure real-time performance. Summary of the Invention

[0005] Based on this, it is necessary to provide a non-steady-state crowd evacuation path planning method and device based on deep reinforcement learning that can improve the practicality and real-time performance of crowd evacuation path planning in response to the above technical problems.

[0006] A non-steady crowd evacuation path planning method based on deep reinforcement learning, the method comprises:

[0007] A discrete grid method is used to simplify the modeling of the non-steady evacuation environment, and the non-steady evacuation environment is abstracted as a two-dimensional region, each grid represents an area, and the grid includes obstacle, exit, hazard source and crowd density information;

[0008] The dynamic diffusion characteristics of the hazard source are modeled to obtain a hazard source diffusion model; the position and range of the fire source are updated in real time based on the hazard source diffusion model to simulate the dynamic development process of the fire;

[0009] In the dynamic development process of the fire, the personnel are regarded as agents, the agents move according to the decision rule of the potential energy field guidance, the objective function is to maximize the number of evacuated people while minimizing the evacuation time, the constraint conditions are determined according to the actual situation of the shopping mall and the agent movement law, and the non-steady crowd evacuation path planning model is constructed by using the objective function and the constraint conditions;

[0010] An APF-MADDPG algorithm is used to convert the path planning process into a Markov decision process, and a deep reinforcement learning framework is constructed by designing state features combined with a potential energy field method; the non-steady crowd evacuation path planning model is trained by using the deep reinforcement learning framework to obtain a trained non-steady crowd evacuation path planning model; and the non-steady crowd evacuation path planning is realized according to the trained non-steady crowd evacuation path planning model.

[0011] A non-steady crowd evacuation path planning method based on deep reinforcement learning, the device comprises:

[0012] An environment modeling module is configured to simplify the modeling of the non-steady evacuation environment by using a discrete grid method, and to abstract the non-steady evacuation environment as a two-dimensional region, each grid representing an area, and the grid including obstacle, exit, hazard source and crowd density information;

[0013] A hazard source dynamic diffusion characteristic modeling module is configured to model the dynamic diffusion characteristics of the hazard source to obtain a hazard source diffusion model; and to update the position and range of the fire source in real time based on the hazard source diffusion model to simulate the dynamic development process of the fire;

[0014] A non-steady crowd evacuation path planning model construction module is configured to model the non-steady crowd evacuation path planning model in the dynamic development process of the fire, the personnel are regarded as agents, the agents move according to the decision rule of the potential energy field guidance, the objective function is to maximize the number of evacuated people while minimizing the evacuation time, the constraint conditions are determined according to the actual situation of the shopping mall and the agent movement law, and the non-steady crowd evacuation path planning model is constructed by using the objective function and the constraint conditions;

[0015] Crowd evacuation path planning modeling is used to adopt the APF-MADDPG algorithm to transform the path planning process into a Markov decision process, and design state features combined with the potential field method to construct a deep reinforcement learning framework; the deep reinforcement learning framework is used to train the non-steady-state crowd evacuation path planning model to obtain a trained non-steady-state crowd evacuation path planning model; and non-steady-state crowd evacuation path planning is implemented based on the trained non-steady-state crowd evacuation path planning model.

[0016] The above-mentioned method and device for non-steady-state crowd evacuation path planning based on deep reinforcement learning first uses a discrete grid approach to simplify the modeling of the non-steady-state evacuation environment, breaking down the complex space into standardized units and integrating core information such as obstacles and hazards. When the environment changes, only the corresponding grid properties need to be updated, eliminating the need for model reconstruction. The standardized format facilitates fast computation, addressing the path lag problem inherent in traditional static modeling. A dynamic hazard diffusion model accurately simulates fire development by updating the fire source status in real time. This allows for path planning based on the latest hazard information, preventing people from straying into hazardous areas. This overcomes the limitations of model predictive control methods, which rely on high-precision models and high computational complexity. The agent design then incorporates potential energy field guidance, simulating individual instincts to seek benefits and avoid harm while also accounting for group interactions through potential energy field superposition to avoid crowd congestion. With the objective function of maximizing the number of people evacuated while minimizing evacuation time, constraints are determined based on the actual situation of the shopping mall and the movement patterns of the agents. The objective function and constraints are then used to construct a non-steady-state crowd evacuation path planning model. A framework built using the APF-MADDPG algorithm transforms path planning into a Markov decision process. The trained model directly outputs the optimal action, reducing real-time computational effort. Multi-agent collaboration addresses group dynamics, and the potential energy field reduces exploration complexity, addressing the computational and training challenges of traditional algorithms. Furthermore, this model is highly portable; switching scenarios requires only adjusting basic parameters, avoiding the poor reusability of traditional methods. It can be quickly adapted to different scenarios and is highly practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 1 is a flowchart of a non-steady-state crowd evacuation path planning method based on deep reinforcement learning in one embodiment;

[0018] Figure 2 Schematic diagram of building scene modeling based on discrete grid method in non-steady-state environment modeling in one embodiment; (a) is a schematic diagram of hazard source modeling, and (b) is a schematic diagram of exit modeling;

[0019] Figure 3 A schematic diagram of a deep reinforcement learning framework in one embodiment;

[0020] Figure 4 A schematic diagram of a state transition process in another embodiment;

[0021] Figure 5 A schematic diagram of the design of the intelligent agent action space in one embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0023] In one embodiment, a non-steady-state scenario of a fire in a large shopping mall is taken as an example to explain the non-steady-state crowd evacuation path planning method based on deep reinforcement learning in detail. The internal structure of the shopping mall is complex, with multiple floors and many stores. When a fire occurs, the fire spreads, the smoke spreads, and some entrances and exits may be blocked. Figure 1 As shown, a non-steady-state crowd evacuation path planning method based on deep reinforcement learning is provided, which includes the following steps:

[0024] Step 102 , using a discrete grid method to simplify modeling of the non-steady-state evacuation environment, abstracting the non-steady-state evacuation environment into a two-dimensional area, where each grid represents an area, and the grid includes information about obstacles, exits, sources of danger, and crowd density.

[0025] The two-dimensional grid breaks down complex spaces such as shopping malls into standardized units. Each grid integrates core information such as obstacles, exits, sources of danger, and crowd density, which not only retains the key characteristics of the environment but also reduces the complexity of data processing. This structured modeling method avoids the reliance of traditional static modeling on fixed scenes. When the accessibility status of entrances and exits changes or the position of obstacles changes in a non-steady-state environment, it is only necessary to update the attribute labels of the corresponding grids without reconstructing the overall model, laying the foundation for real-time path adjustment. At the same time, the standardized format of the grid facilitates rapid reading and calculation by computers, solving the problem of path planning lag caused by ambiguous scene descriptions in dynamic environments caused by traditional methods. Figure 2 The figure shows a schematic diagram of building scene modeling based on the discrete grid method in a non-steady-state environment modeling, (a) is a schematic diagram of hazard source modeling, and (b) is a schematic diagram of exit modeling.

[0026] Step 104 , modeling the dynamic diffusion characteristics of the hazard source to obtain a hazard source diffusion model; based on the hazard source diffusion model, the location and range of the fire source are updated in real time to simulate the dynamic development process of the fire.

[0027] The dynamic diffusion characteristics of hazardous sources are modeled. By updating the fire source's location and range in real time, the model accurately simulates dynamic processes such as smoke spread and high-temperature diffusion during a fire. Compared to traditional static path planning, this dynamic simulation ensures that path planning is always based on the latest hazardous area information. Through real-time perception and instant updates, it can respond to environmental changes without the need for long-term predictions, significantly improving its practicality in real-world disaster scenarios.

[0028] Step 106: During the dynamic development of the fire, the personnel are treated as intelligent agents, and the intelligent agents move according to the decision-making rules guided by the potential energy field. The objective function is to maximize the number of evacuees while minimizing the evacuation time. The constraints are determined according to the actual situation of the shopping mall and the movement rules of the intelligent agents. The objective function and the constraints are used to construct a non-steady-state crowd evacuation path planning model.

[0029] The mall's personnel are treated as agents, each moving according to a decision-making rule guided by a potential energy field. At each time step, the agent perceives the state of its own grid and adjacent grids, calculates the combined potential energy gradient, and selects its next move based on the combined potential energy and crowd density, avoiding fire sources and obstacles, thereby achieving fire avoidance and exit movement. By treating people as agents and making decisions based on the potential energy field, the algorithm simulates the natural reaction of individuals in a real crowd to move away from danger and toward the exit, while also accounting for group interactions through the cumulative effect of the potential energy field. When crowd density in a certain area is too high, the congestion potential around the agent increases, guiding it to automatically choose a more unobstructed path, thus avoiding the inefficient evacuation caused by crowd congestion that traditional individual optimization methods often ignore. Furthermore, with the objective function of maximizing the number of people evacuated and minimizing evacuation time, constraints are set based on the mall's actual conditions, such as aisle width and exit capacity. This ensures that path planning is both compliant with physical space constraints and focuses on the core evacuation objectives, thus avoiding the disconnect between theoretical solutions and practical scenarios.

[0030] Step 108: Use the APF-MADDPG algorithm to transform the path planning process into a Markov decision process, and design state features combined with the potential field method to construct a deep reinforcement learning framework; use the deep reinforcement learning framework to train the non-steady-state crowd evacuation path planning model to obtain a trained non-steady-state crowd evacuation path planning model; and implement non-steady-state crowd evacuation path planning based on the trained non-steady-state crowd evacuation path planning model.

[0031] Use APF-MADDPG algorithm for training, such as Figure 3As shown, during the training process, the agent continuously tries different moving paths in the mall environment, obtains a reward value according to the composite reward function, and the experience replay mechanism randomly extracts experience for network training to update the Actor and Critic network parameters. Among them, Actor and Critic represent the action network and evaluation network respectively, the action network generates action A according to state S, and the evaluation network designs the composite reward function to calculate the reward value. Target Actor and Target Critic are introduced as target networks, and the parameters are copied from the action network and the evaluation network through the soft update mechanism. Through a large number of training iterations, the agent learns the optimal evacuation strategy. After training, when the mall experiences non-steady-state conditions such as fire, the agent can quickly and reasonably plan the evacuation path according to the learned strategy to achieve efficient evacuation. For example, at a certain moment, the agent perceives the spread of fire around it, calculates the potential energy gradient, selects a path away from the fire source and with lower crowd density to move towards the exit, successfully avoids the dangerous area, and improves the safety and efficiency of evacuation.

[0032] By using the APF-MADDPG algorithm, the path planning is converted into a Markov decision process, and through the combination of state features and potential energy field, the agent can quickly generate reasonable actions after perceiving the grid environment. Unlike traditional heuristic algorithms that require repeated search of global paths, the decision model formed by the reinforcement learning framework can directly output local optimal actions, greatly reducing real-time computation. At the same time, the multi-agent collaboration mechanism of MADDPG can handle the dynamic changes brought by group interaction, avoiding the limitations of single-agent decision-making; and the potential energy field as prior knowledge is integrated into the framework, reducing the exploration difficulty of reinforcement learning, so that the model can converge to an efficient strategy under limited training data, solving the problem of long training period and difficulty in practical application in complex scenarios. At the same time, the model's portability further enhances its practicality. When the scene switches from a mall to other gathering places such as a gymnasium, only the basic settings such as grid size and dangerous source diffusion parameters need to be adjusted, without the need to redesign the algorithm framework. This flexibility avoids the poor reusability problem caused by traditional methods being customized for specific scenarios, allowing it to quickly adapt to different non-steady-state evacuation scenarios and truly meet the needs of practical applications.

[0033] The aforementioned deep reinforcement learning-based non-stationary crowd evacuation path planning method first uses a discrete grid approach to simplify the modeling of the non-stationary evacuation environment, breaking the complex space into standardized units and integrating core information such as obstacles and hazards. When the environment changes, only the corresponding grid properties need to be updated, eliminating the need for model reconstruction. The standardized format facilitates fast computation, addressing the path lag problem inherent in traditional static modeling. A dynamic hazard diffusion model accurately simulates fire development by updating the fire source status in real time. This allows for path planning based on the latest hazard information, preventing people from straying into hazardous areas. This overcomes the limitations of model predictive control methods, which rely on high-precision models and high computational complexity. The agent design then incorporates potential energy field guidance, simulating individual instincts to seek benefits and avoid harm while also accounting for group interactions through potential energy field superposition to avoid crowd congestion. With the objective function of maximizing the number of people evacuated while minimizing evacuation time, constraints are determined based on the actual situation of the shopping mall and the agent's movement patterns. The objective function and constraints are then used to construct a non-stationary crowd evacuation path planning model. A framework built using the APF-MADDPG algorithm transforms path planning into a Markov decision process. The trained model directly outputs the optimal action, reducing real-time computational effort. Multi-agent collaboration addresses group dynamics, and the potential energy field reduces exploration complexity, addressing the computational and training challenges of traditional algorithms. Furthermore, this model is highly portable; switching scenarios requires only adjusting basic parameters, avoiding the poor reusability of traditional methods. It can be quickly adapted to different scenarios and is highly practical.

[0034] In one embodiment, the dynamic diffusion characteristics of the hazard source are modeled to obtain a hazard source diffusion model, including:

[0035] Model the dynamic diffusion characteristics of the hazard source, and assume that the hazard source set is , the obstacle set is , the time interval for the spread of hazardous sources is ,experience The set of hazardous sources with diffusion time intervals is , through the neighborhood function Searching adjacent areas, the hazard source diffusion model is:

[0036] .

[0037] In one embodiment, when planning crowd evacuation routes in a non-steady-state environment, agent behavior modeling must reflect the dual characteristics of avoiding dynamic hazard sources and navigating exits. The agent is abstracted as an entity occupying a single cell, whose movement follows decision-making rules guided by a potential energy field. Behavioral modeling is achieved by combining the repulsive force of hazard sources with the attractive force of exits.

[0038] The decision rule guided by the potential energy field is that the agent perceives the state information of its own grid and adjacent grids in each time step, calculates the total potential energy gradient, and selects the area with the smallest total potential energy among the adjacent areas as the next position. If there are multiple equivalent directions, the area with lower crowd density is preferred to avoid the fire and move to the exit. The agent position update formula is: ,speed It is determined by normalizing the total potential energy gradient, that is, ,in is the maximum moving speed of the agent, Indicates the When an agent reaches the exit area, it is considered to have successfully evacuated and removed from the environment.

[0039] In one embodiment, the resultant potential energy gradient is calculated as:

[0040] ;

[0041] in, Indicates the The total potential energy gradient of each agent is Indicates the The repulsive potential energy gradient of each intelligent agent from the danger source, Indicates the The attractive potential energy gradient of each agent from the exit, Indicates the A source of danger, Indicates the An exit.

[0042] In one embodiment, a non-steady-state crowd evacuation path planning model is constructed using an objective function and constraint conditions, including:

[0043] The non-steady-state crowd evacuation path planning model is constructed using the objective function and constraints:

[0044] ;

[0045] Constraints:

[0046] , , .

[0047] in, represents the total number of agents, Indicates the Agents at time The location, Indicates an exit in an evacuation environment. express The time it takes for all agents to leave the evacuation environment, Indicates that the regular expression is true. Indicates the first grid area, Indicates the first The grid area is an obstacle. Indicates the first The grid area is the hazard source. Indicates the first The grid area is the exit.

[0048] In a specific embodiment, Indicates that the current area cannot be used as an exit if it is an obstacle or a danger source. Indicates that the evacuated subject cannot enter the area where there are danger sources or obstacles. Indicates that the evacuation is successful when the crowd moves to the exit.

[0049] In one embodiment, the deep reinforcement learning framework includes a non-steady-state environment, an actor-critic network training module, a dynamic perception module based on a potential energy field, and a dynamic experience pool; the actor-critic network training module adopts an actor-critic architecture, the action network generates an action A based on the state S, the evaluation network designs a composite reward function to calculate the reward value, introduces a target actor and a target critic as the target network, and copies parameters from the action network and the evaluation network through a soft update mechanism.

[0050] In a specific embodiment, the state transfer process is as follows: Figure 4 As shown, the action space of the intelligent agent is designed as Figure 5 shown.

[0051] In one embodiment, the compound reward function is: ;

[0052] in, Indicates the full reward when the agent reaches the target point. Represents the potential energy field guidance item, which uses the dynamic potential energy difference mechanism to guide the agent to move in the direction of potential energy decrease and guide the agent to move towards the goal. represents the risk aversion module consisting of fire rejection terms, represents the collision penalty term, represents the time penalty term.

[0053] In a specific embodiment, the target drive module is driven by the core incentive item When the agent reaches the target point, it will start with full reward and potential field guidance (Using dynamic potential energy difference mechanism to guide the agent to move in the direction of potential energy decrease) guide the agent to move towards the target; the risk avoidance module is composed of fire rejection items (Build an inversely proportional potential energy field, and the penalty term increases inversely when the agent enters the influence radius of the fire source) and the collision penalty term (Using multi-body interaction model to achieve autonomous avoidance during group evacuation) composition, allowing the intelligent agent to avoid risks; the efficiency optimization module uses time penalty items Ensure strategy diversity in the early exploration stage, accelerate convergence in the later stages of training, and improve planning efficiency.

[0054] In one embodiment, the objective function of the evaluation network is:

[0055] ;

[0056] ;

[0057] in, represents the target quantity related to the target of the Critic network, Represents the state sampled from the experience replay buffer D ,action ,award , next state The expectations on these data samples are represents the target Q value, Indicates status Doing action After that, the expected cumulative reward that can be obtained according to the strategy is represents the coefficient used for regularization, Represents the parameters of the Critic network, represents the coefficient of variance regularization, represents the variance, express The cumulative reward value obtained by the function, represents the immediate reward obtained by the agent, represents the discount factor, represents the output of the target Q network, Indicates execution of an action The next state to be entered after the transition, represents the action output by the target action network;

[0058] The objective function of the action network is:

[0059] ;

[0060] in, Express expectations, represents the gradient operator for the action, represents the gradient operator of the action network parameters, Indicates the The strategy output by the action network, Representing Action Network The corresponding input, represents the parameter used to control the strength of KL divergence regularization, represents the old policy, Indicates a new policy.

[0061] In a specific embodiment, the critic network objective function uses centralized TD error optimization to maximize the match between the Q value and the target value, allowing the critic to approach the true value function. The actor network objective function is based on the deterministic policy gradient theorem, which is used to calculate the gradient direction of the actor network parameters to maximize the long-term accumulated reward.

[0062] In one embodiment, a dynamic sensing module based on a potential energy field defines a repulsive potential energy field. Describes the repulsive effect of dynamic danger sources and obstacles on the agent. Less than threshold hour, , is 0 when it is greater than the threshold; attractive potential energy field Describe the attraction of the exit to the agent, where represents the repulsion coefficient, represents the attraction coefficient, Indicates the The coordinates of the agent, Indicates the The coordinates of the exit.

[0063] In one embodiment, the dynamic experience pool is used to set the experience replay mechanism based on the absolute value of the TD error. Set sampling weights , the weight of experience in gradient update is defined as At the same time, a dynamic capacity experience pool is used, and the capacity of the experience pool is dynamically expanded with the training progress. The calculation formula is ,in, Indicates the initial capacity of the experience storage pool. The experience replay buffer adopts a "dictionary-queue" composite architecture.

[0064] In the specific implementation, the experience replay pool module replays the experience generated by the agent in the environment interaction. The experience is stored in the experience replay pool, and experience is randomly extracted through training sampling for network training to improve training efficiency.

[0065] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0066] In one embodiment, a method for non-steady-state crowd evacuation path planning based on deep reinforcement learning is provided, wherein the apparatus includes:

[0067] The environmental modeling module is used to simplify the modeling of the non-steady-state evacuation environment using the discrete grid method. The non-steady-state evacuation environment is abstracted into a two-dimensional area, where each grid represents an area and includes information about obstacles, exits, sources of danger, and crowd density.

[0068] Hazard source dynamic diffusion characteristic modeling is used to model the dynamic diffusion characteristics of the hazard source and obtain a hazard source diffusion model; based on the hazard source diffusion model, the location and range of the fire source are updated in real time to simulate the dynamic development process of the fire;

[0069] A non-steady-state crowd evacuation path planning model was constructed. During the dynamic development of a fire, personnel were treated as intelligent agents, and the agents moved according to decision-making rules guided by the potential energy field. The objective function was to maximize the number of evacuees while minimizing the evacuation time. Constraints were determined based on the actual situation of the mall and the movement patterns of the intelligent agents. The objective function and constraints were used to construct a non-steady-state crowd evacuation path planning model.

[0070] Crowd evacuation path planning modeling is used to adopt the APF-MADDPG algorithm to transform the path planning process into a Markov decision process, and design state features combined with the potential field method to construct a deep reinforcement learning framework; the deep reinforcement learning framework is used to train the non-steady-state crowd evacuation path planning model to obtain a trained non-steady-state crowd evacuation path planning model; and non-steady-state crowd evacuation path planning is implemented based on the trained non-steady-state crowd evacuation path planning model.

[0071] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0072] The above embodiments only express several implementation ways of the present application, and the description is more specific and detailed, but it should not be understood as a limitation to the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A non-steady-state crowd evacuation path planning method based on deep reinforcement learning, characterized in that: The method comprises: The discrete grid method is used to simplify the modeling of the non-steady evacuation environment. The non-steady evacuation environment is abstracted into a two-dimensional area. Each grid represents an area. The grid includes information about obstacles, exits, danger sources, and crowd density. Modeling the dynamic diffusion characteristics of the hazard source to obtain a hazard source diffusion model; based on the hazard source diffusion model, updating the location and range of the fire source in real time to simulate the dynamic development process of the fire; During the dynamic development of the fire, people are treated as intelligent agents, and the intelligent agents move according to the decision rules guided by the potential energy field. The objective function is to maximize the number of evacuees while minimizing the evacuation time. Constraints are determined based on the actual situation of the shopping mall and the movement rules of the intelligent agents. The objective function and constraints are used to construct a non-steady-state crowd evacuation path planning model; The APF-MADDPG algorithm is used to transform the path planning process into a Markov decision process, and a state feature is designed in combination with the potential field method to construct a deep reinforcement learning framework; the non-steady-state crowd evacuation path planning model is trained using the deep reinforcement learning framework to obtain a trained non-steady-state crowd evacuation path planning model; non-steady-state crowd evacuation path planning is implemented based on the trained non-steady-state crowd evacuation path planning model; the deep reinforcement learning framework includes a non-steady-state environment, an Actor-Critic network training module, a dynamic perception module based on a potential field, and a dynamic experience pool; the Actor-Critic network training module adopts an Actor-Critic architecture, the action network generates an action A based on the state S, the evaluation network designs a composite reward function to calculate the reward value, the Target Actor and Target Critic are introduced as the target network, and parameters are copied from the action network and the evaluation network through a soft update mechanism; Model the dynamic diffusion characteristics of the hazard source to obtain a hazard source diffusion model, including: Model the dynamic diffusion characteristics of the hazard source, and assume that the hazard source set is , the obstacle set is , the time interval for the spread of hazardous sources is ,experience The set of hazardous sources with diffusion time intervals is , through the neighborhood function Searching adjacent areas, the hazard source diffusion model is: 。 2. The method according to claim 1, characterized in that The decision rule guided by the potential energy field is that the agent perceives the state information of its own grid and adjacent grids in each time step, calculates the combined potential energy gradient, and selects the area with the smallest combined potential energy among the adjacent areas as the next position. If there are multiple equivalent directions, the area with the lowest crowd density is selected first to avoid the fire and move to the exit. The agent position update formula is: ,speed It is determined by normalizing the total potential energy gradient, that is, ,in is the maximum moving speed of the agent, Indicates the an agent; The calculated total potential energy gradient is: ； in, Indicates the The total potential energy gradient of each agent is Indicates the The repulsive potential energy gradient of each intelligent agent from the danger source, Indicates the The attractive potential energy gradient of each agent from the exit, Indicates the A source of danger, Indicates the An exit.

3. The method according to claim 1, characterized in that The objective function and constraint conditions are used to construct a non-steady-state crowd evacuation path planning model, including: The non-steady-state crowd evacuation path planning model is constructed using the objective function and constraint conditions: ； Constraints: ， ， ； in, represents the total number of agents, Indicates the Agents at time The location, Indicates an exit in an evacuation environment. express The time it takes for all agents to leave the evacuation environment, Indicates that the regular expression is true. Indicates the first grid area, Indicates the first The grid area is an obstacle. Indicates the first The grid area is the hazard source. Indicates the first The grid area is the exit.

4. The method according to claim 1, wherein The compound reward function is: ； in, Indicates the full reward when the agent reaches the target point. Represents the potential energy field guidance item, which uses the dynamic potential energy difference mechanism to guide the agent to move in the direction of potential energy decrease and guide the agent to move towards the goal. represents the risk aversion module consisting of fire rejection terms, represents the collision penalty term, represents the time penalty term.

5. The method according to claim 4, characterized in that The objective function of the evaluation network is: ； ； in, represents the target quantity related to the target of the Critic network, Represents the state sampled from the experience replay buffer D ,action ,award , next state The expectations on these data samples are represents the target Q value, Indicates status Doing action After that, the expected cumulative reward obtained according to the strategy is represents the coefficient used for regularization, Represents the parameters of the Critic network, represents the coefficient of variance regularization, represents the variance, express The cumulative reward value obtained by the function, represents the immediate reward obtained by the agent, represents the discount factor, represents the output of the target Q network, Indicates execution of an action The next state to be entered after the transition, represents the action output by the target action network; The objective function of the action network is: ； in, Express expectations, represents the gradient operator for the action, represents the gradient operator of the action network parameters, Indicates the The strategy output by the action network, Representing Action Network The corresponding input, represents the parameter used to control the strength of KL divergence regularization, represents the old policy, Indicates a new policy.

6. The method according to claim 1, characterized in that The dynamic perception module based on potential energy field defines a repulsive potential energy field Describes the repulsive effect of dynamic danger sources and obstacles on the agent. Less than threshold hour, , when it is greater than the threshold, it is 0; attractive potential energy field Describe the attraction of the exit to the agent, where represents the repulsion coefficient, represents the attraction coefficient, Indicates the The coordinates of the agent, Indicates the The coordinates of the exit.

7. The method according to claim 1, characterized in that The dynamic experience pool is used to set the experience replay mechanism and set the sampling weight based on the absolute value of the TD error δ , the weight of experience in gradient update is defined as At the same time, a dynamic capacity experience pool is used, and the capacity of the experience pool is dynamically expanded as the training progresses. The calculation formula is: ,in, Indicates the initial capacity of the experience storage pool. The experience replay buffer adopts a "dictionary-queue" composite architecture.

8. A non-steady-state crowd evacuation path planning device based on deep reinforcement learning, characterized in that: The device comprises: An environmental modeling module is used to simplify the modeling of the non-steady-state evacuation environment using a discrete grid method, abstracting the non-steady-state evacuation environment into a two-dimensional area, where each grid represents an area and includes information about obstacles, exits, sources of danger, and crowd density; Danger source dynamic diffusion characteristic modeling is used to model the dynamic diffusion characteristics of the hazard source to obtain a hazard source diffusion model; based on the hazard source diffusion model, the location and range of the fire source are updated in real time to simulate the dynamic development process of the fire; the dynamic diffusion characteristics of the hazard source are modeled to obtain a hazard source diffusion model, including: Model the dynamic diffusion characteristics of the hazard source, and assume that the hazard source set is , the obstacle set is , the time interval for the spread of hazardous sources is ,experience The set of hazardous sources with diffusion time intervals is , through the neighborhood function Searching adjacent areas, the hazard source diffusion model is: ； Construct a non-steady-state crowd evacuation path planning model, which is used to treat people as intelligent agents during the dynamic development of the fire. The intelligent agents move according to the decision rules guided by the potential energy field, with the objective function of maximizing the number of evacuees while minimizing the evacuation time. The constraints are determined according to the actual situation of the shopping mall and the movement rules of the intelligent agents, and the non-steady-state crowd evacuation path planning model is constructed using the objective function and the constraints. Crowd evacuation path planning modeling is used to use the APF-MADDPG algorithm to transform the path planning process into a Markov decision process, and design state features combined with the potential field method to construct a deep reinforcement learning framework; use the deep reinforcement learning framework to train the non-steady-state crowd evacuation path planning model to obtain a trained non-steady-state crowd evacuation path planning model; implement non-steady-state crowd evacuation path planning based on the trained non-steady-state crowd evacuation path planning model; the deep reinforcement learning framework includes a non-steady-state environment, an actor-critic network training module, and a dynamic perception module and dynamic experience pool based on a potential field; the actor-critic network training module adopts an actor-critic architecture, the action network generates an action A based on the state S, the evaluation network designs a composite reward function to calculate the reward value, introduces a target actor and a target critic as the target network, and copies parameters from the action network and the evaluation network through a soft update mechanism.

Images