A multi-agent reinforcement learning-based personnel evacuation simulation method and system

Abstract

The application discloses a kind of personnel evacuation simulation method and system based on multi-agent reinforcement learning, it is related to computer simulation and public safety and emergency management technical field.The method carries out parameterization modeling to evacuation environment, constructs global dynamic scene image and local observation state, establishes discrete action space for each evacuation agent, and combines collision avoidance algorithm to execute safe displacement update;Further based on local density, motion smoothness, evacuation time and target distance, improve the construction of immediate reward, rely on convolutional neural network to realize multi-agent collaborative training, and adopt the action sampling mechanism based on fuzzy constraint to generate evacuation decision considering behavior rationality and randomness, finally output individual trajectory, evacuation time, evacuation flow rate and exit neighborhood density distribution result, can be used for evacuation deduction, safety evaluation, model verification and emergency decision support in complex scene, with strong engineering applicability and simulation analysis value.

Description

Technical Field

[0001] This invention belongs to the field of computer simulation and public safety and emergency management technology, and relates to the characterization of crowd evacuation behavior in complex building environments, multi-agent collaborative decision-making training, evacuation path evolution simulation and result evaluation, and in particular to a personnel evacuation simulation method and system based on multi-agent reinforcement learning. Background Technology

[0002] In emergency events such as building fires and public safety accidents, the evacuation process is often accompanied by a high degree of uncertainty and complexity. Individuals are not only affected by objective environmental factors such as spatial layout, exit location, and evacuation instructions during evacuation, but also by their individual risk preferences, information comprehension abilities, and the continuous influence of the behavior of the surrounding crowd, thus exhibiting significant individual differences and behavioral randomness.

[0003] To describe and analyze the behavioral characteristics of evacuated crowds, numerous studies have proposed various evacuation crowd simulation methods based on experimental observations and empirical summaries. Among these, the most typical include rule-based methods, mechanics-based methods, and cellular automata models (CAMs). Rule-based methods typically describe individual path selection, avoidance, and following behaviors through pre-defined behavioral rules; mechanics-based methods, represented by the Social Force Model (SFM), simulate collective movement patterns formed during evacuation by constructing forces between individuals and between individuals and the environment; cellular automata models describe crowd movement and congestion phenomena through discrete space and local transfer rules. These models have been widely applied in engineering practice and experimental reproduction, and can reproduce typical evacuation phenomena such as detours, following, and local congestion to a certain extent.

[0004] However, the aforementioned traditional evacuation models generally rely on pre-designed rule structures or fixed parameters, and the behavioral patterns they can represent are largely limited by the assumptions introduced during the model design phase. Although these rules and parameters are usually derived from experimental observations or empirical inductions, they are essentially still abstract expressions of known evacuation phenomena. In actual evacuation processes, crowd behavior does not stably converge to a certain pattern, but may evolve, shift, or even mutate among multiple potential behavioral patterns as environmental conditions, information acquisition methods, or individual cognitive differences change. Existing models still have significant shortcomings in systematically exploring the diversity, randomness, and dynamic evolution of such collective behaviors in evacuation, and their applicability and generalization ability are limited in complex or insufficiently observed evacuation scenarios. For example, Chinese patent CN119579376B discloses a method for simulating the evacuation of people in a building, which determines the target exit by calculating the predicted queuing and evacuation time of each exit in real time and determines the simulated speed by combining the forward distance of pedestrian characteristics. However, this method is essentially a deterministic optimization model based on the shortest time path, ignoring the limited cognition of pedestrians in emergency situations and the random disturbances in the decision-making process. CN118551561A discloses a low-posture evacuation simulation method for fires. It introduces physical and psychological forces from the fire source into a social force model and combines low-posture force analysis to solve the trajectory of personnel. However, this method is still a local modification of the mechanical interaction mechanism, focusing on the rigid mapping between physiological posture and environmental constraints, and lacks the characterization of individual strategy learning and behavioral evolution in the process of interaction with the environment.

[0005] In recent years, with the development of computing power and intelligent algorithms, reinforcement learning technology has been gradually introduced into crowd evacuation simulation research. Unlike traditional models based on fixed rules or parameters, reinforcement learning constructs an interaction process between the environment and individuals, enabling evacuated individuals to gradually form behavioral strategies based on feedback information during continuous interaction with the environment. This type of method does not rely on pre-set behavioral patterns, but guides the evolution of individual behavior through reward or constraint mechanisms. It can generate multiple potential individual behavioral strategies and collective evacuation patterns in the same evacuation environment, thereby alleviating the dependence of traditional models on existing empirical rules to a certain extent. However, existing learning-based evacuation models still have significant limitations in behavioral selection mechanisms: First, the description of individual cognitive constraints, risk judgment, and decision uncertainty in the behavior generation process is still insufficient, leading to deviations between the obtained strategies and the real crowd decision-making process at the behavioral mechanism level; Second, the action selection stage often adopts a relatively direct random exploration mechanism, which is prone to problems such as insufficient expression of randomness or unreasonable random distribution.

[0006] In summary, existing personnel evacuation simulation technologies still have significant shortcomings in uniformly characterizing the rationality of individual decision-making, the randomness of behavior, and the evolutionary process of group patterns in complex emergency scenarios. Therefore, how to reasonably represent the cognitive constraints, behavioral diversity, and decision-making uncertainty of evacuation individuals within a learning-based evacuation modeling framework is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] (a) Purpose of the invention The purpose of this invention is to address the problem that existing evacuation crowd simulation methods struggle to simultaneously characterize the rationality of individual decision-making and the randomness of behavior. Addressing the shortcomings of traditional evacuation crowd simulation models, such as rule and parameter constraints on simulated behavior patterns, tendency to converge to a single evacuation pattern, and insufficient randomness in the behavior selection phase of existing learning-based methods, this invention proposes a personnel evacuation simulation method and system based on multi-agent reinforcement learning. Within the framework of evacuation behavior modeling and reinforcement learning, a fuzzy stochastic decision-making mechanism consistent with human cognitive characteristics is introduced. Fuzzy mathematics is used to improve the reward function and action sampling method of the evacuation agents, enabling individual behavior to exhibit appropriate fuzziness and randomness while satisfying the rationality of evacuation behavior. This method can generate more diverse individual behaviors and collective evacuation patterns under the same evacuation environment and conditions, simulating the diversity and uncertainty of crowd behavior. It more realistically reflects the diversity, uncertainty, and evolutionary characteristics of crowd behavior in actual emergency situations, providing technical support for simulation analysis and safety assessment in complex evacuation scenarios.

[0008] (II) Technical Solution To achieve the objective of this invention and solve its technical problems, the present invention adopts the following technical solution: The first objective of this invention is to provide a method for simulating evacuation based on multi-agent reinforcement learning, used to model, train, and simulate the behavior of multiple evacuation individuals in a preset evacuation scenario, and output evacuation routes, evacuation times, and congestion distribution results, comprising at least the following steps: SS1. Evacuation Environment and Scene Modeling: Discrete parameterized modeling of the simulated evacuation environment is performed, defining the evacuation space boundary, obstacles, passable areas and evacuation exits, generating the initial positions and target exits of multiple evacuation agents, and establishing the mapping relationship between physical space coordinates and image pixel coordinates, providing a foundation for subsequent visualization and deep reinforcement learning models based on convolutional neural networks. SS2. Agent Perception and State Representation: The static obstacle layout of the evacuation environment and the dynamic position of each evacuation agent are mapped into a global dynamic scene image. Each evacuation agent is distinguished and a local observation area is cropped with its own position as the center. After resampling and normalization, the state input is formed. SS3. Action Space Construction: Construct a discrete action set for each evacuation agent, which includes a stationary action and multiple motion directions. Map the selected discrete action to the preferred velocity vector. Combine the maximum velocity constraint and collision avoidance algorithm to determine the safe velocity in the feasible velocity domain and perform displacement update. SS4. Evacuation Reward and Constraint Feedback: After fuzzing, rule activation, aggregation and defuzzing the three types of state variables, local density, motion smoothness and evacuation time, a continuous reward is generated. The distance improvement reward based on the optimal goal is calculated simultaneously. The two types of rewards are integrated to form an instant reward. SS5. Multi-agent collaborative training and learning: Construct a deep reinforcement learning framework based on convolutional neural networks. Each sparse agent learns collaboratively by sharing network parameters and centralized experience replay, and updates the policy network and target network. SS6. Fuzzy Constraint-Based Action Sampling: Based on the discrete action values ​​output by the policy network, a fuzzy membership degree of evacuated acceptable actions is constructed, and it is coupled with and normalized with random diffusion weights based on action values ​​to obtain the selection probability of each discrete action and complete the action sampling accordingly. SS7. Evacuation Process Evolution and Result Output: Drive each evacuation agent to iteratively execute the sampled actions until the preset termination condition is reached, and output the individual trajectory, individual evacuation time, total evacuation time, evacuation flow rate and exit neighborhood density distribution.

[0009] The second objective of this invention is to provide a personnel evacuation simulation system, comprising multiple modules, for executing the personnel evacuation simulation method based on multi-agent reinforcement learning described above.

[0010] (III) Technical Effects Compared with the prior art, the personnel evacuation simulation method and system based on multi-agent reinforcement learning of the present invention has the following beneficial and significant technical effects: (1) This invention can improve the realism of evacuation behavior simulation by introducing controllable randomness while ensuring the rationality of evacuation behavior. This invention introduces fuzzy mathematics methods in the reward generation stage and proposes action sampling rules based on fuzzy constraints in the action selection stage, thus modeling the randomness of evacuation behavior as a continuous fuzzy set. This mechanism allows the agent to randomly diffuse only within the set of actions that satisfy the basic rationality constraints of evacuation, thereby avoiding unreasonable behavior caused by completely random exploration. Therefore, the result of this invention is that, under the same evacuation scenario and initial conditions, it can generate more diverse individual trajectories and collective evacuation patterns that all conform to the evacuation goals, making the simulation results more statistically close to the behavioral distribution characteristics of real population evacuation. This feature enables this invention to maintain good generalization ability when facing complex building environments or evacuation scenarios that are not fully observed, and it is suitable for application scenarios such as evacuation risk assessment, extreme situation analysis, and emergency management decision support.

[0011] (2) This invention can effectively alleviate the action oscillation problem caused by sparse rewards in reinforcement learning evacuation modeling. In multi-exit evacuation environments, traditional distance-based reward functions tend to cause agents to frequently switch between multiple targets, resulting in path oscillations or policy instability, which affects training convergence and result interpretability. This invention introduces nonlinear distance-based reward improvement based on the optimal target in the reward design and weights the target distance by importance, making the agent more inclined to continuously advance towards a certain target exit over a period of time. This design maintains policy flexibility while suppressing frequent decision reversals caused by short-term value differences. In multi-exit evacuation scenarios, this reward design can significantly reduce the frequency of agents repeatedly switching between adjacent exits, reduce the fluctuation amplitude of Q-values ​​during the learning process, and make the training process more convergent, thereby improving simulation efficiency and stability. Attached Figure Description

[0012] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood in conjunction with the following description of the embodiments, in which: Figure 1 This is a flowchart of the personnel evacuation simulation method based on multi-agent reinforcement learning of the present invention; Figure 2 This is a schematic diagram of the multi-agent reinforcement learning network structure of the present invention; Figure 3 This is a simulated room background image of Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of evacuation behavior simulated by the present invention, wherein: (a) represents pedestrian queuing behavior, which is more rational; and (b) represents bottleneck behavior, which is more random. Detailed Implementation

[0013] This invention aims to provide a method and system for simulating personnel evacuation based on multi-agent reinforcement learning. It is used to model, train, and simulate the behavior of multiple evacuation individuals in a preset evacuation scenario, and output evacuation routes, evacuation times, and congestion distribution results. To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments will be described in more detail below with reference to the accompanying drawings. The described embodiments are some, but not all, embodiments of this invention, and are exemplary, intended to explain the invention, and should not be construed as limiting the invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0014] Example 1: Personnel Evacuation Simulation Method like Figure 1 As shown in the embodiments of the present invention, the personnel evacuation simulation method based on multi-agent reinforcement learning models the evacuation process as a continuous decision-making process of multiple agents in a confined building environment. By constructing a closed-loop mechanism of perception-decision-execution-feedback, evacuation individuals can randomly choose among a variety of acceptable behaviors under the premise of meeting basic evacuation rationality constraints, thereby generating a collective evacuation behavior pattern with diversity and uncertainty. The method mainly includes the following steps in implementation: evacuation environment and scenario modeling, agent perception and state representation, behavior action space construction, evacuation reward and constraint feedback, multi-agent collaborative training and learning, action sampling based on fuzzy constraints, evacuation process evolution and result output. Each step works together to achieve a systematic characterization of the diversity and uncertainty of crowd behavior in complex evacuation situations.

[0015] SS1. Evacuation Environment and Scene Modeling: Discrete parameterized modeling is performed on the simulated evacuation environment, defining evacuation space boundaries, obstacles, passable areas, and evacuation exits. Initial positions and target exits for multiple evacuation agents are generated, and a mapping relationship between physical space coordinates and image pixel coordinates is established, providing a foundation for subsequent visualization and deep reinforcement learning models based on convolutional neural networks. Step SS1 is used to construct a simulation environment for agents to evacuate. Its workflow involves the following steps to construct the physical scene: S110. Global Coordinate System Construction: Based on the topological information of the building or evacuation space to be simulated, a unified global coordinate system is established in two-dimensional or three-dimensional space as the reference coordinate system for geometric modeling of the evacuation space. The evacuation environment is then discretized and modeled within this coordinate system. The evacuation environment includes, but is not limited to, geometric elements such as room boundaries, passable areas, exit locations and dimensions, and internal fixed obstacles.

[0016] S111. Define the Main Region and Walls: Define a passable main region centered at the origin in the global coordinate system, with a width of W and a height of H. Construct boundary walls with a preset width of W, height of H, and wall thickness D based on the topological information of the building or evacuation space to be simulated, thus limiting the activity range and environmental boundary conditions of the evacuation agents. The boundary walls can be described as enclosed by rectangles, polylines, or closed boundaries formed by connecting multiple line segments end to end. For building spaces with locally impassable recessed areas, additional restrictive boundaries can be defined within the main region. Obstacles can be represented as fixed wall piers, facilities and equipment, fences, or other non-passable objects that cause people to detour or experience local congestion. Exits can be further assigned width, location, direction, and target importance parameters to facilitate the subsequent construction of target distance improvement rewards.

[0017] S112. Define exits and obstacles: Based on the topological information of the building or evacuation space to be simulated, open one or more independent evacuation exits with preset location and width parameters, and configure obstacles in the main passable area to form an evacuation scenario to be simulated that includes passage constraints.

[0018] After completing the environmental geometry modeling, an initial evacuation scenario is generated within the coordinate system according to the evacuation scenario requirements. Specifically, based on the preset crowd size and distribution rules, the initial position coordinates of individuals within the passable area are generated randomly or distributed according to given locations, and each evacuation individual is assigned an ID. To avoid situations where agents overlap with walls or obstacles or where agents overlap excessively during the initialization phase, minimum spacing constraints and boundary feasibility checks can be further applied during initial position generation.

[0019] Specifically, the module executes the following definition process to initialize and deploy the agent: S120. Determine the initial generation area: Within the main passable area, based on the purpose of the building to be simulated and the distribution characteristics of the crowd, delineate the initialization area for placing the intelligent agent.

[0020] S121. Crowd Size and Identity Configuration: Generate multiple evacuation agents based on the preset crowd size, and assign a unique identity ID to each evacuation agent for indexing agent status, actions, rewards, trajectories and statistical results during training.

[0021] S122. Generate Initial Positions: According to preset distribution rules, a two-dimensional coordinate point (x, y) is randomly generated within the initialization area for each agent to be simulated, serving as its initial position. This ensures that all evacuation agents enter a unified scene and form an initial crowd distribution state at the start of the simulation. During deployment, initial positions can be generated through random sampling or sampling based on a given density field. If a candidate position does not meet boundary constraints, obstacle occupancy constraints, or minimum neighbor constraints, resampling can be performed until a valid position is obtained.

[0022] S123. Associating Target Exits: Assign one or more initial target exit coordinates to each agent, enabling it to have target pointing information corresponding to the evacuation exit at the initial moment, and allowing the agent to adjust the target exit selection according to environmental changes and evacuation rules during subsequent scenario evolution. The agent will also change its target selection accordingly based on scenario changes and evacuation rules.

[0023] S124. Coordinate Mapping: Establish a coordinate mapping system to linearly map the physical space coordinates (unit: meters) defined above to the screen pixel coordinates used for graphics rendering, thus providing a foundation for subsequent visualization and deep reinforcement learning models based on convolutional neural networks. The physical space is mainly used for position updates, distance calculations, velocity clipping, and evacuation statistics, while the pixel space is mainly used for global dynamic scene image generation, local observation region clipping, and convolutional neural network state input. The two maintain a strict correspondence through the mapping relationship.

[0024] SS2. Agent Perception and State Representation: The static obstacle layout of the evacuation environment is mapped to the dynamic positions of each evacuation agent to form a global dynamic scene image. Each evacuation agent is distinguished, and its local observation region is cropped centered on its own position. After resampling and normalization, this becomes the state input. This step transforms the global physical state of the simulated environment into a set of egocentric local observations usable by each agent's decision-making model; this is the state space input to the subsequent deep reinforcement learning network. The core function is to generate a two-dimensional raster image I. i ∈R H×W H and W serve as the state input for agent i, respectively, representing the height and width of the state image. By employing an egocentric local state representation approach, the geometric relationships of the neighborhood environment and the distribution relationships of neighboring individuals can be preserved while mitigating the problems of excessive dimensionality and redundant information caused by direct input of the global state.

[0025] Specifically, mapping the static obstacle layout of the evacuation environment to the dynamic positions of each evacuation agent into a global dynamic scene image includes at least the following sub-steps: S210. Reference Image Loading: First, load the reference image I, which represents the static obstacle layout of the evacuation environment and was pre-generated in step SS1. base Its image size is W. world × Height H world Pixels. In I base In the image, the obstacle area is filled with a grayscale value of C. wall The passable area is represented by the grayscale value C. free .

[0026] S211. Pixel coordinate mapping: For each agent j currently existing in the simulation environment (totaling N), agent The module sets its two-dimensional position coordinates p in the world coordinate system. j =(x j ,y j Mapping to the image pixel coordinate system, this mapping is achieved through a linear transformation function f. map-to-creen (·)accomplish: (u j ,v j )=f map-to-creen (p j ;extent,W world H world ) Where extent=(x min ,y min ,x max ,y max This defines the boundary correspondence between world coordinates and image pixel coordinates, x min x max Let y be the minimum and maximum values ​​of the x-coordinate in the world coordinate system. max y min x represents the minimum and maximum values ​​of the ordinate in the world coordinate system; j y j U represents the x and y coordinates of the evacuation agent j in the world coordinate system. j v j The x and y coordinates of the evacuation agent j in the image pixel coordinate system are represented.

[0027] S212. Agent position rendering: In the reference image I base The pixel coordinates (u) obtained by mapping each evacuation agent j on the replica image. j ,v j With r as the center, j =R j / s scale To plot the radius (where R) j Let s be the physical radius of the agent. scale(Scaling from world units to pixel units), using grayscale value C agent Draw a solid circle. This step generates a global dynamic binary image I containing the positional information of all agents. global .

[0028] S213. Global Dynamic Scene Generation: Define the image after rendering the locations of all evacuated agents as the global dynamic scene I. global This binary image simultaneously contains static obstacle layout information and the dynamic position information of all evacuation agents at the current time step. During training, I global It is continuously refreshed over time, and its update frequency is consistent with the environmental state update frequency.

[0029] When differentiating and cropping the local observation area for each evacuation agent, at least the following should be included: S220. Feature Agent Highlighting and Differentiation: For a feature agent i in the state space to be acquired, create a global dynamic scene image I. global copy In this copy, the solid circle corresponding to agent i itself is moved from C... agent Change to a preset highlight color C self This operation aims to distinguish the agent itself from other neighboring agents and obstacles in the state space image.

[0030] S221. Boundary padding: processing global dynamic scene image copies Boundary filling is performed so that the target evacuation agent can still obtain a complete local observation area when it is close to the edge of the image.

[0031] S222. Self-center clipping: Clipping based on the pixel coordinates (u) of the target agent i itself. i ,v i Centered on the image after boundary padding, Cut out a piece with a side length of L from the middle. crop square area This step defines the agent's observation space. It is used to characterize the distribution of obstacles, the distribution of adjacent evacuation agents, and the positional relationship of itself in a local space centered on itself. In reality, it generally corresponds to the range of human vision or the range of information received. The specific parameters are determined by the building environment and purpose to be simulated.

[0032] When resampling, normalizing, and forming state input for a local observation region, at least the following should be included: S230. Resampling: Resample the cropped local observation window image. The state image is obtained by resampling to a fixed target size H×W using a bilinear interpolation algorithm. This ensures that the tensor size input to subsequent models is uniform. Using bilinear interpolation helps maintain the continuity of spatial structure when compressing or enlarging local observation regions, and avoids the introduction of obvious false boundaries due to coarse sampling.

[0033] S231. Batch normalization: The set of observed images of all agents obtained after the previous resampling step. Perform batch normalization processing Each image is normalized to ensure that the input data distribution has zero mean and unit variance, where μ is the batch image mean, σ is the batch image standard deviation, and ε is a small positive number that prevents zero, which is beneficial to the stability and convergence speed of model training.

[0034] S232. State Tensor Assembly: Assemble all state images after normalization. Assembled according to the evacuation agent dimension, outputting a shape of (N) agent A three-dimensional tensor (H,W) encodes the binary spatial layout information of static obstacles, other agents, and the agent's own position within a certain perception range centered on the agent itself, thus constituting the state representation of agent i. It is used as input for subsequent deep reinforcement learning networks to estimate action value and update policies.

[0035] SS3. Construction of the Behavior and Action Space: For each evacuation agent, a discrete action set is constructed, comprising one stationary action and multiple motion directions. The selected discrete action is mapped to a preferred velocity vector. Combining maximum velocity constraints and collision avoidance algorithms, a safe velocity is determined within the feasible velocity domain, and displacement updates are performed. This step defines the set of discrete actions that the deep reinforcement learning model can output and provides a deterministic function that maps the discrete action to a continuous two-dimensional preferred velocity vector. This preferred velocity vector serves as a key input to the underlying collision avoidance physics simulator. Considering the local movement characteristics of individuals during evacuation and computational efficiency requirements, this invention restricts the individual's selectable motion directions to their Moore's neighborhood. By decoupling high-level discrete decision-making from low-level continuous velocity execution, it facilitates stable training of the reinforcement learning network within a limited action space and also allows for the introduction of a classic collision avoidance model to ensure the safety and physical rationality of displacement updates.

[0036] When constructing a discrete set of actions for each evacuation agent and completing the action indexing representation, it should include at least: S310. Definition of Discrete Action Direction Set: The set of directions is defined by the current position of the i-th evacuation agent at the current time step t. Centered on a point, construct a discrete set of motion directions consisting of 8 directions and 1 stationary motion, defined as A. i ={a0,a1,…,a8}, where A i Let a represent the discrete action set of the i-th evacuation agent, where a0 represents stationary actions and a1~a8 represent actions in the front, back, left, right and four diagonal directions.

[0037] S311. Action direction vector association: for a discrete action set A i Each action a in k Pre-associate a direction vector e k and satisfy And the static motion a0 corresponds to the zero vector e0=(0,0), where k=1,2,…,8. Represents the direction vector e k The length of the module.

[0038] S312. Action Index Space Establishment: In the deep reinforcement learning framework, the above effective action set A is... i The mapping is a discrete action index space k∈{0,1,…,8}. This allows the evacuation agent to make decisions based on the current observed state at each decision time. Select an action from the action set.

[0039] When mapping discrete motions to preferred velocity vectors and applying kinematic constraints, at least the following should be included: S320. Preferred velocity vector generation: For the discrete action selected by evacuation agent i at time step t Generate the preferred velocity vector according to the deterministic mapping function. g(·) represents the mapping function from discrete motion to continuous velocity, v i,max e represents the maximum velocity scalar of evacuation agent i. k This represents the direction vector corresponding to the selected action. When the selected action is a stationary action a0, .

[0040] S321. Single-step displacement upper limit constraint: Let the simulation step size be Δt, then the maximum allowable displacement d of the evacuation agent i in a single time step is... i,max Satisfy d i,max =v i,max Δt.

[0041] S322. Preferred motion expression can be executed: with preferred velocity vector As the target input of the underlying collision avoidance solver, and with v i,max Together with Δt, it limits the velocity amplitude and displacement range of the evacuation agent i at the current time step, so that the action space is consistent with the actual local movement capability of the evacuation agent.

[0042] The Reciprocal Velocity Obstacles (RVO) algorithm, a classic method in evacuation simulation, is introduced to prune velocities. For the impassable region surrounding agent i (environmental obstacles, other agents, etc., specifically determined by the building to be simulated), a set of velocity obstacles is constructed, and the safest velocity closest to the desired direction is selected within the defined feasible velocity domain. The action selected in the previous step is then executed to generate the agent's displacement update for the next time step. Specifically, when pruning the preferred velocity vector and performing displacement updates based on the RVO algorithm, at least the following steps are included: S330. Velocity Obstacle Construction: Construct an evacuation agent i at time step t based on the set of impassable objects around it. Speed ​​obstacle set Impassable objects include environmental obstacles and nearby evacuation agents. VO(i,o) represents the velocity obstacle region of evacuation agent i relative to object o.

[0043] S331. Feasible velocity domain determination: Construct the feasible velocity domain while satisfying the maximum velocity constraint. ,in Let v represent the feasible velocity domain of evacuation agent i at time step t, and v represent the candidate velocity vector. This represents the magnitude of the candidate velocity vector.

[0044] S332. Safe speed selection and displacement update: in the feasible speed domain Internal selection and preferred velocity vector closest safe speed And complete the position update according to the safe speed. ,in This represents the safe speed of evacuation agent i at time step t. This represents the position vector of the evacuation agent i at the next time step.

[0045] SS4. Evacuation Rewards and Constraints Feedback: After sequentially fuzzifying, activating rules, aggregating, and defuzzifying three types of state variables—local density, motion smoothness, and evacuation time—continuous rewards are generated. Simultaneously, distance improvement rewards based on the optimal goal are calculated, and the results are integrated to form an immediate reward. This step is one of the core innovations of this invention. Based on fuzzy mathematics theory, a multi-dimensional evaluation function is constructed to generate immediate reward signals for the agent during evacuation. This step simulates the reasoning and decision-making process of humans in emergency situations, transforming environmental state characteristics into continuous reward values, enabling the agent to exhibit more realistic evacuation behavior. Compared to methods that only use distance reduction or whether an exit has been reached as rewards, this step simultaneously introduces three types of variables: congestion level, directional smoothness, and evacuation urgency. This ensures that reward feedback not only guides the agent to continuously move towards the exit but also suppresses obviously unreasonable frequent turns, blind squeezing into locally congested areas, and target oscillation behavior in multi-exit scenarios.

[0046] Specifically, the calculation of the nonlinear distance improvement reward based on the optimal goal, used to evaluate the agent's continuous improvement in moving towards the goal, includes at least: S410. Construction of Multi-Target Distance Set: For each agent i, given the set of distances from its current time step to each target point. The set of distances from the previous time step to each target point K represents the target number of exits, k=1,2,…,K. First, distance is weighted according to importance; this step reflects the agent's preference for each evacuation exit in the building. S411. Target Importance Weighting: Let r be the importance weight vector of each target exit. important =[r1,r2,…,r K ], where r k The importance weight of the k-th target exit is represented by k=1,2,…,K; the weight is obtained by weighting the distance sets between the current and previous time steps element by element. , in, This indicates an element-wise division operation. , These represent the weighted distance set of the current time step and the distance set of the previous time step, respectively. By introducing target importance weights, the priority, accessibility, or desired guidance direction of different exits in the evacuation organization can be differentiated and encoded.

[0047] S412. Calculate the distance improvement reward: Calculate the nonlinear distance improvement reward for evacuation agent i based on the weighted distance set. ,in τ represents the distance improvement reward for evacuation agent i at time step t, w1 represents the weighting coefficient of the distance improvement reward, and τ represents the distance improvement reward. d λ represents the distance-improved sensitivity adjustment parameter.

[0048] This reward function is designed based on the nonlinear distance to the optimal target and observations of real evacuation behavior. The agent receives a positive reward for consistently and effectively approaching a target, aiming to encourage a continued tendency to move towards the target exit. This is because, in real evacuation, pedestrians often continuously move towards a particular exit. Using traditional distance-based reward reduction methods could cause the agent to oscillate between two exits that are close to each other, negatively impacting training effectiveness. The exponential τ... d λ is used to adjust the sensitivity of distance improvement, and the importance weights allow for differentiated evaluation of different objectives.

[0049] When fuzzing local density variables, motion smoothness variables, and time variables, at least the following should be included: S420. State Variable Acquisition: Acquire the local density variables of evacuated agent i at time step t. Motion smoothness variables and the evacuation time variable T t , This represents the number of evacuated individuals within a unit area surrounding evacuation agent i. T represents the angle between the preferred direction of movement in the next step and the actual direction of movement in the previous step. t This represents the cumulative elapsed time from the start of the evacuation to the current time step t. S421. Fuzzy set partitioning: for local density variables The number of pedestrians per unit area around the agent is divided into three fuzzy sets: low, medium, and high; for the motion smoothness variable... The angle between the agent's next movement direction and the previous movement direction is divided into three fuzzy sets in the universe of discourse: smaller, general, and larger; for the time variable T t The time elapsed since the start of evacuation is used to assess the level of urgency, and the domain of discourse is divided into three fuzzy sets: safe, general, and dangerous.

[0050] S422. Quantization of motion smoothness: ... Represented as ,in Let represent the preferred velocity vector of evacuation agent i at time step t. This represents its actual execution speed vector at the previous time step. To prevent extremely small positive numbers with a denominator of zero.

[0051] S423. Construction of Symmetric Triangular Membership Functions: For each of the above input variables , T t A corresponding fuzzy set is predefined, and the input variables are fuzzified using a symmetric triangular membership function, where the symmetric triangular membership function is... Let a, b, and c represent the left endpoint, vertex, and right endpoint of the trigonometric membership function, respectively. Let the time variable T... t For example, its fuzzy set includes three categories: "safe," "general," and "dangerous." For any time variable value T... t Each time point is assigned a membership degree to each fuzzy set, which is used to describe the degree to which the time point belongs to multiple semantic states simultaneously.

[0052] S424. Membership Mapping: Mapping each input variable to multiple membership values, resulting in... In this context, superscripts L1, M1, and H1 represent low, medium, and high, respectively; superscripts S2, M2, and L2 represent relatively small, moderate, and relatively large, respectively; and superscripts S3, M3, and D3 represent safe, moderate, and dangerous, respectively.

[0053] In this way, the original precise numerical input is converted into multiple membership values, thereby realizing the expression of the uncertainty and continuity of the environmental state.

[0054] When performing rule activation, aggregation, and defuzzification based on a fuzzy rule base, at least the following should be included: S430. Fuzzy Rule Base Construction: A set of fuzzy rule bases is pre-constructed, generated based on the safety and efficiency principles of evacuation scenarios. Each rule in the rule base adopts the following general form: IF (density state) AND (smoothness state) AND (time state) THEN (R reward status) Where R represents the fuzzy set of rewards.

[0055] S431. Rule Activation Strength Calculation: At the current time step t, perform activation calculation for each rule in the rule base, and obtain the membership degree corresponding to each fuzzy set in the rule's preconditions; calculate the activation strength of the rule using the minimum operator. The activation strength is used to indicate the applicability of the rule under the current environmental conditions. This represents the activation strength of the m-th rule at time step t, where m = 1, 2, ..., M, and M represents the total number of fuzzy rules. These represent the membership degrees of the fuzzy sets corresponding to the local density, motion smoothness, and time variables in the preconditions of the rule, respectively.

[0056] Therefore, this invention does not select a single rule for judgment, but allows multiple rules to participate in the reward decision at the same time step to simulate the uncertainty of real pedestrian evacuation.

[0057] S432. Reward Fuzzy Set Aggregation: For all activated fuzzy rules, the corresponding reward fuzzy sets in the rule conclusions are weighted according to their activation strength, and then aggregated using the maximum operator to obtain a comprehensive output fuzzy reward set. , where y represents a discrete reward point in the reward universe. Let represent the membership degree of the fuzzy set of rewards corresponding to the conclusion of the m-th rule at the reward point y. This represents the aggregated output fuzzy reward set. This output fuzzy set will form a continuous distribution over the reward universe, used to represent the overall evaluation result of the reward bias in the current state.

[0058] S433. Centroid Defuzzification: After obtaining the fuzzy reward set, a centroid defuzzification strategy is used to convert the fuzzy set into a single precise reward value. ,in Let Y represent the fuzzy reward value of evacuation agent i at time step t, and let Y represent the discretized reward universe set. R This represents a very small positive number to prevent the denominator from being zero. Specifically, by discretizing the reward universe, a weighted average of the membership degrees corresponding to each reward value is calculated to obtain the immediate reward value at the current time step. This reward value is a continuous real number and can be directly used as the immediate reward signal in reinforcement learning algorithms.

[0059] After the aforementioned fuzzy reward is generated, the module further integrates this reward with the sparse event reward during the evacuation process, which in this invention is the distance improvement reward, to obtain the final reward for each action. Specifically, when integrating the fuzzy reward and the distance improvement reward to form instant reward feedback, it includes at least: S440. Reward Integration: Integrating the fuzzy reward value at time step t Distance Improvement Rewards By integrating the results, we can obtain the final instantaneous reward of the evacuated agent i at time step t. .

[0060] S441. Motion Feedback Association: [This will be used in conjunction with other methods] As an evacuation agent i, in state Next action The feedback signal after that, and the state at the next time step. Together they constitute the experience of state transition.

[0061] S442. Reward Output: Output the immediate reward to the multi-agent collaborative training and learning module for subsequent action value estimation, experience playback storage, and network parameter updates.

[0062] SS5. Multi-agent collaborative training and learning: A deep reinforcement learning framework based on convolutional neural networks is constructed, in which evacuation agents collaboratively learn by sharing network parameters and centralized experience replay, and update the policy network and target network. This step implements a multi-agent deep reinforcement learning training framework based on convolutional neural networks. Through sharing neural network parameters, centralized experience replay, and collaborative learning mechanisms, multiple agents learn collaborative policies in evacuation scenarios. The key to this step is that although the multiple evacuation agents are in different local states, they share the same decision network parameters, thereby learning generalized evacuation behavior patterns in a unified policy space and reducing parameter size and training resource consumption.

[0063] Specifically, this step performs the following defined process for multi-agent collaborative training, such as... Figure 2 As shown: S510. The module uses a deep convolutional neural network (CNN) to process the agent's visual state input. This CNN network consists of the following key components: a. Feature Extraction Backbone Network: Given an input image size of 84×84, the network comprises three convolutional layers specifically designed for processing 84×84 grayscale images. The first convolutional layer uses 32 8×8 convolutional kernels with a stride of 4 to extract low-level spatial features from the input channels. The second convolutional layer uses 64 4×4 convolutional kernels with a stride of 2 to capture medium-scale spatial patterns. The third convolutional layer uses 64 3×3 convolutional kernels with a stride of 1 to extract high-level abstract features. Batch normalization and ReLU activation functions are applied after each convolutional layer to extract multi-level state representations from low-level spatial features to high-level abstract features.

[0064] b. Feature Flattening and Fully Connected Layers: The 3D feature map output from the convolutional layers is flattened into a 1D feature vector, which is then processed through two fully connected layers. The network employs a Dueling architecture, feeding the feature vectors into the value stream and the dominance stream, respectively. The value stream outputs a single-state value estimate, while the dominance stream outputs the dominance value for each action.

[0065] c. Noise injection mechanism: The fully connected layer uses a noisy linear layer instead of a traditional linear layer. The weights and biases of the noisy linear layer are decomposed into deterministic mean parameters and random noise parameters.

[0066] S520. The module creates two Deep Q Networks (Dueling Deep Q Networks) with the same structure as described above, i.e., the Dueling-DQN algorithm: the current policy network Q... θ and target network , where θ is a trainable parameter, θ － The target parameters are θ. All agents share a set of network parameters θ. The current policy network is used to output the value estimate of each discrete action in the current state, and the target network is used to calculate the relatively stable target Q value to reduce target drift during training.

[0067] S530. Construct a unified experience replay pool to store the transition experiences of all agents. The replay pool adopts a priority sampling mechanism.

[0068] During training, all agents work in parallel using their current local observations. As input, through the shared current policy network Q θ Calculate the value of each action and select actions using the fuzzy constraint diffusion rule from Module 6. Subsequently, all agents execute actions simultaneously, the environment returns the next state and reward, and the experience of each agent is stored in its corresponding queue. When conditions are met, multi-step rewards are calculated and stored in the shared replay pool. Next, when the number of experiences in the replay pool reaches a threshold, batch experiences are periodically sampled from the replay pool, the temporal difference loss is calculated, and the current network parameters θ are updated through backpropagation. Finally, the current network parameters θ are periodically copied to the target network parameters θ. － This achieves stable updates of the target Q value.

[0069] SS6. Action sampling based on fuzzy constraints: Based on the discrete action values ​​output by the policy network, fuzzy membership degrees of acceptable evacuation actions are constructed. These degrees are then coupled with and normalized to obtain the selection probability of each discrete action, and action sampling is performed accordingly. This step is one of the core innovations of this invention, proposing an action sampling method called the Fuzzy-Constrained Diffusion Rule. This method integrates fuzzy mathematics theory, deterministic drift, and random diffusion mechanisms to generate a decision-making strategy in the discrete action space that considers both value orientation and exploratory robustness. Importantly, this method's action calculation module can introduce randomness that conforms to the actual evacuation behavior of the agent, adaptable to the simulated building, while ensuring that the agent's action selection is realistic, in order to discover new stable evacuation patterns.

[0070] Specifically, when constructing the fuzzy membership degree of evacuation-acceptable actions based on the discrete action values ​​output by the policy network, it should include at least: S610. Action Value Acquisition: For the current state of evacuated agent i at time step t Obtain its discrete action set A i ={a0,a1,…, Value estimation for each discrete action in} ,in This indicates that the evacuation agent i performs the k-th discrete action a at time step t. k Action value estimation, Q represents the total number of discrete actions. θ This represents the current policy network with parameter θ.

[0071] S611. Determining the maximum action value: Determine the maximum action value in the current state based on the action value estimate. ,in The maximum action value of evacuation agent i at time step t is represented by the reference point for the optimal action in the value-oriented sense under the current state. These value estimates constitute the deterministic drift term of action selection.

[0072] S612. Construction of the Fuzzy Set of Acceptable Actions: Define the Q-value fuzzy membership function. This step innovatively introduces a fuzzy concept of evacuation acceptable actions, defining the fuzzy set A. acceptable And the degree to which the k-th discrete action belongs to the fuzzy set is denoted as Its membership function This indicates that the evacuation agent i performs action a at time step t. k The acceptable level of action; S613. Calculation of exponential fuzzy membership degree: Membership degree Calculated using the following exponential decay function: Here, δ>0 is a key hyperparameter called the diffusion gate, which controls the degree of boundary fuzziness of the fuzzy set. When δ is large, even actions with values ​​significantly different from the maximum value have high membership, indicating stronger exploratory behavior; when δ is small, only actions with values ​​close to the maximum value are considered acceptable, and the strategy is more greedy. Membership calculation defines a hyperparameter denoted by Q. max The exponential membership function with δ as the decay parameter, taking δ as the reference point.

[0073] When constructing random diffusion components based on action values ​​and obtaining unnormalized selection weights, at least the following should be included: S620. Construction of Random Diffusion Logarithmic Weights: A softmax-based stochastic process is introduced to generate unnormalized logits values ​​for each action of the evacuated agent i at time step t: The logarithmic weight of the k-th discrete action is represented by τ>0, which is a temperature parameter that controls the intensity of randomness. S621. Numerical stabilization: To achieve numerical stability, the logarithmic weights of logits are adjusted. Standardization process is required to obtain , This represents the standardized logarithmic weights after stabilization.

[0074] S622. Calculation of random diffusion components: Subsequently, the exponential values ​​of the random diffusion components for each discrete action are calculated based on the standardized logarithmic weights. , Indicates action a k The random diffusion component at time step t.

[0075] S623. Unnormalized selection weight calculation: Finally, calculate the fuzzy membership degrees. With random diffusion components Multiply to obtain the unnormalized selection weight for each action. , Indicates action a k The unnormalized selection weights, with the fuzzy membership degree serving as a fuzzy constraint gate, filter and weight the random diffusion process, so that actions with lower acceptability are significantly suppressed in subsequent sampling.

[0076] μ k As a fuzzy constraint gate, it filters or weights random diffusion processes. Only those belonging to the acceptable action fuzzy set (i.e., those with high μ) are considered. k Actions with a value of μ will have their random diffusion components significantly preserved; while actions that are clearly unacceptable (μ) will have their random diffusion components significantly preserved. k (Approaching 0) is almost completely suppressed.

[0077] When normalizing unnormalized weights and generating action selection probabilities, at least the following should be included: S630. Construction of Normalized Denominator: Summing the unnormalized weights of all discrete actions of evacuated agent i at time step t to construct the normalized denominator. ε>0 is a very small positive number, which prevents division by zero errors.

[0078] S631. Action selection probability calculation: Calculate the final selection probability of each action based on the normalized denominator. , This indicates that the evacuation agent i selects action a at time step t. k The probability of.

[0079] S632. Probability constraint satisfied: The final selection probability satisfies... and This makes all discrete actions form a complete probability distribution; S633. Probability Distribution Output: Outputs the action selection probability distribution. The output is sent to the action sampling execution unit as the basis for action sampling at the current time step.

[0080] When performing action sampling and outputting the sampling results based on action selection probability, at least the following should be included: S640. Multinomial sampling execution: Select a probability distribution based on the action of the generated evacuation agent i at time step t. Multinomial sampling is used to select the action for each agent at the current time step. ,satisfy .

[0081] S641. Discrete Action Determination: Based on Determine the discrete action at the current time step .

[0082] S642. Action Output Association: Associate discrete actions The output is sent to the preferred velocity mapping and safe velocity pruning process in step SS3 to generate the corresponding preferred velocity vector and safe execution velocity; thereby enabling a closed-loop connection between high-level policy sampling and low-level physical execution.

[0083] S643. Sampling Mechanism Constraint Effect: The action value orientation term, fuzzy acceptable constraint term, and random diffusion term are applied together to the action sampling process, so that the action selection retains adjustable randomness only within the acceptable range of actions that meet the basic requirements of evacuation rationality.

[0084] SS7. Evacuation Process Evolution and Result Output: The module drives each evacuation agent to iteratively execute sampled actions until a preset termination condition is reached, and outputs individual trajectories, individual evacuation times, total evacuation times, evacuation flow rates, and density distribution in the exit neighborhood. When the system evolves to the termination condition (all individuals complete evacuation or the maximum simulation time is reached), this module summarizes and outputs the evacuation results. Specifically, the module executes the following definition process to construct its action space: outputs the motion trajectory and evacuation time of each individual; outputs the total evacuation time; outputs the evacuation flow rate curve; outputs the local density distribution in the exit neighborhood; and outputs the overall evacuation visualization process.

[0085] Preferably, in step SS7, the preset termination condition is that all evacuation agents complete the evacuation and / or the simulation time reaches the maximum preset duration; when the preset termination condition is met, the motion trajectory of each evacuation agent, individual evacuation time, total evacuation time, evacuation flow rate curve, local density distribution in the exit neighborhood, and the overall evacuation visualization process are uniformly summarized and output to form a comprehensive evaluation result of the spatiotemporal evolution characteristics of the evacuation process.

[0086] Example 2: Application Case Based on the above embodiment 1, this embodiment 2 further provides an application example based on a single-exit room scenario to verify the applicability of the personnel evacuation simulation method based on multi-agent reinforcement learning of the present invention in typical confined spaces, as well as its ability to represent different evacuation behavior patterns.

[0087] like Figure 3 As shown, this embodiment simulates a single-exit room, 50m long and wide, with the initial positions of the crowd randomly distributed. A total of 30 agents are evenly distributed throughout the room, tasked with rapidly evacuating the room. No additional fixed obstacles are placed within this scenario to highlight behavioral evolution phenomena such as crowd convergence, queuing, competition for passage, and localized congestion within the exit neighborhood under the same geometric boundary conditions. Specific parameters are shown in Table 1.

[0088] After parameter settings were completed, the multi-agent reinforcement learning training framework began operation. Each evacuation agent input its local observation state into a convolutional neural network to obtain an action value estimate for each discrete action. This estimate, combined with local density, motion smoothness, evacuation time, and improvement in target distance, generated an immediate reward. Subsequently, a fuzzy constraint-based action sampling mechanism was employed to generate discrete actions for each agent at the current time step, while ensuring the basic rationality of evacuation behavior. A mutual velocity barrier method was used to safely prune the preferred velocity, ultimately obtaining the actual execution speed and updating the individual's position. This process was iterated until all evacuation agents completed the evacuation or the maximum simulation duration was reached.

[0089] Table 1 Summary of parameters for reinforcement learning evacuation motion model The final simulated evacuation effect diagram is as follows Figure 4 As shown, by adjusting the randomness parameters of the crowd in the model (such as δ, τ), the evacuation patterns of the crowd exhibit rational queuing and rapid, orderly evacuation (such as...). Figure 4 (a) Most evacuation agents can move relatively stably towards the exit, gradually forming a queue with consistent direction and clear order before the exit. At this time, individual path deflection is relatively small, exit utilization is relatively continuous, and the overall evacuation process exhibits good orderliness and high passage efficiency. (and congestion at bottlenecks, such as...) Figure 4 (b) The lateral disturbances of evacuation agents in the exit neighborhood are more obvious, the local competition is stronger, and they are more likely to form high-density aggregation and blockage in front of the bottleneck. Some individuals have greater turning and cross movements, and the overall evacuation process shows stronger uncertainty and volatility.

[0090] As the results above demonstrate, the method of this invention can not only reproduce queuing behavior in single-exit scenarios within a unified modeling framework, but also further characterize congestion, competition, and local disorder in bottleneck areas by adjusting fuzzy constraint action sampling parameters. This indicates that the method of this invention can effectively simulate the random disturbances, cognitive differences, and group evolution characteristics widely present in real evacuation behavior while maintaining goal orientation.

[0091] The objectives of this invention have been fully and effectively achieved through the above embodiments. Those skilled in the art will understand that this invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments described above. Although the invention has been described with reference to what is currently considered the most practical and preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments, and any modifications that do not depart from the functional and structural principles of the invention will be included within the scope of the claims.

Claims

1. A method for simulating personnel evacuation based on multi-agent reinforcement learning, characterized in that, It should include at least the following steps: SS1. Perform discrete parameterized modeling of the simulated evacuation environment, define the evacuation space boundary, obstacles, passable areas and evacuation exits, generate the initial positions and target exits of multiple evacuation agents and establish the mapping relationship between physical space coordinates and image pixel coordinates; SS2. Map the static obstacle layout of the evacuation environment and the dynamic position of each evacuation agent to a global dynamic scene image. Distinguish each evacuation agent and crop the local observation area with its own position as the center. After resampling and normalization, form the state input. SS3. For each evacuation agent, construct a set of discrete actions containing one stationary action and multiple motion directions. Map the selected discrete actions to the preferred velocity vector. Combine the maximum velocity constraint and collision avoidance algorithm to determine the safe velocity within the feasible velocity domain and perform displacement update. SS4. After sequentially performing fuzzing, rule activation, aggregation, and defuzzing on three types of state variables—local density, motion smoothness, and evacuation time—a continuous reward is generated. Simultaneously, the distance improvement reward based on the optimal goal is calculated. The two types of rewards are integrated to form an instant reward. SS5. Construct a deep reinforcement learning framework based on convolutional neural networks, in which each evacuated agent learns collaboratively by sharing network parameters and centralized experience replay, and updates the policy network and target network. SS6. Construct fuzzy membership degrees of evacuation acceptable actions based on the discrete action values ​​output by the policy network, and couple and normalize them with random diffusion weights based on action values ​​to obtain the selection probability of each discrete action and complete action sampling accordingly. SS7. Drive each evacuation agent to iteratively execute the sampled actions until the preset termination condition is reached, and output the individual trajectory, individual evacuation time, total evacuation time, evacuation flow rate and exit neighborhood density distribution.

2. The method according to claim 1, characterized in that, In step SS1, when performing discrete parameterized modeling of the evacuation environment, at least the following sub-steps are included: S110. Global coordinate system construction: Based on the topological information of the building or evacuation space to be simulated, establish a unified global coordinate system in two-dimensional or three-dimensional space; S111. Definition of Main Area and Walls: Define the main area that can be accessed inside the building or evacuation space to be simulated, and construct the outer boundary walls of the main area based on the preset width, height and wall thickness parameters; S112. Exit and obstacle configuration: Based on the topological information of the building or evacuation space to be simulated, set one or more evacuation exits with preset location and size parameters on the boundary wall, and configure obstacles in the passable main area to form an evacuation scenario to be simulated that includes passage constraints. The initial deployment of evacuation agents should include at least the following sub-steps: S120. Initialization Area Delineation: Within the main passable area, based on the purpose of the building or evacuation space to be simulated and the characteristics of crowd dispersal, delineate the initialization area for deploying evacuation agents. S121. Crowd Size and Identity Configuration: Generate multiple evacuation agents based on the preset crowd size, and assign a unique identity to each evacuation agent; S122. Initial position generation: According to the preset distribution rules, generate two-dimensional initial position coordinates for each evacuation agent in the initialization area, so that each evacuation agent enters the unified scene at the start of the simulation and forms the initial crowd distribution state. S123. Target Exit Association: Assign one or more initial target exit coordinates to each evacuation agent, so that it has target pointing information corresponding to the evacuation exit at the initial moment.

3. The method according to claim 1 or 2, characterized in that, In step SS2, when mapping the static obstacle layout of the evacuation environment and the dynamic positions of each evacuation agent to a global dynamic scene image, at least the following sub-steps are included: S210. Reference Image Loading: Load the reference image I used to characterize the static obstacle layout of the evacuation environment. base Its image size is width W world × Height H world The obstacle area is filled with a grayscale value C. wall The passable area is filled with a grayscale value of C. free ; S211. Pixel coordinate mapping: Mapping the two-dimensional position coordinates p of each evacuated agent j in the world coordinate system. j =(x j ,y j ) through linear transformation function (u j ,v j )=f map-to-creen (p j ;extent,W world H world Mapped to the image pixel coordinate system, where j=1,2,…N agent N agent x represents the total number of evacuated agents. j y j U represents the x and y coordinates of the evacuation agent j in the world coordinate system. j v j This represents the x and y coordinates of the evacuation agent j in the image pixel coordinate system, extent=(x min ,y min ,x max ,y max (x) represents the range corresponding to the world coordinate boundary and the image pixel boundary. min x max Let y be the minimum and maximum values ​​of the x-coordinate in the world coordinate system. max y min These are the minimum and maximum values ​​of the ordinate in the world coordinate system; S212. Agent position rendering: In the reference image I base On the replica image, the pixel coordinates (u) corresponding to each evacuation agent j are used. j ,v j With r as the center, j =R j / s scale Plot a grayscale value of C for the radius. agent A solid circle, R j To evacuate the radius of agent j in physical space, S scale Indicates the scaling ratio; S213. Global Dynamic Scene Generation: Define the image after rendering the locations of all evacuated agents as the global dynamic scene I. global .

4. The method according to claim 3, characterized in that, In step SS2, when distinguishing each evacuation agent and cropping the local observation area centered on its own position, at least the following is included: S220. Feature Agent Highlighting and Differentiation: For the target evacuation agent i whose state input needs to be acquired, create a global dynamic scene image I. global copy and its corresponding solid circle grayscale value is changed by C agent Replace with preset high-brightness grayscale value C self ; S221. Boundary padding: processing global dynamic scene image copies Boundary filling is performed so that the target evacuation agent can still obtain a complete local observation area when it is close to the image edge; S222. Self-center clipping: using the pixel coordinates (u) of the target evacuation agent i i ,v i Using ) as the center, crop out a side of length L from the image after boundary filling. crop Square local observation area ; When resampling, normalizing, and forming state input for a local observation region, at least the following should be included: S230. Resampling: The local observation area corresponding to each evacuation agent i is resampled using a bilinear interpolation algorithm. Resample to a fixed target size H×W to obtain a state image with uniform size. ; S231. Batch normalization processing: For all resampled state images Perform batch normalization processing This ensures that each state image satisfies a zero-mean and unit-variance distribution, where μ is the batch image mean, σ is the batch image standard deviation, and ε is a zero-minimum positive number. S232. State Tensor Assembly: Assemble all state images after normalization. Assembled according to the evacuation agent dimension, forming a shape equal to the preset number N of evacuation agents. agent The three-dimensional state tensor S with state image height H and state image width W. t ; S233. Status Input / Output: Set S... t The two-dimensional raster image corresponding to each evacuation agent i is used as its state input at the current time step. .

5. The method according to claim 4, characterized in that, In step SS3, when constructing a discrete action set for each evacuation agent and completing the action indexing representation, it includes at least the following: S310. Definition of Discrete Action Direction Set: The set of directions is defined by the current position of the i-th evacuation agent at the current time step t. Construct a discrete action set A with reference center. i ={a0,a1,…,a8}, where a0 represents a stationary motion, and a1~a8 represent motions in the front, back, left, right, and four diagonal directions; S311. Action direction vector association: for a discrete action set A i Each action a in k Pre-associate a direction vector e k and satisfy The stationary motion a0 corresponds to the zero vector e0=(0,0), where k=1,2,…,8. Represents the direction vector e k The modulus length; S312. Action Index Establishment: Index each action a... k The mapping is a discrete action index k∈{0,1,…,8}; When mapping discrete motions to preferred velocity vectors and applying kinematic constraints, at least the following should be included: S320. Preferred velocity vector generation: For the discrete action selected by evacuation agent i at time step t Generate the preferred velocity vector according to the deterministic mapping function. g(·) represents the mapping function from discrete motion to continuous velocity, v i,max e represents the maximum velocity scalar of evacuation agent i. k This represents the direction vector corresponding to the selected action. When the selected action is a0... ; S321. Single-step displacement upper limit constraint: Let the simulation step size be Δt, then the maximum allowable displacement d of the evacuation agent i in a single time step is... i,max Satisfy d i,max =v i,max Δt; When pruning the preferred velocity vector and performing displacement updates based on the RVO algorithm, at least the following should be included: S330. Velocity Obstacle Construction: Construct an evacuation agent i at time step t based on the set of impassable objects around it. Speed ​​obstacle set Impassable objects include environmental obstacles and nearby evacuation agents. VO(i,o) represents the velocity obstacle area of ​​evacuation agent i relative to object o. S331. Feasible velocity domain determination: Construct the feasible velocity domain while satisfying the maximum velocity constraint. v is the candidate velocity vector. Let v be the modulus; S332. Safe speed selection and displacement update: in the feasible speed domain Internal selection and preferred velocity vector closest safe speed And complete the location update at a safe speed. , This represents the position vector of the evacuation agent i at the next time step.

6. The method according to claim 5, characterized in that, In step SS4, the nonlinear distance improvement reward based on the optimal objective is calculated, including at least: S410. Construction of Multi-Target Distance Sets: For each evacuation agent i, obtain the distance sets from its current time step cur and previous time step prev to each target exit, respectively: ， Where K is the target export quantity. , Let be the distances from evacuation agent i to the k-th target exit at the current time step and the previous time step, respectively, where k = 1, 2, ..., K; S411. Target Importance Weighting: Let r be the importance weight vector of each target exit. important =[r1,r2,…,r K ], where r k The importance weight of the k-th target exit is represented by k=1,2,…,K; the weight is obtained by weighting the distance sets between the current and previous time steps element by element. , , This indicates an element-wise division operation. , These represent the weighted distance set of the current time step and the distance set of the previous time step, respectively. S412. Distance Improvement Reward Calculation: Calculate the nonlinear distance improvement reward for evacuation agent i based on the weighted distance set. w1 represents the distance improvement reward weighting coefficient, τ d λ represents the distance-improvement sensitivity adjustment parameter.

7. The method according to claim 6, characterized in that, In step SS4, when blurring the local density variable, motion smoothness variable, and time variable, at least the following should be included: S420. State Variable Acquisition: Acquire the local density variables of evacuated agent i at time step t. Motion smoothness variables and the evacuation time variable T t ; S421. Fuzzy set partitioning: dividing local density variables Divided into three fuzzy sets: low, medium, and high, with motion smoothness variables... Divided into three fuzzy sets: smaller, general, and larger, with time variable T. t It is divided into three fuzzy sets: safe, general, and dangerous. S422. Quantization of motion smoothness: ... Represented as ,in Let represent the preferred velocity vector of evacuation agent i at time step t. This represents its actual execution speed vector at the previous time step. To prevent extremely small positive numbers with a denominator of zero; S423. Construction of Symmetric Triangular Membership Functions: For each input variable , T t Constructing symmetric triangular membership functions from corresponding fuzzy sets a, b, and c represent the left endpoint, vertex, and right endpoint of the trigonometric membership function, respectively. S424. Membership Mapping: Mapping each input variable to multiple membership values, resulting in... In this context, superscripts L1, M1, and H1 represent low, medium, and high, respectively; superscripts S2, M2, and L2 represent relatively small, moderate, and relatively large, respectively; and superscripts S3, M3, and D3 represent safe, moderate, and dangerous, respectively.

8. The method according to claim 7, characterized in that, In step SS4, when performing rule activation, aggregation, and defuzzification based on the fuzzy rule base, at least the following are included: S430. Fuzzy rule base construction: A fuzzy rule base based on evacuation safety and traffic efficiency criteria is pre-built. Each rule adopts a general form: IF density state AND smoothness state AND time state, THEN reward state; S431. Rule activation strength calculation: At the current time step t, for the m-th rule, m=1,2,…,M, where M represents the total number of fuzzy rules, the activation strength is calculated using the minimum operator. , This represents the activation strength of the m-th rule at time step t. These represent the membership degrees of the fuzzy sets corresponding to the local density, motion smoothness, and time variables in the preconditions of the rule, respectively. S432. Reward Fuzzy Set Aggregation: For all activated fuzzy rules, the reward fuzzy set in the rule conclusion is truncated according to its activation strength, and then aggregated using the maximum operator to obtain the comprehensive output fuzzy reward set. y is a discrete reward point in the reward universe. Let y be the membership degree of the fuzzy set of rewards corresponding to the conclusion of the m-th rule at the reward point y. S433. Centroid Defuzzification: The centroid method is used to defuzzify the aggregated output fuzzy reward set to obtain the fuzzy reward value. Y represents the set of discretized reward domains, ε R This indicates an extremely small positive number that prevents the denominator from being zero; When integrating fuzzy rewards with distance improvement rewards to form instant reward feedback, it should at least include: S440. Reward Integration: Integrating the fuzzy reward value at time step t Distance Improvement Rewards By integrating the results, we can obtain the final instantaneous reward of the evacuated agent i at time step t. ; S441. Motion Feedback Association: [This will be used in conjunction with other methods] As the evacuation agent i, in state Next action The feedback signal after that, and the state at the next time step. Together they constitute the experience of state transition.

9. The method according to claim 1, characterized in that, In step SS5, the deep reinforcement learning framework uses a CNN network to process the visual state input. The input image size is 84×84. The CNN network includes three convolutional layers, a feature flattening layer, and a fully connected layer. The three convolutional layers use structures of 32 8×8 convolutional kernels with a stride of 4, 64 4×4 convolutional kernels with a stride of 2, and 64 3×3 convolutional kernels with a stride of 1, respectively. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The policy network and target network adopt a Dueling-DQN structure. Convolutional features are flattened and input into the value stream and advantage stream respectively to output state value estimates and advantage values ​​for each discrete action. The fully connected layer adopts a noisy linear layer, where deterministic mean parameters and random noise parameters jointly represent weights and biases. All evacuation agents share the current policy network parameters and construct a unified experience replay pool to store all transfer experiences. The experience replay pool adopts a priority sampling mechanism, and during training, multi-step rewards are calculated, the current policy network parameters are updated based on temporal difference loss, and periodically copied to the target network parameters.

10. The method according to claim 8, characterized in that, In step SS6, when constructing the fuzzy membership degree of evacuation-acceptable actions based on the discrete action values ​​output by the policy network, it includes at least: S610. Action Value Acquisition: For the current state of evacuated agent i at time step t Obtain its discrete action set A i ={a0,a1,…, Action value estimation for each action in} , Q represents the total number of discrete actions. θ Represents the current policy network with parameter θ; S611. Determining the maximum action value: Determine the maximum action value in the current state based on the action value estimate. , used to characterize the optimal action reference point in the current state; S612. Construction of the fuzzy set of acceptable actions: Define the fuzzy set A of evacuation acceptable actions. acceptable And the degree to which the k-th discrete action belongs to the fuzzy set is denoted as , This indicates that the evacuation agent i performs action a at time step t. k Acceptable membership degree; S613. Calculation of exponential fuzzy membership: Calculate the fuzzy membership of each discrete action according to the exponential decay function. δ>0 represents the diffusion gate parameter; When constructing random diffusion components based on action values ​​and obtaining unnormalized selection weights, at least the following should be included: S620. Construction of Random Diffusion Log-Weights: For the k-th discrete action of evacuated agent i at time step t, calculate its unnormalized log-weights. τ>0 indicates the temperature parameter; S621. Numerical stabilization processing: for Standardization process is required to obtain , This represents the standardized logarithmic weights after stabilization. S622. Calculation of random diffusion components: based on Calculate the random diffusion component of each discrete action. , Indicates action a k The random diffusion component at time step t; S623. Unnormalized selection weight calculation: Calculate fuzzy membership degrees With random diffusion components Multiplying them together yields the unnormalized selection weights for the k-th discrete action. .

11. The method according to claim 10, characterized in that, In step SS6, when normalizing the unnormalized selection weights and generating action selection probabilities, at least the following is included: S630. Construction of Normalized Denominator: Summing up all discrete actions of evacuated agent i at time step t with unnormalized selected weights, and introducing a minimum positive number ε to construct the normalized denominator. ; S631. Action Selection Probability Calculation: Calculate the final selection probability of each discrete action based on the normalized denominator. , This indicates that the evacuation agent i selects action a at time step t. k The probability of; S632. Probability constraint satisfied: The final selection probability satisfies... and This makes all discrete actions form a complete probability distribution; S633. Probability Distribution Output: Outputs the action selection probability distribution. The output is sent to the action sampling execution unit as the basis for action sampling at the current time step; When sampling an action based on its selection probability and outputting the sampling result, at least the following should be included: S640. Multinomial sampling execution: Based on the probability distribution of the action of evacuated agent i at time step t. The sampling action index for the current time step is generated using a polynomial sampling method. ; S641. Discrete Action Determination: Based on Determine the discrete action at the current time step ; S642. Action Output Association: Associate discrete actions The output is the preferred velocity mapping and safe velocity clipping process in step SS3, which is used to generate the corresponding preferred velocity vector and safe execution velocity. S643. Sampling Mechanism Constraint Effect: The action value orientation term, fuzzy acceptable constraint term, and random diffusion term are applied together to the action sampling process, so that the action selection retains adjustable randomness only within the acceptable range of actions that meet the basic requirements of evacuation rationality.

12. The method according to claim 1, characterized in that, In step SS7, the preset termination condition is that all evacuation agents complete the evacuation or the simulation time reaches the maximum duration. When the preset termination condition is met, the motion trajectory of each evacuation agent, individual evacuation time, total evacuation time, evacuation flow rate curve, local density distribution of the exit neighborhood, and overall evacuation visualization process are uniformly summarized and output.

13. A personnel evacuation simulation system, comprising multiple modules, characterized in that, Used to perform the personnel evacuation simulation method based on multi-agent reinforcement learning as described in any one of claims 1 to 12.

Images