Emergency shelter multi-objective site selection optimization method based on integrated reinforcement learning

Abstract

The application discloses an emergency protection site multi-target location optimization method based on integrated reinforcement learning, belongs to the field of site location optimization, and comprises the following steps: S1, acquiring candidate points and demand point data in a region, and constructing a multi-target reinforcement learning problem; S2, performing Markov modeling based on a single-agent decision deep reinforcement learning problem; S3, constructing an integrated reinforcement learning framework; S4, training the constructed integrated reinforcement learning network; and S5, obtaining a location result by using the integrated reinforcement learning framework and performing visual presentation; the application introduces an integrated learning strategy and an adaptive optimization technology, combines spatial information technology of a remote sensing and a geographic information system core, realizes multi-target location optimization of urban emergency protection sites, and enhances robustness and practicability of an aggregated reinforcement model of an Actor-Critic network.

Description

Technical Field

[0001] This invention belongs to the field of emergency protection site selection optimization, specifically involving a multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning. Background Technology

[0002] In recent years, urbanization has progressed rapidly worldwide. However, the rapid evolution of urban spatial structures and high-density population concentration have also brought enormous pressure on resource allocation, environmental carrying capacity, and public safety. Under the complex interaction of urban systems and climate risks, the vulnerability of high-density, highly connected urban systems has increased significantly, making them highly susceptible to triggering and exacerbating multiple disaster chains, posing unprecedented challenges to urban emergency management systems. Currently, urban emergency shelters have significant shortcomings in coverage, service capacity, and timeliness, making it difficult to meet the needs of rapid evacuation and temporary resettlement of residents in high-density urban spaces. Therefore, constructing a scientifically sound and reasonable site selection optimization model for emergency shelters is of great significance for developing risk-oriented urban spatial optimization strategies and achieving efficient allocation of emergency resources and the scientific construction of urban emergency refuge systems.

[0003] Current technologies tend to construct facility location models from a mathematical programming perspective, combined with the characteristics of emergency safety location problems. Among these, research on single-objective location problems is the most in-depth and mature. Single-objective location problems are centered on a single objective function and, depending on the objective, can be classified into types such as coverage problems, P-median problems, and P-center problems. With the deepening of research, research models for emergency protection site location have evolved from single-objective to multi-objective. Different site selection optimization objectives and constraints have been proposed based on the actual conditions of the research area to address potential scenarios and needs during disasters. The safety site layout optimization problem under multi-objective constraints is classified as an NP-hard problem due to its complexity, making it extremely difficult to solve using precise computational methods. Currently, commonly used algorithms in emergency protection site location optimization problems include genetic algorithms, particle swarm optimization algorithms, and ant colony optimization algorithms, but they have significant limitations and are not suitable for solving large-scale complex location problems.

[0004] Deep reinforcement learning algorithms have shown greater advantages over other algorithms in handling decision-making problems in complex and unknown environments, and are increasingly being introduced into location selection problems to overcome the limitations of traditional optimization algorithms. Based on Markov decision processes, deep reinforcement learning can effectively solve some problems with high-dimensional features, and significantly reduces the need for manual parameter tuning compared to heuristic algorithms. It has obvious advantages in optimization results and training efficiency, and has been applied to various real-world scenarios. However, while existing deep reinforcement learning algorithms have improved performance in handling complex problems, the instability of training results still needs to be addressed. Summary of the Invention

[0005] To address the problems mentioned in the background art, this invention proposes a multi-objective site selection optimization method for emergency protection sites based on ensemble reinforcement learning. It improves the actor-critic network structure and introduces an ensemble learning strategy, thereby enhancing the robustness and practicality of the model through the aggregation of the actor-critic network.

[0006] Technical Solution: To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] An optimization method for multi-objective site selection in emergency protection areas based on ensemble reinforcement learning includes the following steps:

[0008] S1: Obtain candidate points and demand points data within the region, and construct a multi-objective reinforcement learning problem;

[0009] S2: Perform Markov modeling for a deep reinforcement learning problem based on single-agent decision-making;

[0010] S3: Building an integrated reinforcement learning architecture;

[0011] S4: Train the constructed ensemble reinforcement learning network;

[0012] S5: Use an integrated reinforcement learning framework to obtain the location selection results and present them visually.

[0013] As a preferred option, the specific implementation process in S1 is as follows:

[0014] There are m demand points, forming a set. ,in, Indicate demand point i;

[0015] There are n candidate points, forming a set. ,in, Let j represent a candidate point, and each candidate point has a corresponding service radius.

[0016] As a preferred approach, the objective function for constructing the multi-objective reinforcement learning problem is:

[0017]

[0018] in, This represents the population size at demand point i. Indicates whether candidate point j provides service to demand point i; This represents the calculated distance from demand point i to candidate point j; This indicates that the value of the objective function needs to be maximized.

[0019] As a preferred option, the specific content of S2 is as follows:

[0020] Markov modeling is performed on a single-agent decision-making deep reinforcement learning problem based on Markov decision processes; and a quintuple is employed. Express;

[0021] Where S represents the state set, A represents the action set, P represents the state transition probability, and R represents the reward function. This represents the discount factor.

[0022] As a preferred option, the reward function includes: the distance from each demand point to the nearest location, the proportion of overlapping areas between locations, and the total population that can be covered by all selected locations. Each component has been normalized, specifically:

[0023] ,

[0024] in, Indicates the distance between emergency protection sites and points of need; Indicates the demand point at time t. The distance to the nearest location; m represents the number of demand points; Indicates the maximum distance in the scene;

[0025] ,

[0026] in, Indicates the overlap rate of protective services; This indicates the number of demand points covered by multiple locations at time t. This indicates the number of locations requiring coverage at time t.

[0027] ,

[0028] in, This represents the sum of the weighted population at each demand point; This represents the weighted sum of the populations covered by all locations at time t. Indicates the weighted total population;

[0029] ,

[0030] in, Describe the final objective function; , and This represents the weighting factor.

[0031] As a preferred option, the specific implementation process in S3 is as follows:

[0032] S31: The policy-based PPO algorithm is selected as the basic framework to calculate the policy and value;

[0033] S32: Employ an actor-critic neural network structure to fit strategy and value;

[0034] S33: Calculate the total loss function;

[0035] S34: Introduce an ensemble learning module into the PPO framework;

[0036] S35: Introduces adaptive optimization techniques to accelerate training efficiency.

[0037] As a preferred option, the specific implementation details in S31 are as follows:

[0038] The policy-based PPO algorithm is selected as the basic framework, and an ensemble learning policy is introduced, where the policy is a mapping from the state space to the action space; the methods for calculating the policy and value are as follows:

[0039] ,

[0040] ,

[0041] in, Representation strategy; Representing state Take action below The probability of; These represent the parameters that can be updated in the neural network that support the fitting strategy; Indicates time Next state The value function, E represents the parameters that can be updated in the neural network that represent the fit value; E represents the expectation. Indicates an immediate reward; Indicates the discount factor; Representing state Value function estimation.

[0042] As a preferred embodiment, in S32, the specific implementation includes a shared feature extraction network, an actor network, and a critic network.

[0043] A shared feature extraction network is used to extract general features from the input state and includes linear layers, ReLU activation layers, and normalization layers.

[0044] The actor network is used to fit the policy function and consists of two residual blocks. Each residual block contains a linear layer, a ReLU activation layer, and a normalization layer. The final layer of the actor network is... The function outputs the probability distribution of actions;

[0045] The critic network is used to fit the value function and consists of two residual blocks, each containing a linear layer, a ReLU activation layer, and a normalization layer. The final layer of the critic network outputs the state value estimate through the linear layer.

[0046] As a preferred option, the specific implementation process in S4 is as follows:

[0047] S41: Initialize the discount factor and learning rate, and initialize the neural network;

[0048] S42: Initialize the initial state based on the data in the simulation area;

[0049] S43: Change the current state The input is fed into a neural network for analysis, and the output is an action. ;

[0050] S44: Update the structure parameters and obtain reward feedback from the preset reward function. and current status ;

[0051] S45: Store the samples collected within this time step into the experience pool;

[0052] S46: Update network parameters;

[0053] S47: Determine if the learning process is complete; otherwise, jump back to S42.

[0054] As a preferred option, the specific implementation process in S5 is as follows:

[0055] S51: Input the demand points and candidate points into the multi-objective ensemble reinforcement learning model;

[0056] S52: Model outputs location selection results;

[0057] S53: Optimize the site selection results and visualize them.

[0058] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0059] (1) Based on the basic framework of deep reinforcement learning, this invention fully considers multiple optimization objectives. In view of the shortcomings of the existing emergency protection site selection optimization model, it introduces ensemble learning strategy and adaptive optimization technology. Combining spatial information technology with remote sensing and geographic information system as the core and computer technology with deep learning and reinforcement learning as the core, it realizes multi-objective site selection optimization of urban emergency protection sites: First, it acquires candidate point and demand point data in the region and constructs a multi-objective reinforcement learning problem; Second, it performs Markov modeling for the multi-objective site selection problem of urban emergency protection sites; Then, it constructs an ensemble reinforcement learning architecture, using the policy-based PPO algorithm as the basic framework, and introduces an ensemble learning strategy to aggregate three Actor-Critic networks; Then, it trains the constructed ensemble reinforcement learning network; Finally, it uses the constructed ensemble reinforcement learning framework to obtain the site selection results and presents them in a visual format.

[0060] (2) This invention fully considers the characteristics of multi-objective optimization models, selects planning objectives that are combined with actual conditions, and makes full use of the advantages of deep reinforcement learning and ensemble learning. It has the characteristics of fast calculation speed, high model stability, good visualization effect, and wide applicability to solving problems of all sizes. It provides an effective method for multi-objective site selection optimization of urban emergency protection sites and can be applied to multi-objective planning problems in scenarios of all sizes.

[0061] (3) The present invention is reliable and practical. It has good accuracy and practicality for multi-target site selection optimization of urban emergency protection sites. It has good effect when compared with actual conditions. It has good practicality and feasibility. Attached Figure Description

[0062] Figure 1 This is a flowchart of the optimization method of the present invention;

[0063] Figure 2 This is a framework diagram of the integrated reinforcement learning algorithm of this invention;

[0064] Figure 3 The network structure diagram of the actor-critic network for integrating reinforcement learning algorithms in this invention;

[0065] Figure 4 This is a flowchart illustrating the integrated reinforcement learning network training process of this invention.

[0066] Figure 5 This is a graph showing the reward changes during the training process of the integrated reinforcement learning algorithm of this invention;

[0067] Figure 6 A graph showing the change in the number of emergency protection sites selected during the training process of the integrated reinforcement learning algorithm of this invention;

[0068] Figure 7 This is a diagram showing the site selection results of the present invention;

[0069] Figure 8 This is a comparison chart of the cumulative probability and probability density before and after the optimization of this invention. Detailed Implementation

[0070] The present invention will be further illustrated below with reference to specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0071] like Figures 1-8 As shown in the figure, the multi-objective site selection optimization method for emergency protection sites based on ensemble reinforcement learning provided in this embodiment introduces ensemble learning strategies and adaptive optimization techniques. It combines spatial information technology with remote sensing and geographic information systems as its core, and computer technology with deep learning and reinforcement learning as its core, to achieve multi-objective site selection optimization for urban emergency protection sites. Specifically, it includes the following steps:

[0072] S1. Obtain candidate points and demand points data within the region, and construct a multi-objective reinforcement learning problem;

[0073] The location problem can be described as:

[0074] There are m demand points, each with specific protection requirements, forming a set. ;in, Indicates demand point i.

[0075] There are n candidate sites for protective shelters. Several facilities need to be selected as protective shelters to provide protective services, forming a set. ,in, Let j represent candidate point j, and each candidate point has a specific service radius. When selecting a protected site, the following objectives must be achieved:

[0076] (1) Minimize the total protection distance from the demand point to the protection site.

[0077] (2) Maximize the total number of people served by the shelter while taking into account risk factors.

[0078] (3) Minimize the overlap of the protection area served.

[0079] To simplify the model's complexity, details with minimal impact on performance are omitted. Specifically, the impact of location capacity on site selection is not considered; it is assumed that each location has sufficient capacity to accommodate the population within its service radius. The selection process allows for demand points not covered by any location (i.e., service blind spots), which aligns with reality. However, as one of the optimization objectives, the model will ensure that service blind spots are minimized.

[0080] Table 1. Variables and parameters for the site selection of emergency protection facilities

[0081]

[0082] The objective function for constructing a multi-objective reinforcement learning problem is:

[0083]

[0084] in, This represents the population size at demand point i. Indicates whether candidate point j provides service to demand point i; This represents the calculated distance from demand point i to candidate point j; This indicates that the value of the objective function needs to be maximized.

[0085] The first item ( The first item represents the total population that can be covered by all selected locations, and the second item ( The third item represents the sum of the distances from each demand point to the nearest location; () indicates the overlapping area between different locations.

[0086] The constraints for the multi-objective reinforcement learning problem are:

[0087]

[0088]

[0089]

[0090]

[0091] in, Indicate demand points to candidate point Calculate the distance. Indicate demand points to candidate point The actual distance; This indicates the maximum distance; if the demand point is within the service area of ​​the selected location, the actual distance is used directly. Otherwise calculate the distance. It will be assigned the maximum distance in the scene. . This indicates that the distance between the point of service demand and the corresponding selected location does not exceed the service radius. Indicate candidate points Is it a demand point? Provide services; , is a Boolean variable Provide binary constraints.

[0092] S2: Perform Markov modeling for a deep reinforcement learning problem based on single-agent decision-making;

[0093] The problem is modeled as a single-agent decision-making deep reinforcement learning problem based on Markov decision processes, and a quintuple is employed. The expression is given by: where S represents the state set, A represents the action set, P represents the state transition probability, and R represents the reward function. This represents the discount factor. Specifically:

[0094] State: The state of an agent consists of three parts:

[0095] (1) Position matrix, size is m represents the total number of demand points; n represents the total number of candidate points; l represents the total number of selected points; the values ​​in the matrix are all represented by {0,1}, which represent whether the demand point is within the service range of the location point.

[0096] (2) Population size is a quantity of... The vector.

[0097] (3) An indicator vector used to record whether each candidate location is currently selected, containing the selection status of each location and a stop flag, with a size of [missing information]. The values ​​in this array are all represented by {0,1}.

[0098] Action: At each time step, the agent selects a new location from the candidate locations based on its state and policy, denoted by k, representing the selected candidate location. The serial number.

[0099] State transition function: In the context of the addressing problem, the transition is deterministic, that is, given time step t... and , It is uniquely certain. Therefore, ;in, This indicates the state at time step t; Indicates the action at time step t; Indicates the state Take action below Then, the environment transitions to the next state. The probability of.

[0100] Reward function: The reward function is a key part of the entire algorithm. It determines the agent's action choices and further affects the quality of model training. Therefore, the reward function needs to accurately reflect the goal that the agent is expected to learn.

[0101] In the multi-objective location problem of this embodiment, the reward function consists of three parts: the distance from each demand point to the nearest location point, the proportion of overlapping areas between location points, and the total population that can be covered by all selected location points. Each part has been normalized.

[0102]

[0103] in, Indicates the demand point at time t. The distance to the nearest location (i.e., the selected candidate point), if Not served by any location, this distance is the maximum distance set in the scene. .

[0104]

[0105] in, Indicates the overlap rate of protective services; This indicates the number of demand points covered by multiple locations at time t. This represents the number of demand points covered by all locations at time t.

[0106]

[0107] in, This represents the sum of the weighted population at each demand point; This represents the weighted sum of the populations covered by all locations at time t. This represents the weighted total population.

[0108]

[0109] in, Describe the final objective function; , and These are weighting factors adjusted according to the specific requirements of the problem.

[0110] S3: Building an integrated reinforcement learning architecture;

[0111] The policy-based PPO algorithm is selected as the basic framework, and an ensemble learning strategy is introduced to enhance the robustness of the model through the aggregation of three Actor-Critic networks.

[0112] S31: The policy-based PPO algorithm is selected as the basic framework to calculate the policy and value;

[0113] Specifically, a policy is a mapping from the state space to the action space. The policy uses... This indicates the state. Take action below The probability of , where, These are the parameters that can be updated in the neural network that fits the strategy. For time... Next state Value function estimation express, These are the parameters that can be updated in a neural network that fits the value. The strategy and value are calculated as follows:

[0114]

[0115]

[0116] Where E represents expectation; Indicates an immediate reward; Indicates the discount factor; Representing state Value function estimation.

[0117] S32: Employ an actor-critic neural network structure to fit strategy and value;

[0118] The PPO model employs an actor-critic neural network structure to fit the strategy and value. It consists of three parts: a shared feature extraction network, an actor network, and a critic network.

[0119] The Actor network takes the size of the state space as its input dimension and the size of the action space as its output dimension, and is used to fit the policy function; the Critic network takes the size of the state space as its input dimension and has a single output dimension, and is used to fit the value function.

[0120] To achieve gradient updates of the policy and value functions, the model uses the advantage estimation function. Measuring the quality of an action relative to the average level:

[0121]

[0122] in, Represents the advantage estimation function; This indicates the cumulative return on the discount; Indicates time Next state The value function, This represents a discount factor used to weigh the importance of future rewards. Indicates a future time step; Indicates future time steps Instant rewards obtained at that time.

[0123] The shared feature extraction network consists of a three-layer structure: a linear layer, a ReLU activation layer, and a normalization layer, and is used to extract general features from the input state.

[0124] The actor network and the critic network each consist of two residual blocks, each containing a linear layer, a ReLU activation layer, and a normalization layer. Residual connections enhance gradient propagation and training stability. The final layer of the actor network... The function outputs the action probability distribution, while the final layer of the Critic network outputs a state value estimate through a linear layer. Compared to traditional network structures, this design reduces computational redundancy by sharing a feature extraction network and alleviates the gradient vanishing problem through residual connections, significantly improving training performance in complex location scenarios.

[0125] In addition, a temperature parameter was added to the Actor actions, and the distribution of action probabilities was altered in conjunction with the Softmax function. This applies to the action space. One of the actions initial probability Its probability after temperature adjustment for:

[0126]

[0127] in, Represents the initial probability of action a. Unnormalized probability value after adjustment for temperature parameter T; Represents the initial probability of the i-th action in action space A. Unnormalized value after temperature adjustment For temperature parameters, The higher the value, the more evenly distributed the actions will be, and the more the strategy will be inclined towards exploration; The lower the value, the more certain the action choices will be, and the more the strategy will lean towards exploration. This parameter allows for a trade-off between exploration and exploration.

[0128] In the early stages of reinforcement learning training, the learning level of the policy is insufficient, requiring more exploration, while in the later stages, more exploitation is needed. Therefore, a temperature decay mechanism is introduced, with the initial temperature set to 1.0. With each update, the temperature decays complementaryally at a rate of 0.995 until a minimum temperature of 0.1 is set, realizing the transition of the model from high exploration to high exploitation and preventing the model from converging too early.

[0129] PPO is a hetero-policy algorithm where the agent being learned is different from the agent interacting with the environment. To improve data utilization efficiency, PPO employs importance sampling, allowing data generated by the existing agent interacting with the environment to be used for policy updates of the new agent. After each update, the new agent copies its parameters to the existing agent, continuing the next interaction and update cycle. This process is repeated continuously. The problem with importance sampling is that the difference in policy probability distributions between the old and new agents affects the sampling effectiveness; a large difference can lead to poor training results. The difference between different policy probability distributions is quantified as follows:

[0130]

[0131] in, This represents the difference between the probability distributions of different strategies; Indicates the new strategy in state Select action The probability of; Indicates the old strategy in state Select action The probability of; Indicates the parameters that can be updated in the new strategy; This indicates the parameters that can be updated in the old strategy.

[0132] S33: Calculate the total loss function;

[0133] The core of PPO lies in ensuring training stability by limiting the policy update magnitude, while retaining the advantages of the policy gradient method.

[0134] The optimization objective of the Actor network is based on the shearing ratio, specifically:

[0135]

[0136] in, Represents the policy loss function of the actor network; Expressing expectations; Represents the advantage estimation function; The clip() function represents the clipping factor. Cut the code to limit the differences between the old and new strategies;

[0137] The Critic network optimizes the state-value function through mean square error, specifically:

[0138]

[0139] in, Represents the value loss function of the critic network; Indicates time Next state The value function, This indicates the cumulative return on the discount; It expresses expectation.

[0140] To encourage exploration, an entropy regularization term is added, specifically:

[0141]

[0142]

[0143] in, This represents the entropy regularization loss; Expressing expectations; Representation Strategy In state Entropy below; Representation Strategy Entropy; Indicates the policy in the state Select action The probability of; Indicates an action; Indicates the state.

[0144] Combining the above three types of losses, the total loss function for:

[0145]

[0146] in, , This is a hyperparameter.

[0147] Parameter updates are performed using gradient descent, specifically:

[0148]

[0149]

[0150] in, and The learning rates are for the actor network and the critic network, respectively. Represents the policy gradient of the actor network; This represents the policy gradient of the commentator network; This represents the total loss function.

[0151] S34: Introduce an ensemble learning module into the PPO framework;

[0152] The introduced ensemble learning module includes: N parallel learning policies and value functions. Each policy is initialized using a different random seed and updated by sampling a subset of data from the shared experience buffer generated by the interaction between the aggregation policy and the environment through bootstrap sampling, thereby reducing data correlation. The performance of each policy is evaluated using discounted average reward, specifically:

[0153]

[0154] in, Indicate the performance of the i-th strategy; Represented as a strategy The average reward; As the attenuation factor, For strategy The average reward. The aggregation strategy is generated by a performance-weighted average of the strategies, specifically:

[0155]

[0156]

[0157] in, Indicates the aggregation strategy; This represents the parameters of the i-th strategy; Indicates the total number of strategies participating in the aggregation; This represents the weight of the i-th strategy; Indicate the performance of the i-th strategy; This represents the performance of the j-th strategy.

[0158] S35: Introduces adaptive optimization technology to accelerate training efficiency;

[0159] The action space is defined as either selecting a candidate location or stopping, and its size equals the number of candidate locations plus one. Unlike the reward function calculation method for selecting a candidate location, when the model chooses to stop, the reward function is calculated as follows: first, the overall score of the currently selected location is calculated; then, for each unselected location, the hypothetical gain compared to the current value is calculated, and the negative of the maximum value (i.e., the optimal marginal benefit) is taken as the reward for the stopping action. This improvement incorporates both the timing and criteria for location selection and stopping into the model's learning process, further enhancing training speed and performance.

[0160] Because the model has a strong ability to extract problem features, it can often converge the policy function completely in a relatively small number of steps. Therefore, an early stopping mechanism is introduced, given a window size. and convergence threshold Record the past The rewards for each training step are calculated, and the variance of these rewards is calculated. A smaller variance indicates more stable training results. When the variance is less than... If that happens, stop training the entire model.

[0161] The model uses masks to shield invalid actions at each action selection step. On one hand, an indicator vector is constructed to record whether each candidate location is selected, preventing the selection of a previously selected action and avoiding unnecessary operations and resource waste during training. On the other hand, when selecting an action, hard constraints are used to avoid selecting points that are "too close" to any currently selected location. The definition of "too close" between two locations is:

[0162]

[0163] in, and Location and The service radius; Indicate location and The distance.

[0164] S4: Train the constructed ensemble reinforcement learning network;

[0165] Train a multi-construction ensemble reinforcement learning network. Specifically:

[0166] S41: Initialize hyperparameters such as discount factor and learning rate, and initialize the neural network. The weights and other parameters of each learner's neural network are initialized using different methods. By randomly setting different parameters, individual learners with significant differences are generated.

[0167] S42: Initialize the initial state based on the data in the simulation area;

[0168] S43: Change the current state The input is fed into a neural network for analysis, and the output is the designed action. ;

[0169] S44: Update the structure parameters and obtain reward feedback from the preset reward function. and current status ;

[0170] S45: Samples collected within this time step Store in the experience pool;

[0171] S46: Update network parameters;

[0172] S47: Determine if the learning process is complete; otherwise, jump back to S42.

[0173] S5: Use an integrated reinforcement learning framework to obtain the location selection results and present them visually.

[0174] The location selection results are obtained using an ensemble reinforcement learning framework and then visualized. Specifically:

[0175] S51: Input the demand points and candidate points into the multi-objective ensemble reinforcement learning model;

[0176] S52: The model outputs the number of the selected refuge site, i.e., the site selection result;

[0177] S53: Visualize the optimized site selection results.

[0178] In this embodiment, the method of the present invention is compared with some existing methods. The comparison algorithms include the SA algorithm and the PPO algorithm.

[0179] In terms of training time, PPO and the method of this invention have similar training times, both on the same order of magnitude, and are much shorter than the training time of SA.

[0180] In terms of training performance metrics, PPO maintained the highest overall score among all algorithms, improving by 0.3%-11.4% compared to SA and by 5.1%-20.3% compared to PPO, especially showing significant advantages in distance score and population score.

[0181] Five experiments were conducted to assess training robustness. The results showed significant differences in training results for PPO, while no difference was found between SA and the proposed method. The results indicate that although SA possesses superior performance, as a heuristic swarm intelligence algorithm, its training speed is far slower than reinforcement learning algorithms, limiting its application value in large-scale problems such as real-world scenarios. The proposed algorithm, requiring simultaneous training of multiple learners, has a longer training time than PPO, which only requires training one learner. However, the results demonstrate that the proposed algorithm achieves better training results with only a small sacrifice in training speed. Furthermore, as the data scale increases, the speed disadvantage of the proposed algorithm continuously diminishes, while its performance advantage becomes increasingly significant. Therefore, the proposed algorithm has great application potential for large-scale problems.

[0182] In this embodiment, the convergence speed and learning effect of the method of the present invention were also evaluated. For example... Figure 5 As shown, the reward function feedback value changes with the number of training rounds during the training process. The model learns very quickly and approaches the optimal solution around the 100th round. However, since the exploration rate is still high at this time, there are some fluctuations between the 100th and 700th rounds. Around the 1000th round, as high exploration gradually shifts to high exploitation, the reward and policy become stable and eventually converge to the optimal solution. The average reward for the last 30 rounds is 14.10.

[0183] like Figure 6As shown, the change in the number of emergency shelter locations selected during training is illustrated. Similar to the reward function, the high exploration rate in the early stages leads to fluctuations in the number of selections, while the stable strategy in the later stages causes the number to steadily decrease and converge. The above training process demonstrates the powerful feature extraction capability and high stability of the algorithm of this invention.

[0184] On the other hand, the distance distribution from each residential point to its nearest emergency shelter is statistically analyzed, and the data before and after optimization using the method of this invention is compared, for example... Figure 8 As shown, the optimized average distance is 2.518 km, the median distance is 2.351 km, and more than 53% of the settlements are less than 2.5 km away, an increase of 21% compared to before optimization; more than 95% of the settlements are less than 5 km away, an increase of 25% compared to before optimization. The method of this invention can obtain good optimization results for the layout of emergency protection sites, and the solution speed is fast and the solution results are stable.

[0185] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An integrated reinforcement learning-based multi-objective site selection optimization method for emergency protection sites, characterized in that: Comprising the following steps: S1: Obtain candidate points, demand point data in the region, and construct a multi-objective reinforcement learning problem; There are m demand points, constituting a set wherein, denotes demand point i; There are n candidate points, constituting a set wherein, representing candidate point j, each candidate point has a corresponding service radius; The objective function of the multi-objective reinforcement learning problem is: ， wherein, represents the number of people at demand point i; represents whether candidate point j serves demand point i; represents the calculated distance from demand point i to candidate point j; represents the value of the objective function that needs to be maximized; S2: Perform Markov modeling based on a single-agent decision deep reinforcement learning problem; Perform Markov modeling based on a single-agent decision deep reinforcement learning problem of a Markov decision process; and use a five-tuple <S, A, P, R, γ> to express; where S denotes a set of states, A denotes a set of actions, P denotes a state transition probability, and R denotes a reward function, denotes a discount factor; The reward function includes: the distance from each demand point to the nearest site point, the proportion of overlapping areas between each site point, and the total population covered by all selected site points. Each part is normalized, specifically: ， wherein, denotes the distance between the emergency shelter and the demand point; denotes the distance between the demand point and the nearest shelter at time t; m denotes the number of demand points; denotes the maximum distance in the scenario; ， wherein, represents the overlap rate of the guard service; represents the number of demand points covered by multiple sites at time t, represents the number of demand points covered by all sites at time t; ， wherein, denotes the sum of the weighted population numbers of the demand points; denotes the sum of the weighted population numbers covered by all sites at time t, denotes the total number of weighted populations; ， wherein, represents the final objective function; , and represents a weight factor; S3: Construct an integrated reinforcement learning architecture; S4: Train the constructed integrated reinforcement learning network; S5: Obtain the site selection result using the integrated reinforcement learning framework and perform visual presentation.

2. The integrated reinforcement learning based multi-objective optimization method for emergency shelter site selection according to claim 1, characterized in that: In S3, the specific implementation process is: S31: Select the policy-based PPO algorithm as the basic framework to calculate the policy and value; S32: Use an actor-critic neural network structure to fit the policy and value; S33: Calculate the total loss function; S34: Introduce an ensemble learning module in the PPO framework; S35: Introduce an adaptive optimization technique to accelerate training efficiency.

3. The integrated reinforcement learning based multi-objective optimization method for emergency shelter site selection according to claim 2, characterized in that: In S31, the specific implementation content is: Select the policy-based PPO algorithm as the basic framework, introduce the ensemble learning strategy, and the policy is the mapping from the state space to the action space; the calculation method of the policy and the value is: ， ， where represents a policy; represents a state takes an action with probability; represents parameters in a neural network that can be updated to fit the policy; represents a value function for a state at time , represents parameters in a neural network that can be updated to fit the value; E represents an expectation; represents an immediate reward; a discount factor; represents a value function estimate for a state .

4. The integrated reinforcement learning based multi-objective optimization method for emergency shelter site selection according to claim 2, characterized in that: In S32, the specific implementation content includes a shared feature extraction network, an actor network, and a critic network. The shared feature extraction network is used to extract general features from the input state, including linear layers, ReLU activation layers, and normalization layers. The actor network is used to fit the policy function, including two residual blocks, each residual block containing a linear layer, a ReLU activation layer, and a normalization layer, the final layer of the actor network outputs an action probability distribution by the function outputs an action probability distribution; The critic network is used to fit the value function, including two residual blocks, each containing linear layers, ReLU activation layers, and normalization layers. The final layer of the critic network outputs the state value estimate through a linear layer.

5. The integrated reinforcement learning based multi-objective optimization method for emergency shelter site selection according to claim 1, characterized in that: In S4, the specific implementation process is: S41: Initialize the discount factor and learning rate, and initialize the neural network; S42: Initialize the initial state according to the data situation of the simulated region; S43: the current state input to neural network for analysis, output action ; S44: update the structure parameters and get reward feedback from the preset reward function and the current state ; S45: Store the samples collected in this time step into the experience pool; S46: Update the network parameters; S47: Determine whether the learning process is complete, and if not, jump back to S42.

6. The integrated reinforcement learning based multi-objective optimization method for emergency shelter site selection according to claim 1, characterized in that: In S5, the specific implementation process is: S51: Input the demand points and candidate points into the multi-objective integrated reinforcement learning model; S52: The model outputs the site selection result; S53: Optimize the site selection result and perform visual expression.

Images