Emergency protection place multi-target site selection optimization method based on integrated reinforcement learning

Abstract

The invention discloses an emergency protection place multi-target site selection optimization method based on integrated reinforcement learning, and belongs to the field of place site selection optimization, and the method comprises the steps: S1, obtaining candidate point and demand point data in a region, and constructing a multi-target reinforcement learning problem; s2, Markov modeling is carried out, and a deep reinforcement learning problem is decided based on a single agent; s3, constructing an integrated reinforcement learning framework; s4, training the constructed integrated reinforcement learning network; s5, obtaining a site selection result by using an integrated reinforcement learning framework, and performing visual presentation; according to the method, an integrated learning strategy and a self-adaptive optimization technology are introduced, and a spatial information technology with remote sensing and geographic information system cores is combined, so that multi-target site selection optimization of the urban emergency protection site is realized, and the robustness and practicability of the model are enhanced through aggregation of the Actor-Critic network.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of emergency shelter site selection optimization, and particularly relates to a multi-objective site selection optimization method for emergency shelters based on integrated reinforcement learning. BACKGROUND

[0002] In recent years, the urbanization process has been rapidly advancing worldwide. However, the rapid evolution of urban spatial structure and the high-density concentration of population also bring great pressure on resource allocation, environmental carrying capacity, and public safety. Under the interaction of complex urban systems and climate risks, the vulnerability of urban systems with high population density and high connectivity significantly increases, and various disaster chains are easily triggered and intensified, posing unprecedented challenges to urban emergency management systems. Currently, urban emergency shelters have obvious shortcomings in coverage, service capacity, and timeliness, and are difficult to meet the rapid evacuation and temporary resettlement needs of residents in high-density urban spaces. Therefore, it is of great significance to construct a scientific and reasonable site selection optimization model for emergency shelters to develop risk-oriented urban spatial optimization strategies and achieve efficient allocation of emergency resources and scientific construction of urban emergency shelter systems.

[0003] In the prior art, facility site selection models are constructed from the perspective of mathematical programming and combined with the characteristics of emergency safety site selection problems. Among them, the research on single-objective site selection problems is the most in-depth and mature. Single-objective site selection problems focus on a single objective function, and according to different objectives, single-objective site selection problems can be divided into coverage problems, P-median problems, and P-center problems. With the deepening of research, the research model for emergency shelter site selection has evolved from single-objective to multi-objective, and different site selection optimization objectives and constraints have been proposed for emergency safety site selection to solve the scenarios and needs that may occur during disasters. Due to its complexity, the safety site layout optimization problem under multi-objective constraints is classified as an NP-hard problem, which makes it very difficult to solve using precise calculation methods. Currently, the algorithms commonly used in emergency shelter site selection optimization problems include genetic algorithms, particle swarm optimization algorithms, and ant colony optimization algorithms, but they have great limitations and are not suitable for solving large-scale complex site selection problems.

[0004] Deep reinforcement learning algorithms have more advantages than other algorithms in dealing with decision-making problems in complex and unknown environments, and have been gradually introduced into site selection problems to overcome the limitations of traditional optimization algorithms. Based on Markov decision processes, deep reinforcement learning can well solve some problems with high-dimensional features, and significantly reduce the need for manual parameter tuning compared to heuristic algorithms, with obvious advantages in optimization results and training efficiency. It has been applied to various practical scenarios. However, existing deep reinforcement learning algorithms, although improving the performance of handling complex problems, still need to be improved in terms of the instability of training results. SUMMARY

[0005] In view of the problems mentioned in the background art, the present application proposes an emergency protection site multi-objective location optimization method based on integrated reinforcement learning, improves the actor-critic network structure and introduces an integrated learning strategy, and enhances the robustness and practicality of the model through the aggregation of the Actor-Critic network.

[0006] Technical scheme: In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0007] An emergency protection site multi-objective location optimization method based on integrated reinforcement learning, comprising the following steps:

[0008] S1: Obtain the candidate point and demand point data in the region, and construct a multi-objective reinforcement learning problem;

[0009] S2: Perform Markov modeling based on a single-agent decision deep reinforcement learning problem;

[0010] S3: Construct an integrated reinforcement learning architecture;

[0011] S4: Train the constructed integrated reinforcement learning network;

[0012] S5: Obtain the location result using the integrated reinforcement learning framework and perform visual presentation.

[0013] As a preferred, in S1, the specific implementation process is:

[0014] There are m demand points, forming a set wherein, denotes the demand point i;

[0015] There are n candidate points, forming a set wherein, denotes the candidate point j, and each candidate point has a corresponding service radius.

[0016] As a preferred, the objective function of the multi-objective reinforcement learning problem is:

[0017]

[0018] wherein, denotes the population of the demand point i; denotes whether the candidate point j provides service to the demand point i; denotes the calculated distance from the demand point i to the candidate point j; denotes the value that needs to be maximized for the objective function.

[0019] As a preferred, in S2, the specific content is:

[0020] Markov modeling is performed, and a single-agent decision deep reinforcement learning problem based on a Markov decision process is performed; and a five-tuple is expressed.

[0021] Among them, S represents a state set, A represents an action set, P represents a state transition probability, R represents a reward function, represents a discount factor.

[0022] As a preferred, the reward function includes: the distance from each demand point to the nearest site point, the overlapping area ratio between each site point, and the total population that can be covered by all selected site points, each part is normalized, specifically:

[0023] ,

[0024] Among them, represents the distance between the emergency shelter and the demand point; represents the distance between the demand point and the nearest site at time t; m represents the number of demand points; represents the maximum distance in the scene;

[0025] ,

[0026] Among them, represents the overlap rate of protective services; represents the number of demand points covered by multiple sites at time t, represents the number of demand points covered by all sites at time t;

[0027] ,

[0028] Among them, represents the sum of the weighted population of demand points; represents the sum of the weighted population covered by all sites at time t, represents the total number of weighted populations;

[0029] ,

[0030] Among them, represents the final objective function; , and represent weight factors.

[0031] As a preferred, in S3, the specific implementation process is:

[0032] S31: Select the PPO algorithm based on strategy as the basic framework to calculate the strategy and value;

[0033] S32: fitting the policy and the value with the actor-critic neural network structure;

[0034] S33: calculating the total loss function;

[0035] S34: introducing the ensemble learning module in the PPO framework;

[0036] S35: introducing the adaptive optimization technique to accelerate the training efficiency.

[0037] As preferred, in S31, the specific implementation content is:

[0038] The PPO algorithm based on policy is selected as the basic framework, and the ensemble learning strategy is introduced. The strategy is the mapping from the state space to the action space. The calculation method of the policy and the value is:

[0039] ,

[0040] ,

[0041] Among them, represents the policy; represents the state The probability of taking action ; represents the parameters in the neural network that can be updated to fit the policy; represents the value function of the state at time , represents the parameters in the neural network that can be updated to fit the value; E represents the expectation; represents the immediate reward; represents the discount factor; represents the value function estimate of the state .

[0042] As preferred, in S32, the specific implementation content is: including a shared feature extraction network, an actor network and a critic network;

[0043] The shared feature extraction network is used to extract general features from the input state, including linear layers, ReLU activation layers and normalization layers;

[0044] The actor network is used to fit the policy function, including two residual blocks, each residual block containing linear layers, ReLU activation layers and normalization layers. The final layer of the actor network outputs the action probability distribution through function;

[0045] The critic network is used for fitting a value function, including two residual blocks, each of which contains a linear layer, a ReLU activation layer and a normalization layer, and the final layer of the critic network outputs a state value estimate through a linear layer.

[0046] As preferred, in S4, the specific implementation process is as follows:

[0047] S41: initialize the discount factor and the learning rate, and initialize the neural network;

[0048] S42: initialize the initial state according to the data condition of the simulation area;

[0049] S43: input the current state to the neural network for analysis, and output the action ;

[0050] S44: update the structure parameters and obtain the reward feedback from the preset reward function and the current state ;

[0051] S45: store the samples collected in the time step into the experience pool;

[0052] S46: update the network parameters;

[0053] S47: determine whether the learning process is completed, and if not, jump back to S42.

[0054] As preferred, in S5, the specific implementation process is as follows:

[0055] S51: input the demand point and the candidate point into the multi-objective integrated reinforcement learning model;

[0056] S52: the model outputs the site selection result;

[0057] S53: optimize the site selection result and perform visual expression.

[0058] Advantages: compared with the prior art, the present application has the following advantages:

[0059] (1) The present application is based on the basic framework of deep reinforcement learning, fully considers multiple optimization objectives, introduces the integrated learning strategy and adaptive optimization technology in view of the deficiencies of the existing emergency protection site optimization model, combines the spatial information technology based on remote sensing and geographic information system, the computer technology based on deep learning and reinforcement learning, and realizes the multi-objective site selection optimization of urban emergency protection sites: firstly, the candidate point and demand point data in the region are acquired, and a multi-objective reinforcement learning problem is constructed; secondly, Markov modeling is carried out for the multi-objective site selection problem of urban emergency protection sites; then, the integrated reinforcement learning architecture is constructed, the PPO algorithm based on strategy is taken as the basic framework, and the integrated learning strategy is introduced to aggregate the three Actor-Critic networks; then, the integrated reinforcement learning network is trained; finally, the site selection result is obtained by using the constructed integrated reinforcement learning framework, and visual presentation is carried out.

[0060] (2) The present application fully considers the characteristics of the multi-objective optimization model, selects the planning target combined with the actual situation, fully utilizes the advantages of deep reinforcement learning and integrated learning, has the characteristics of fast calculation speed, high model stability, good visualization effect and wide application in solving large-scale problems, provides an effective method for the multi-objective site selection optimization of urban emergency protection sites, and can be applied to multi-objective planning problems in large-scale scenes.

[0061] (3) The present application is reliable and practical, has good accuracy and practicability for the multi-objective site selection optimization of urban emergency protection sites, has good effect compared with the actual situation, and has good practicability and feasibility. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The flowchart of the optimization method of the present application is shown in the figure;

[0063] Figure 2 The framework diagram of the integrated reinforcement learning algorithm of the present application is shown in the figure;

[0064] Figure 3 The actor-critic network structure diagram of the integrated reinforcement learning algorithm of the present application is shown in the figure;

[0065] Figure 4 The flowchart of the integrated reinforcement learning network training of the present application is shown in the figure;

[0066] Figure 5 The reward change diagram in the training process of the integrated reinforcement learning algorithm of the present application is shown in the figure;

[0067] Figure 6 The emergency protection site selection quantity change diagram in the training process of the integrated reinforcement learning algorithm of the present application is shown in the figure;

[0068] Figure 7 The site selection result diagram of the present application is shown in the figure;

[0069] Figure 8 This is a comparison diagram of the cumulative probability and probability density before and after optimization of the present invention. DETAILED DESCRIPTION

[0070] The present invention will be further illustrated below with reference to specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0071] like Figures 1-8 As shown, the multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning provided in this embodiment introduces integrated learning strategies and adaptive optimization technology, combines spatial information technology based on remote sensing and geographic information systems, and computer technology based on deep learning and reinforcement learning, to achieve multi-objective site selection optimization for urban emergency protection sites. Specifically, the method includes the following steps:

[0072] S1. Obtain candidate point and demand point data in the area and construct a multi-objective reinforcement learning problem;

[0073] The location selection problem is described as:

[0074] There are m demand points, each with specific protection requirements, forming a set ;in, Represents demand point i.

[0075] There are n candidate shelters, and several facilities need to be selected as shelters to provide protection services, forming a set ,in, Denotes candidate point j, each candidate point has a specific service radius. When selecting a shelter, the following goals must be achieved:

[0076] (1) Minimize the total protection distance from the demand point to the protection station.

[0077] (2) Taking risk factors into consideration, maximize the total number of people served by the shelter.

[0078] (3) Minimize the overlap of service areas of protection stations.

[0079] To simplify the model's complexity, details that have little impact on model performance are omitted. Specifically, the impact of a site's population capacity on the site selection problem is not considered; rather, the site's capacity is assumed to be large enough to accommodate the population within its service radius. The site selection process allows for demand points that cannot be served by any site (i.e., blind spots), which is consistent with reality. However, as one of the optimization objectives, the model ensures that blind spots are minimized.

[0080] Table 1 Variables and parameters of the emergency protection site selection problem

[0081]

[0082] The objective function for constructing a multi-objective reinforcement learning problem is:

[0083]

[0084] in, represents the population of demand point i; Indicates whether candidate point j provides services to demand point i; Represents the calculated distance from demand point i to candidate point j; Indicates that the value of the objective function needs to be maximized.

[0085] The first item ( ) represents the total population that can be covered by all selected sites, and the second term ( ) represents the sum of the distances from each demand point to the nearest location point; the third item ( ) represents the overlapping area between the site points.

[0086] The constraints of the multi-objective reinforcement learning problem are:

[0087]

[0088]

[0089]

[0090]

[0091] in, Indicates demand points To the candidate point The calculated distance, Indicates demand points To the candidate point The actual distance; Indicates the maximum distance; if the demand point is within the service range of the selected site, the actual distance is used directly Otherwise, calculate the distance Will be assigned the maximum distance in the scene . It means that the distance between the demand point being served and the corresponding selected location point does not exceed the service radius. Indicates candidate points Is it a demand point? Provide services; , is a Boolean variable Provides binary constraints.

[0092] S2: Markov modeling is performed, based on single-agent decision deep reinforcement learning problem;

[0093] The problem is modeled as a single-agent decision deep reinforcement learning problem based on Markov decision process, and a five-tuple representation is adopted, where S represents the state set, A represents the action set, P represents the state transition probability, R represents the reward function, and γ represents the discount factor. Specifically:

[0094] State: The state of the agent contains three parts:

[0095] (1) Position matrix, size , m represents the total number of demand points; n represents the total number of candidate points; l represents the total number of selected points; the values in the matrix are represented by {0, 1}, indicating whether the demand point is within the service range of the site point.

[0096] (2) Population number, which is a vector of size .

[0097] (3) An indicator vector is used to record whether each candidate site is currently selected, containing the selection status of each site and an abort flag, with a size of , the values in which are represented by {0, 1}.

[0098] Action: At each time step, the agent selects a new site from the candidate sites according to the state and policy, denoted by k, representing the serial number of the selected candidate site .

[0099] State transition function: In the environment of the site selection problem, the transition is deterministic, i.e. given and , is uniquely determined. Therefore, ; where represents the state at time step t; represents the action at time step t; represents the probability of the environment transitioning to the next state after taking action in state .

[0100] Reward function: The reward function is a key part of the entire algorithm, which determines the action selection of the agent and further affects the quality of model training, so the reward function needs to accurately reflect the goal that the agent wants to learn.

[0101] In the multi-objective site selection problem of this embodiment, the reward function is composed of three parts: the distance from each demand point to the nearest site point, the proportion of the overlapping area between each site point, and the total population covered by all selected site points, each part is normalized:

[0102]

[0103] wherein, denotes the distance from demand point to the nearest site (i.e. the selected candidate point) at time t, if not served by any site, the distance is set to the maximum distance in the scenario.

[0104]

[0105] wherein, denotes the overlap rate of the protection service; denotes the number of demand points covered by multiple sites at time t, denotes the number of demand points covered by all sites at time t.

[0106]

[0107] wherein, denotes the sum of the weighted population numbers of demand points; denotes the sum of the weighted population numbers covered by all sites at time t, denotes the total number of weighted populations.

[0108]

[0109] wherein, denotes the final objective function; , and are weight factors adjusted according to the specific requirements of the problem.

[0110] S3: Construct an integrated reinforcement learning architecture;

[0111] Select the policy-based PPO algorithm as the basic framework, and introduce an integrated learning strategy to enhance the robustness of the model through the aggregation of three Actor-Critic networks.

[0112] S31: Select the policy-based PPO algorithm as the basic framework to calculate the policy and value;

[0113] Specifically, the policy is the mapping from the state space to the action space, and the policy is denoted by , which means taking action under state where, are the parameters in the neural network that can be updated to fit the policy. The value function estimate for state at time is denoted by are the parameters in the neural network that can be updated to fit the value. The policy and value are computed by:

[0114]

[0115]

[0116] where E denotes expectation; denotes the immediate reward; denotes the discount factor; denotes the value function estimate for state at time

[0117] S32: fitting the policy and value using an actor-critic neural network structure;

[0118] The PPO model fits the policy and value using an actor-critic neural network structure. It consists of a shared feature extraction network, an actor network, and a critic network.

[0119] The input dimension of the actor network is the size of the state space, and the output dimension is the size of the action space, which is used to fit the policy function; the input dimension of the critic network is the size of the state space, and the output dimension is 1, which is used to fit the value function.

[0120] To realize the gradient update of the policy and value functions, the model measures the advantage of the action relative to the average level through the advantage estimation function

[0121]

[0122] where, denotes the advantage estimation function; denotes the discounted cumulative return; denotes the value function for state at time denotes the discount factor, which is used to weigh the importance of future rewards; denotes a certain time step in the future; denotes the immediate reward obtained at future time step

[0123] ​​​​The shared feature extraction network contains three layers of linear layer, ReLU activation layer and normalization layer, which is used to extract general features from the input state.

[0124] The actor network and critic network are both composed of two residual blocks, each of which contains a linear layer, a ReLU activation layer and a normalization layer, and is enhanced by a residual connection to facilitate gradient transmission and training stability. The final layer of the actor network outputs an action probability distribution through a linear layer, while the final layer of the critic network outputs a state value estimate through a linear layer. Compared with traditional network structures, this design reduces computational redundancy through a shared feature extraction network, alleviates the gradient vanishing problem through a residual connection, and significantly improves the training performance in complex location selection scenarios.

[0125] In addition, a temperature parameter is added to the actor action to change the distribution of action probability through a Softmax function. For the initial probability of an action in the action space A , the probability after temperature adjustment is:

[0126]

[0127] where represents the initial probability of action a , and the unnormalized probability value after temperature adjustment T is represented by represents the initial probability of the i-th action in the action space A , and the unnormalized value after temperature adjustment is represented by is the temperature parameter, The higher the temperature parameter, the more uniform the action distribution, and the more the strategy tends to explore; The lower the temperature parameter, the more prominent the action selection, and the more the strategy tends to exploit. This parameter can achieve the trade-off between exploration and exploitation.

[0128] In the early stage of reinforcement learning training, the strategy needs to be explored more, while in the later stage, it needs to be exploited more. Therefore, a temperature decay mechanism is introduced, with the initial temperature set to 1.0, and the temperature complementary decayed by a rate of 0.995 at each step update until the minimum temperature is set to 0.1, realizing the transition of the model from high exploration to high exploitation and avoiding premature convergence of the model.

[0129] PPO is a hetero-strategy algorithm, which requires the learning agent to interact with the environment different from the agent interacting with the environment. In order to improve the utilization efficiency of data, PPO uses importance sampling method, so that the data generated by the old agent interacting with the environment can be used for the policy update of the new agent needing to learn. After completing one step of update, the new agent will copy the parameters to the old agent, continue the next step of interaction and update, and the importance sampling problem is repeatedly circulated. The problem of importance sampling is that the difference between the policy probability distribution of the new agent and the old agent will affect the sampling effect, and the larger difference is easy to lead to poor training results. The difference between different policy probability distributions is quantified as follows:

[0130]

[0131] wherein, represents the difference between different policy probability distributions; represents the probability of the new policy selecting action under state ; represents the probability of the old policy selecting action under state ; represents the parameters that can be updated in the new policy; represents the parameters that can be updated in the old policy.

[0132] S33: calculate the total loss function;

[0133] The core of PPO is to ensure the stability of training by limiting the amplitude of policy update, while retaining the advantages of policy gradient method.

[0134] The optimization objective of the actor network is based on the clipping ratio, specifically:

[0135]

[0136] wherein, represents the policy loss function of the actor network; represents the expectation; represents the advantage estimation function; is a clipping factor, and the clip() function represents clipping to limit the difference between the new policy and the old policy;

[0137] The critic network optimizes the state value function through mean square error, specifically:

[0138]

[0139] wherein, represents the value loss function of the critic network; represents the time the lower state the value function, denotes the discounted cumulative return; denotes the expectation.

[0140] To encourage exploration, an entropy regularization term is added, which is:

[0141]

[0142]

[0143] where, denotes the entropy regularization loss; denotes the expectation; denotes the entropy of the policy in state ; denotes the entropy of the policy ; denotes the probability of the policy choosing action in state ; denotes the action; denotes the state.

[0144] The total loss function is:

[0145]

[0146] where, , are hyperparameters.

[0147] The parameter update is performed by gradient descent, which is:

[0148]

[0149]

[0150] where, and are the learning rates of the actor network and the critic network, respectively. denotes the policy gradient of the actor network; denotes the policy gradient of the critic network; denotes the total loss function.

[0151] S34: Introduce an ensemble learning module in the PPO framework;

[0152] The introduced ensemble learning module includes: N parallel learning of policy and value function Each policy is initialized with different random seeds and updated by bootstrap sampling from the shared experience buffer generated by the ensemble policies interacting with the environment, reducing data correlation; the performance of each policy is evaluated by discounted average reward, specifically:

[0153]

[0154] wherein, represents the performance of the i-th policy; represents the average reward of the policy ; and is a decay factor, is the average reward of the policy . The ensemble policy is then generated by each policy based on the weighted average of the performance, specifically:

[0155]

[0156]

[0157] wherein, represents the ensemble policy; represents the parameters of the i-th policy; represents the total number of policies participating in the aggregation; represents the weight of the i-th policy; represents the performance of the i-th policy; represents the performance of the j-th policy.

[0158] S35: Introducing adaptive optimization techniques to accelerate training efficiency;

[0159] The action space is defined as selecting a candidate site point or stopping, and the space size is equal to the number of candidate site points plus one. Unlike the way of calculating the reward function when selecting a candidate point, when the model chooses to stop, the calculation method of the reward function is as follows: first, calculate the comprehensive score of the current selected site point, then for each unselected site point, calculate its hypothetical gain after being selected compared to the current one, take the maximum value (i.e. optimal marginal revenue) as the reward of the stop action. Such improvements add the timing and criteria of site selection to the model's learning process, which can further improve the speed and performance of training.

[0160] Since the model has strong ability to extract problem features, it can often fully converge the policy function in relatively few steps, so an early stopping mechanism is introduced, given a window size and a convergence threshold , record the rewards of the past training steps, calculate the variance of these rewards, the smaller the variance, the more stable the current training result. When the variance is less than At this time, the training of the entire model is stopped.

[0161] The model uses a mask to shield illegal actions at each step of action selection. On the one hand, an indication vector is constructed to record whether each candidate site point is selected, so as to prevent the selection of an action that has been selected, and to avoid empty operations and resource waste in training. On the other hand, a hard constraint is used to avoid selecting a point that is too close to the currently selected site point when selecting an action. Two site points are defined as being too close if:

[0162]

[0163] wherein and are the service radii of the site points and , respectively. represents the distance between the site points and .

[0164] S4: training the constructed integrated reinforcement learning network;

[0165] training the constructed integrated reinforcement learning network. Specifically:

[0166] S41: initializing the discount factor, learning rate, and other hyperparameters, and initializing the neural network, wherein the weights and other parameters of each learner neural network are initialized differently. By randomly setting different parameters, individual learners that differ greatly from each other are generated.

[0167] S42: initializing the initial state according to the data situation of the simulation area;

[0168] S43: inputting the current state into the neural network for analysis, and outputting a designed action ;

[0169] S44: updating the structure parameters and obtaining reward feedback and the current state from the preset reward function;

[0170] S45: storing the samples collected in the time step into the experience pool;

[0171] S46: updating the network parameters;

[0172] S47: determining whether the learning process is complete, and if not, jumping back to S42.

[0173] S5: obtaining the site selection result using the integrated reinforcement learning framework, and performing visual presentation.

[0174] The site selection result is obtained by using the integrated reinforcement learning framework and is visually presented. Specifically:

[0175] S51: input the demand points and candidate points into the multi-objective integrated reinforcement learning model;

[0176] S52: the model outputs the number of selected shelters, i.e., the site selection result;

[0177] S53: the site selection result is optimized and visually expressed.

[0178] In this embodiment, the method of the present application is compared with some existing methods, and the comparison algorithms include: SA algorithm and PPO algorithm.

[0179] In the training time index, the training time of PPO and the method of the present application is similar, and is in the same order of magnitude, which is much smaller than the training time of SA.

[0180] In the training performance index, the total score of PPO remains the highest among all algorithms, which is improved by 0.3%-11.4% compared with SA and 5.1%-20.3% compared with PPO, especially in the distance score and population score.

[0181] In the training robustness index, 5 experiments are respectively performed, and the results show that the training results of PPO have large differences, and the results of SA and the method of the present application have no differences. The results show that although SA has better performance, as a heuristic swarm intelligence algorithm, the training speed is much lower than that of the reinforcement learning algorithm, which leads to its lack of application value in large-scale problems in real scenarios; the training time of the algorithm of the present application is higher than that of PPO which only needs to train one learner, but the result data proves that the algorithm of the present application only sacrifices a small part of the training speed to achieve better training effect, and with the growth of the data size, the speed disadvantage of the algorithm of the present application is continuously reduced, and the performance advantage is more and more significant. Therefore, the algorithm of the present application has great application potential in large-scale problems.

[0182] In this embodiment, the convergence speed and learning effect of the method of the present application are also evaluated. As shown in Figure 5 , the reward function feedback value in the training process is shown with the change of the training round number, the learning rate of the model is fast, and the optimal solution is approached at about the 100th round, but since the exploration rate is still high at this time, there is some fluctuation from the 100th to the 700th round; at about the 1000th round, with the gradual transition from high exploration to high development, the reward and strategy become stable, and finally converge to the optimal solution, and the average reward of the last 30 rounds is 14.10.

[0183] As shown in Figure 6As shown in the figure, the number of emergency shelter selection in the training process is shown, and the reward function is similar. The characteristics of high exploration in the early stage lead to the shock of the number of selections, and the stable strategy makes the number steadily reduce and converge. The above training processes reflect the strong feature extraction ability and high stability of the algorithm.

[0184] On the other hand, the distance distribution of each settlement to its nearest emergency shelter is counted, and the data before and after optimization using the method of the application are compared as shown in the figure. Figure 8 The average distance after optimization is 2.518 kilometers, the median distance is 2.351 kilometers, more than 53% of the settlements have a nearest distance less than 2.5 kilometers, an increase of 21% compared with before optimization; more than 95% of the settlements have a nearest distance less than 5 kilometers, an increase of 25% compared with before optimization. The method of the application can obtain good emergency shelter layout optimization results, and has fast solving speed and stable solving results.

[0185] The above only describes the preferred embodiments of the application, and it should be noted that for those skilled in the art, without departing from the principles of the application, several improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the application.

Claims

1. A multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning, characterized by: The following steps are involved: S1: Obtain data on candidate points and demand points in the region and construct a multi-objective reinforcement learning problem; S2: Perform Markov modeling based on single-agent decision-making deep reinforcement learning problems; S3: Building an integrated reinforcement learning architecture; S4: Training the multi-built ensemble reinforcement learning network; S5: Use the integrated reinforcement learning framework to obtain the site selection results and perform visualization.

2. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 1 is characterized by: In S1, the specific implementation process is: There are m demand points, forming a set ,in, represents demand point i; There are n candidate points, forming a set ,in, Represents candidate point j, and each candidate point has a corresponding service radius.

3. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 2 is characterized by: The objective function for constructing a multi-objective reinforcement learning problem is: ； in, represents the population of demand point i; Indicates whether candidate point j provides services to demand point i; Represents the calculated distance from demand point i to candidate point j; Indicates that the value of the objective function needs to be maximized.

4. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 1 is characterized by: In S2, the specific content is: Perform Markov modeling and deep reinforcement learning problems based on Markov decision process for single agent decision making; and use quintuples Express; Among them, S represents the state set, A represents the action set, P represents the state transition probability, and R represents the reward function. Represents the discount factor.

5. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 4 is characterized by: The reward function includes: the distance from each demand point to the nearest location, the overlap ratio between each location, and the total population that can be covered by all selected locations. Each part is normalized, specifically: ， in, Indicates the distance between the emergency protection site and the demand point; Denotes the demand point at time t The distance to the nearest place; m represents the number of demand points; Indicates the maximum distance in the scene; ， in, represents the overlap ratio of protection services; represents the number of demand points covered by multiple sites at time t, represents the number of demand points covered by all sites at time t; ， in, represents the sum of weighted populations of demand points; represents the weighted population sum of all places covered at time t, represents the total weighted population; ， in, represents the final objective function; 、 and Represents the weight factor.

6. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 1 is characterized by: In S3, the specific implementation process is: S31: Select the policy-based PPO algorithm as the basic framework to calculate the strategy and value; S32: Using actor-critic neural network structure to fit strategy and value; S33: Calculate the total loss function; S34: Introducing an integrated learning module into the PPO framework; S35: Introducing adaptive optimization technology to accelerate training efficiency.

7. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 6 is characterized by: In S31, the specific implementation content is: The policy-based PPO algorithm is selected as the basic framework, and an integrated learning strategy is introduced. The strategy is the mapping from state space to action space. The calculation method of strategy and value is: ， ， in, express strategy; Indicates status Take action probability; The parameters that can be updated in the neural network representing the fitted strategy; Indicates time Lower state The value function of The updated parameters in the neural network represent the fitted value; E represents the expectation; Indicates immediate reward; represents the discount factor; Indicates status The value function estimate of .

8. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 6 is characterized by: In S32, the specific implementation content is: including a shared feature extraction network, an actor network, and a critic network; The shared feature extraction network is used to extract common features from the input state, which includes a linear layer, a ReLU activation layer, and a normalization layer; The actor network is used to fit the policy function and consists of two residual blocks. Each residual block contains a linear layer, a ReLU activation layer, and a normalization layer. The final layer of the actor network is The function outputs the probability distribution of actions; The critic network is used to fit the value function and consists of two residual blocks. Each residual block contains a linear layer, a ReLU activation layer, and a normalization layer. The final layer of the critic network outputs the state value estimate through the linear layer.

9. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 1 is characterized by: In S4, the specific implementation process is: S41: Initialize the discount factor and learning rate, and initialize the neural network; S42: Initialize the initial state according to the data of the simulation area; S43: Current status Input to the neural network for analysis and output action ; S44: Update structural parameters and obtain reward feedback from the preset reward function and the current status ; S45: storing the samples collected in the time step into the experience pool; S46: Update network parameters; S47: Determine whether the learning process is completed, otherwise jump back to S42.

10. The multi-objective site selection optimization method for emergency protection sites based on integrated reinforcement learning according to claim 1, characterized in that: In S5, the specific implementation process is: S51: Input the demand points and candidate points into the multi-objective integrated reinforcement learning model; S52: Model outputs site selection results; S53: Optimize the site selection results and visualize them.

Images