Method and device for determining congestion relief strategy based on reinforcement learning and digital twin

Abstract

The present invention provides a method and device for determining a crowd relief strategy based on reinforcement learning and digital twins, the method comprising: in a virtual simulation environment built based on digital twins, inputting the flow data of each area within the target range into a target intelligent agent model to obtain a target action set at the current moment, the target action set being the optimal action sequence at the current moment output by the target intelligent agent model; based on the target action set, obtaining an optimal crowd relief strategy, the optimal crowd relief strategy being a strategy that minimizes the average crowding degree of each area. The method and device for determining a crowd relief strategy based on reinforcement learning and digital twins provided by the present invention can quickly obtain the optimal crowd relief strategy, thereby reducing the crowding degree of people in each area.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a method and device for determining a congestion relief strategy based on reinforcement learning and digital twins. Background Art

[0002] Crowd management has important research significance in the service operation of public places. For pedestrians in public places, the degree of crowding has a great impact on their daily travel experience, so the flow of people in public places needs to be properly regulated under normal travel.

[0003] Existing simulation designs are all special events designed according to their own specific needs. Although there is a growing interest in strategies to control and regulate pedestrian flow, there is currently no comprehensive framework to study and integrate pedestrian-specific dynamic management systems and dynamic control strategies.

[0004] Pedestrian flow simulation and prediction has been widely covered by many different modeling schemes, but control and guidance strategies remain to be explored. In public places, traditional dynamic management algorithms that aim to reduce pedestrian delay time cannot meet the needs of keeping each area of ​​a public place within a safe and controllable capacity. Most of these control strategies are modeled with the ultimate optimization goal of reducing pedestrian delay time, which cannot alleviate the congestion in each area. Summary of the invention

[0005] The present invention provides a method and device for determining a congestion relief strategy based on reinforcement learning and digital twins, so as to solve the technical problem that pedestrian control strategies in the prior art cannot quickly and effectively obtain the degree of crowd relief.

[0006] The present invention provides a method for determining a congestion relief strategy based on reinforcement learning and digital twins, comprising:

[0007] In a virtual simulation environment built based on digital twins, the flow data of people in each area within the target range is input into the target intelligent agent model to obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model;

[0008] Based on the target action set, an optimal congestion relief strategy is obtained, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area;

[0009] The target agent model is determined by the following steps:

[0010] Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions;

[0011] Based on the first action, the second action and the third action, the intelligent agent model is trained to obtain a trained target intelligent agent model, wherein the first action is any action in the action space, the second action is an action determined based on the demonstration data, and the third action is an action determined based on the intelligent agent model.

[0012] In some embodiments, before training the agent model based on the first action, the second action, and the third action, the method further includes:

[0013] When the exploration probability is greater than 0 and less than a preset exploration weight, determining the first action;

[0014] When the exploration probability is greater than or equal to the preset exploration weight and less than a dynamic influence factor, determining the second action, where the dynamic influence factor is a parameter selected during each round of training of the agent model;

[0015] When the exploration probability is greater than or equal to the dynamic impact factor and less than or equal to 1, the third action is determined.

[0016] In some embodiments, the dynamic impact factor is determined by the following formula:

[0017]

[0018] Among them, y ep is the dynamic impact factor, ∈ is the preset exploration weight, β is the preset hyperparameter, ep is the training round of the agent model, ep_all is the total training round of the agent model, r ep is the average reward of the ep-th training.

[0019] In some embodiments, the state space is: S t ={S 1,t , S 2,t , S 3,t , ..., S i,t},S t represents the set of people in each area at time t, S i,t It is determined by the following formula:

[0020]

[0021] Among them, S i,t represents the number of people in the ith area at time t, N i represents the i-th, n i,j,t It represents the number of people in the jth road section in the i-th area at time t.

[0022] In some embodiments, the crowd diversion action includes at least one of the following:

[0023] issuing electronic vouchers to terminals located in each of the areas, the electronic vouchers being used to indicate consumption areas;

[0024] providing a route guidance tool in each of said areas;

[0025] Obstacle guiding means are provided in each of the areas.

[0026] In some embodiments, the reward value function is:

[0027]

[0028] Among them, r t is the reward value function, the r t represents the congestion degree of each area in the time period t to t+w, w represents the action interval time, S n-max represents the number of people in the nth area who reach the crowding threshold, S n,m represents the number of people in the nth area at the mth moment, r m,n Represents the reward of the nth area at the mth moment.

[0029] The present invention also provides a device for determining a congestion relief strategy based on reinforcement learning and digital twins, comprising:

[0030] A first determination module is used to input the crowd flow data of each area within the target range into the target intelligent agent model in a virtual simulation environment built based on the digital twin, and obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model;

[0031] A second determination module is used to obtain an optimal congestion relief strategy based on the target action set, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area;

[0032] The target agent model is determined by the following steps:

[0033] Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions;

[0034] Based on the first action, the second action and the third action, the intelligent agent model is trained to obtain a trained target intelligent agent model, wherein the first action is any action in the action space, the second action is an action determined based on the demonstration data, and the third action is an action determined based on the intelligent agent model.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a method for determining a congestion relief strategy based on reinforcement learning and digital twins as described in any one of the above-mentioned methods.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for determining a congestion relief strategy based on reinforcement learning and digital twins as described in any one of the above.

[0037] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a method for determining a congestion relief strategy based on reinforcement learning and digital twins as described in any one of the above.

[0038] The method and device for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention are used to stably and quickly obtain the optimal congestion relief strategy through the design of reinforcement learning algorithms, the design of optimized exploration spaces, the design of neural network training frameworks, and the training and learning design of models. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of a method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention;

[0041] Figure 2It is a schematic diagram of model training for applying the method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention;

[0042] Figure 3 It is a schematic diagram of a simulation system that applies the method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention;

[0043] Figure 4 It is a schematic diagram of the construction process of a congestion relief strategy model using the congestion relief strategy determination method based on reinforcement learning and digital twins provided by the present invention;

[0044] Figure 5 It is one of the simulation schematic diagrams of the method for determining the congestion relief strategy based on reinforcement learning and digital twin provided by the present invention;

[0045] Figure 6 This is the second simulation schematic diagram of the method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention;

[0046] Figure 7 It is a structural schematic diagram of a device for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention;

[0047] Figure 8 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The execution subject of the method for determining the congestion relief strategy based on reinforcement learning and digital twins provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a teller machine or a self-service machine, etc., which is not specifically limited by the present invention.

[0050] The following takes the method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention executed by a computer as an example to explain the technical solution of the present invention in detail.

[0051] Figure 1 A flow chart of a method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention. Figure 1 The present invention provides a method for determining a congestion relief strategy based on reinforcement learning and digital twins, which may include: step 110.

[0052] Step 110: In a virtual simulation environment built based on digital twins, the crowd flow data of each area within the target range is input into the target intelligent agent model to obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model.

[0053] Step 120: Based on the target action set, an optimal congestion relief strategy is obtained, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area.

[0054] Before step 110, the target agent model is determined by the following steps:

[0055] 1. Based on the state space, action space and reward value function in the simulation cycle, the intelligent agent model is constructed. The state space is the set of people in each area, the action space is the set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion of each area within the action interval time. The action interval time is the time difference between the execution of two adjacent crowd diversion actions. It can be understood that the number of people in each area in the simulation cycle can be real data or simulated data, which is not specifically limited here.

[0056] The congestion relief strategy determination method based on reinforcement learning and digital twins provided by the present invention constructs an intelligent agent model based on a reinforcement learning algorithm.

[0057] It should be noted that reinforcement learning is inspired by the trial and error method in animal learning, and uses the reward value obtained by the interaction between the agent and the environment as a feedback signal to train the agent. Reinforcement learning can generally be represented by a Markov decision process (MDP), the main elements of which include (S, A, R, T, γ).

[0058] Among them, S represents the state of the environment, A represents the action taken by the agent, R represents the reward value obtained, T represents the state transition probability, and γ represents the discount factor.

[0059] The agent’s strategy π represents a mapping from state space to action space. t When , take action a according to strategy π t (a t ∈A), and then transfer to the next state S according to the state transition probability T t+1 , and receive the reward value r from the environment feedback t (r t ∈R).

[0060] The goal of reinforcement learning is to continuously optimize the agent's strategy to obtain the maximum reward value. The agent's reward value function and action value function are V(S t ) and Q(S t , a t ), used to evaluate the agent in state S t The optimal strategy of the agent can be obtained by optimizing the value function.

[0061] Deep Reinforcement Learning (DRL) algorithms are mainly divided into two categories: value function algorithms and policy gradient algorithms.

[0062] The value function algorithm indirectly obtains the agent's strategy by iteratively updating the value function. When the value function iteration reaches the optimal, the agent's optimal strategy is obtained through the optimal value function.

[0063] The policy gradient algorithm directly uses the function approximation method to establish a policy network, selects actions through the policy network to obtain reward values, and optimizes the policy network parameters along the gradient direction to obtain the optimized policy to maximize the reward value.

[0064] The present invention can establish a simulation scheme for pedestrian movement in public places, design a simulation environment based on a pedestrian movement model, and propose an optimization scheme for alleviating pedestrian congestion in public places through the design and optimization of a reinforcement learning algorithm based on a deep Q-network (DQN), the design of a training framework for the DQN model, and network training. The method can quickly learn a congestion relief strategy in a simulation environment for public places, and the DQN model can also converge better and faster.

[0065] Before building the agent model, the environment is modeled first. The environment model is a multi-layer map that contains static map information and crowd-related information. The static map information describes the physical properties of the environment, and the crowd-related information includes crowd location information, crowd quantity information, crowd density information, etc.

[0066] The physical properties of the environment are described abstractly through a map. The map can be a grid of squares, which can include squares corresponding to different areas, and the agent can move through the regional squares. For example, the area corresponding to a public place can be divided into several grids, and pedestrians can move in each grid.

[0067] Pedestrians flow in public places based on crowd movement models. In the absence of interference and guidance measures, there is congestion, specifically, there is a maximum number of people in a certain area that exceeds the safe and controllable range.

[0068] In this embodiment, the agent model is first defined as follows:

[0069] 1. Define the state of the agent model

[0070] The state of the agent model is defined based on the number of people in each area, and all state sets can constitute the state space.

[0071] In some embodiments, the state space is: S t ={S 1,t , S 2,t , S 3,t , ..., S i,t},S t represents the set of people in each area at time t, S i,t It is determined by the following formula:

[0072]

[0073] Among them, S i,t represents the number of people in the ith area at time t, N i represents the i-th, n i,j,t It represents the number of people in the jth road section in the i-th area at time t.

[0074] In actual execution, state S t It represents the set of people in each area at time t. The total number of areas I can be set according to the needs. The value range of time t is [0, T-1], and T represents the total number of running moments in a simulation.

[0075] The congestion relief strategy determination method based on reinforcement learning and digital twins provided by the present invention provides flexible pedestrian movement data and has certain openness and flexibility.

[0076] 2. Define the actions of the agent model

[0077] The crowd diversion actions performed in each area are defined as the actions of the intelligent agent model, and all action sets can constitute the action space. The crowd diversion actions are the incentive signals of the intelligent agent model.

[0078] In some embodiments, the crowd diversion action includes at least one of the following:

[0079] issuing electronic vouchers to terminals located in each area, the electronic vouchers being used to indicate consumption areas;

[0080] Set up route guidance tools in each area;

[0081] Set up obstacle guidance tools in each area.

[0082] In actual execution, the action a of the agent model t Defined as the next action a at time t t+w Only action a is executed in [t, t+w-1] t .

[0083] The action space can be represented by A. A can include all action sets in the action space, and each action represents the execution of crowd diversion measures in each area. w represents the action interval time, which is a fixed value.

[0084] Crowd diversion measures can be to issue electronic coupons to terminals located in each area. The electronic coupons can be consumption coupons, fast-track tickets for attractions, or store coupons, etc. The above electronic coupons can probabilistically guide tourists to flow to designated areas.

[0085] Consumption vouchers or store coupons can guide pedestrians to move to target consumption areas, and scenic spot fast-track tickets can guide pedestrians to move to target scenic spots. The present invention only considers the traffic-attracting effect of electronic vouchers, and does not consider economic losses.

[0086] For example, by issuing consumption coupons to selected pedestrians in the physical proximity area, pedestrians will choose with probability p whether to use the consumption coupons to go to the destination before the next action selection after obtaining the consumption coupons. Set the simulation time to 540 minutes, and issue consumption coupons every 30 minutes. Each time the consumption coupons are issued, the issuing area, quantity and type need to be set. i The type of consumption coupon means that pedestrians will get certain discounts when they go to the area with serial number i. The action space is formed by the combination of different setting parameters.

[0087] The crowd diversion measure may also be to set up a route indication tool in each area. The route indication tool may be, for example, a crowd evacuation route sign or a robot guidance tool.

[0088] Crowd diversion measures may also include setting up obstacle guidance tools in each area. The obstacle guidance tools may be, for example, roadblocks or gates.

[0089] Understandably, the flow of people can of course be regulated by guiding people.

[0090] The congestion strategy relief method provided by the present invention effectively solves the strategy problem of using diversion measures to guide pedestrian flow in the field of pedestrian flow traffic control, and effectively reduces the congestion level of various areas in public place scenes.

[0091] 3. Define the reward value function of the intelligent model

[0092] The reward function of the agent model can be defined as the crowdedness of each area within the action time interval.

[0093] In some embodiments, the reward value function is:

[0094]

[0095] Among them, r t is the reward value function, r t represents the congestion degree of each area in the period from t to t+w, w represents the action interval time, S n_max represents the number of people who reach the crowding threshold in the nth area, S n,m represents the number of people in the nth area at the mth moment, r m,n Represents the reward of the nth area at the mth moment.

[0096] In actual implementation, in S n,m n_max In the case of m,n is 1; in S n,m ≥S n_max In the case of m,n is -1.​

[0097] 4. Define the reinforcement learning selection algorithm.

[0098] After the state space, action space, and reward value function are defined, the agent model can be constructed. The DQN algorithm based on the value function is used, and the three-layer convolutional neural network is used to construct the value network. In addition, the memory pool and training pool size are initialized.

[0099] In the action selection during the training process, the prior exploration strategy is introduced, that is, the action selection function in the agent exploration process is optimized, and the agent model is updated through the loss function. The specific Loss function is:

[0100] Loss=(r t +γQ max (a′,s′)-Q(a,s)) 2

[0101] Among them, r t is the reward value function, γ is the discount factor, Q(a,s) is the expected benefit of taking action a(a∈A) in the state s(s∈S) at a certain time t, and Q(a′,s′) is the expected benefit of taking action a′(a′∈A) in the state s′(s′∈S) at the previous time t′.

[0102] The method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention can improve the training speed and training quality of the intelligent agent model based on an exploration-optimized reinforcement learning algorithm based on demonstration data.

[0103] 2. Based on the first action, the second action and the third action, the intelligent agent model is trained to obtain a trained target intelligent agent model, where the first action is any action in the action space, the second action is an action determined based on the demonstration data, and the third action is an action determined based on the intelligent agent model.

[0104] It is understandable that the reinforcement learning algorithm learns a better strategy in different tasks through trial and error when the intelligent agent interacts with the environment.

[0105] However, the balance between exploration and exploitation. Blindly choosing a strategy without sufficient exploration will bring certain problems, because this will cause the model to fall into a local optimum or even fail to converge.

[0106] Most traditional reinforcement learning methods rely on heuristic exploration strategies, such as ∈-greedy algorithms and parameter noise methods. When the rewards of a task are few and sparse, the agent must visit a large number of states to obtain some meaningful information through trial and error. This long-standing problem leads to slow convergence and even failure of the ongoing task.

[0107] Therefore, random exploration is not a suitable solution for tasks with complex exploration space.

[0108] When the DQN algorithm explores the strategy space, it uses a greedy strategy. However, this method makes it difficult for the agent to find the optimal solution when the strategy space is very large. In addition, in the process of exploring the agent and the environment, it is easy to only seek a local optimal solution. Therefore, multiple random action explorations are required, which is time-consuming and computationally intensive.

[0109] Considering that there is a solution to fix the optimization exploration space through demonstration data, the details are as follows:

[0110]

[0111] Among them, P(s t ) is the action probability of the demonstration data learned by the Sequence Generative Adversarial Nets with Policy Gradient (SeqGAN). This method causes the exploration space to be limited to the demonstration data, resulting in the inability of the agent to explore more possibilities through open exploration.

[0112] In some embodiments, before training the agent model based on the first action, the second action, and the third action, the method further includes:

[0113] When the exploration probability is greater than 0 and less than a preset exploration weight, determining a first action;

[0114] When the exploration probability is greater than or equal to the preset exploration weight and less than the dynamic influence factor, determine the second action, where the dynamic influence factor is a parameter selected during each round of training of the agent model;

[0115] When the exploration probability is greater than or equal to the dynamic influence factor and less than or equal to 1, the third action is determined.

[0116] The present invention proposes to combine the traditional greedy strategy with the optimization exploration space strategy based on demonstration data in the exploration, and invents a dynamic influencing factor to dynamically regulate the exploration weight of the open exploration and optimization exploration space. The specific interactive action selection method is as follows:

[0117]

[0118]

[0119] a gd =π * (a|s)

[0120] Among them, a t is the action at time t, p is the exploration probability, ∈ is the preset exploration weight, y ep is the dynamic impact factor, a rd is the first action, a gd For the second action, a ag For the third action, s t+1 It is the state at the next moment t+1 after moment t.

[0121] a rd is any action randomly selected in the action space.

[0122] a gd is the action determined based on the demonstration data, that is, the guidance action given based on the demonstration data. Specifically, a gd Can refer to actions that can have a positive impact on congestion relief based on human management experience.

[0123] a ag is the action determined based on the agent model. * is a general strategy for learning from demonstration data.

[0124] The present invention adopts strategy exploration optimization based on prior knowledge, and the intelligent agent model can converge better than the traditional DQN model using strategy. At the same time, the application of reinforcement learning method and the setting of state, reward, action, etc. are realized in crowded scenes.

[0125] The method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention optimizes the exploration space by giving the intelligent agent action exploration guidance based on demonstration data and a balance between random actions, so that the intelligent agent can adapt to the environment better and faster.

[0126] In this step, the intelligent agent model is optimized through multiple rounds of training to obtain the state and feedback actions, and finally the target intelligent agent model is obtained.

[0127] In step 110, in a virtual simulation environment built based on digital twins, the crowd flow data of each area within the target range is input into the target intelligent agent model, that is, the state at a moment is input into the target intelligent agent model. Through a single interaction, the optimal action at the current moment output by the target intelligent agent model can be obtained. Through multiple interactions, the optimal action sequence at the current moment output by the target intelligent agent model, that is, the target action set, can be obtained.

[0128] The target range is a pre-selected movable range of a crowd, and the target range can be divided into multiple areas.

[0129] In step 120, based on the target action set, an optimal congestion relief strategy can be obtained. The optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area.

[0130] In actual implementation, the crowd flow data of each area is first obtained and input into the target agent model. The crowd flow data can be the number of people, crowd density, crowd location or crowd movement trajectory in each area.

[0131] Based on the target agent model, the optimal congestion relief strategy in different scenarios can be obtained. For example, the distribution of consumer coupons every 30 minutes can minimize the average congestion during the regulation process through this regulation guidance. Each time the consumer coupons are distributed, the distribution area, quantity and type need to be set.

[0132] The congestion relief strategy determination method based on reinforcement learning and digital twins provided by the present invention is used to stably and quickly obtain the optimal congestion relief strategy through the design of reinforcement learning algorithm, the design of optimization exploration space and the design of neural network training framework and the training and learning design of the model.

[0133] In some embodiments, the dynamic impact factor is determined by the following formula:

[0134]

[0135] Among them, y ep is the dynamic influencing factor, ∈ is the preset exploration weight, β is the preset hyperparameter, ep is the training round of the intelligent model, ep_all is the total training round of the intelligent model, r ep is the average reward of the ep-th training.

[0136] In actual implementation, the discriminator weights are optimized from w through the following loss function i to w i+1 :

[0137]

[0138] Among them, it means represents expectation, τ i represents the strategy to be learned, τ E Denotes the demonstration data strategy, D w (s,a) represents the probability that the discriminator identifies the policy to be learned.

[0139] pass Optimizing the generator θ t to θ t+1 .

[0140] like Figure 2 As shown, Indicates the Nth real trajectory data, indicating that the data format is<s,a> τ N Indicates the Nth trajectory data generated by the generator, indicating that the data format is the same as<s,a> .

[0141] Considering that the action exploration guidance strategy based on demonstration data is a general strategy, the strategy model of the agent in the training process will gradually exceed the action guidance strategy model based on demonstration data. After that, the reference value provided by the guidance action based on demonstration data to the agent will gradually decrease. At this time, consider reducing the frequency of guidance actions when the agent is exploring the space. A dynamic influence factor formula based on the average reward impact is proposed:

[0142]

[0143] Among them, y ep is the dynamic influencing factor, ∈ is the preset exploration weight, β is the preset hyperparameter, ep is the training round of the intelligent model, ep_all is the total training round of the intelligent model, r ep is the average reward of the ep-th training.

[0144] The congestion relief strategy determination method based on reinforcement learning and digital twins provided by the present invention enables the intelligent agent model to quickly learn the optimal congestion relief strategy.

[0145] In some embodiments, based on the concept of digital twins, a passenger flow simulation system for public places is proposed. By mapping the real world into the virtual simulation world, strategies to alleviate congestion are studied in the virtual simulation world.

[0146] Figure 3 A schematic diagram of a simulation system for applying the method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the present invention. Figure 3 The simulation system designed by the present invention mainly consists of a data acquisition module, a model building module, a virtual world simulation module, a business configuration module and a strategy learning module.

[0147] Through the independent development of the model building module and the business configuration module, the configuration of strategy simulation scenarios for various congestion relief scenarios is completed. In the scenario of alleviating passenger congestion in public places, reinforcement learning is applied in this simulation system to explore strategies and interact with the simulation system.

[0148] The data collection module mainly includes obtaining crowd movement information and environmental information through devices, such as each person's personal trajectory and two-dimensional floor plans of public places. In addition, environmental data information includes hardware information required for subsequent strategy learning.

[0149] In the entire simulation system, due to the privacy of the data, the collection of real crowd data in the data collection module is not necessary. If there is an encrypted crowd movement model generator provided by the data company, it can also be used.

[0150] The model building module mainly includes crowd flow modeling and environmental modeling. Modeling is performed based on the data provided by the data acquisition module, mainly including crowd flow modeling and environmental modeling.

[0151] In the environmental modeling, the movable position of the area, the attributes of buildings such as service facilities, and all other attributes that can be added are defined. Crowd flow modeling is not necessary. Due to the privacy of the data, a third-party encrypted crowd flow generation model can be used directly.

[0152] The virtual world simulation module uses the crowd flow model and the environment model as the driving engine of the virtual simulation world, driving the self-operation of the simulation system. Through programming of the simulation environment entities and the physical environment, it adapts to the model access of various driving logics, such as the import of historical data, Markov transition probability matrix and LSTM model prediction logic.

[0153] The passenger flow simulation environment for public places provides a flexible pedestrian movement driver interface, which can set the urban road network and build service facility models. It has a certain degree of openness and flexibility, and provides the possibility of data interface in addition to traveler attributes and service facility attributes.

[0154] The business configuration module is built for the upper-level virtual world simulation model. The business configuration module is an open module that uses the digital stream data of the virtual simulation world to program and configure the required services. For example, the data monitoring service can display the number of people in each area of ​​the virtual public place and generate a heat map of the flow of people.

[0155] The strategy interface service module can transfer the action instructions fed back from the strategy learning module to the simulation module for crowd control.

[0156] The visualization service module can visualize virtual public places in the terminal. Through the coordination and docking of the business configuration and virtual simulation module, high scalability for crowd relief in public places can be achieved.

[0157] The scalability here refers to two aspects: one is the high scalability of the business configuration module, which can generate a variety of different services through programming. The digital virtual simulation model mapped through the real world can theoretically have any data flow required by the business configuration. The other is the high scalability for optimizing congestion relief strategies.

[0158] Through modeling of the environment, various diversion measures can be set up in theory, such as issuing consumption coupons to divert traffic, setting up roadblocks and guiding personnel, etc. The strategy interface service connects with the coordination of pedestrian flow modeling and environmental modeling by configuring interface parameters to achieve simulation.

[0159] The strategy learning module is also called the decision optimization module. It optimizes decisions by calling various services of the business configuration module. Taking reinforcement learning as an example, the intelligent agent interacts with the simulation environment through data monitoring services and strategy interface services to obtain status and feedback actions, etc., and optimizes the decision model through multiple rounds of iterations.

[0160] The simulation system provided by the present invention has good scalability and adaptability to algorithms such as reinforcement learning. The present invention designs a public place simulation system based on digital twins, including a data acquisition module, a model building module, a virtual world simulation module, a business configuration module and a strategy learning module. The programmability of the business configuration module and the environment modeling makes the simulation system highly scalable. Through the independent development of the model building module and the business configuration module, the configuration of various strategy simulation scenarios in the congestion relief scenario is completed.

[0161] Figure 4 A schematic diagram of the construction process of a congestion relief strategy model for applying the congestion relief strategy determination method based on reinforcement learning and digital twins provided by the present invention. Figure 4 The present invention provides a method for constructing a congestion relief strategy model, which may include the following steps:

[0162] Step 410: Generate a pedestrian movement model and a public place physical model based on real-world data.

[0163] Step 420: Based on the menge design simulation environment, the pedestrian movement model and the physical model are imported. The menge simulation diagram is initialized as follows: Figure 5 As shown, the menge simulation diagram runs as follows Figure 6 In actual implementation, pedestrian movement model and physical model can also be visualized and simulated based on Unity.

[0164] Step 430: Design the DQN reinforcement learning algorithm and define the state, action, reward, and loss functions.

[0165] Step 440: Design a prior knowledge-based policy exploration method.

[0166] Step 450: Connect the model to the simulation environment through an interface to enable interaction between the agent and the environment.

[0167] Step 460: Obtain the crowd alleviation strategy model corresponding to the public place.

[0168] In actual execution, the preparation phase requires the following steps:

[0169] Obtain the pedestrian flow movement data, environmental information, or scenario information of the public place;

[0170] Construct the physical model of the public place scenario and the pedestrian movement model of the public place;

[0171] Load the model into the simulation module;

[0172] Perform business programming configuration and define the parameters of each service interface;

[0173] Initialize the memory pool of reinforcement learning, the scale of the training pool, and other parameters of reinforcement learning;

[0174] Define the settings of the state, action, reward, and Loss functions in the DQN algorithm.

[0175] In actual execution, the training phase requires the following steps:

[0176] Initialize the Q network and the target Q network.

[0177] The loop iteration epoch = 1, 2,..., M; initialize the state S1; loop through the time steps t = 1, 2,..., T; where the time step represents a certain moment.

[0178] When the time step is divisible by 30, use the prior-based greedy algorithm to generate the action a for interacting with the environment t . Generate a random number p in the range [0, 1]. When 0 < p < 0.1, randomly select an action from the action space. When 0.1 < p < y, select the expert action given by the prior knowledge. When y < p < 1, select the action with the maximum value output by the target Q network.

[0179] Execute the selected action, calculate the reward and the state at the next action time step by counting the reward and state between the current moment and the next action time step.

[0180] Put the data pair (S t , at , r t , S t+1 ) is stored in the memory pool.

[0181] Every c time steps, a batch of data is randomly extracted from the memory pool to update the network training.

[0182] The acquired status and issued actions are both interacted through the interface provided by the policy service module in the business configuration module.

[0183] The following describes the congestion relief strategy determination device based on reinforcement learning and digital twins provided by the present invention. The congestion relief strategy determination device based on reinforcement learning and digital twins described below and the congestion relief strategy determination method based on reinforcement learning and digital twins described above can refer to each other.

[0184] Figure 7 This is a schematic diagram of the structure of the device for determining the congestion relief strategy based on reinforcement learning and digital twins provided by the present invention. Figure 7 The present invention provides a congestion relief strategy determination device based on reinforcement learning and digital twins, which may include: a first determination module 710 and a second determination module 720.

[0185] The first determination module 710 is used to input the crowd flow data of each area in the target range into the target intelligent agent model in a virtual simulation environment built based on the digital twin, and obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model;

[0186] A second determination module 720 is configured to obtain an optimal congestion relief strategy based on the target action set, wherein the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area;

[0187] The target agent model is determined by the following steps:

[0188] Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions;

[0189] Based on the first action, the second action and the third action, the intelligent agent model is trained to obtain a trained target intelligent agent model, wherein the first action is any action in the action space, the second action is an action determined based on the demonstration data, and the third action is an action determined based on the intelligent agent model.

[0190] The congestion relief strategy determination device based on reinforcement learning and digital twins provided by the present invention is used to stably and quickly obtain the optimal congestion relief strategy through the design of reinforcement learning algorithm, the design of optimization exploration space and the design of neural network training framework and the training and learning design of the model.

[0191] In some embodiments, the apparatus further comprises:

[0192] A third determination module, configured to determine the first action when the exploration probability is greater than 0 and less than a preset exploration weight;

[0193] When the exploration probability is greater than or equal to the preset exploration weight and less than a dynamic influence factor, determining the second action, where the dynamic influence factor is a parameter selected during each round of training of the agent model;

[0194] When the exploration probability is greater than or equal to the dynamic impact factor and less than or equal to 1, the third action is determined.

[0195] In some embodiments, the dynamic impact factor is determined by the following formula:

[0196]

[0197] Among them, y ep is the dynamic impact factor, ∈ is the preset exploration weight, β is the preset hyperparameter, ep is the training round of the agent model, ep-all is the total training round of the agent model, r ep is the average reward of the ep-th training.

[0198] In some embodiments, the state space is: S t ={S 1,t , S 2,t , S 3,t , ..., S i,t},S t represents the set of people in each area at time t, S i,t It is determined by the following formula:

[0199]

[0200] Among them, S i,t represents the number of people in the ith area at time t, Ni represents the i-th, n i,j,t It represents the number of people in the jth road section in the i-th area at time t.

[0201] In some embodiments, the crowd diversion action includes at least one of the following:

[0202] issuing electronic vouchers to terminals located in each of the areas, the electronic vouchers being used to indicate consumption areas;

[0203] providing a route guidance tool in each of said areas;

[0204] Obstacle guiding means are provided in each of the areas.

[0205] In some embodiments, the reward value function is:

[0206]

[0207] Among them, r t is the reward value function, the r t represents the congestion degree of each area in the time period t to t+w, w represents the action interval time, S n_max represents the number of people in the nth area who reach the crowding threshold, S n,m represents the number of people in the nth area at the mth moment, r m,n Represents the reward of the nth area at the mth moment.

[0208] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a method for determining a congestion relief strategy based on reinforcement learning and digital twins, and the method includes:

[0209] In a virtual simulation environment built based on digital twins, the flow data of people in each area within the target range is input into the target intelligent agent model to obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model;

[0210] Based on the target action set, an optimal congestion relief strategy is obtained, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area;

[0211] The target agent model is determined by the following steps:

[0212] Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions;

[0213] Based on the first action, the second action and the third action, the intelligent agent model is trained to obtain a trained target intelligent agent model, wherein the first action is any action in the action space, the second action is an action determined based on the demonstration data, and the third action is an action determined based on the intelligent agent model.

[0214] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0215] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the above methods, the method comprising:

[0216] In a virtual simulation environment built based on digital twins, the flow data of people in each area within the target range is input into the target intelligent agent model to obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model;

[0217] Based on the target action set, an optimal congestion relief strategy is obtained, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area;

[0218] The target agent model is determined by the following steps:

[0219] Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions;

[0220] Based on the first action, the second action and the third action, the intelligent agent model is trained to obtain a trained target intelligent agent model, wherein the first action is any action in the action space, the second action is an action determined based on the demonstration data, and the third action is an action determined based on the intelligent agent model.

[0221] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the method for determining a congestion relief strategy based on reinforcement learning and digital twins provided by the above methods, the method comprising:

[0222] In a virtual simulation environment built based on digital twins, the flow data of people in each area within the target range is input into the target intelligent agent model to obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model;

[0223] Based on the target action set, an optimal congestion relief strategy is obtained, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area;

[0224] The target agent model is determined by the following steps:

[0225] Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions;

[0226] Based on the first action, the second action and the third action, the intelligent agent model is trained to obtain a trained target intelligent agent model, wherein the first action is any action in the action space, the second action is an action determined based on the demonstration data, and the third action is an action determined based on the intelligent agent model.

[0227] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0228] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining a congestion relief strategy based on reinforcement learning and digital twins, characterized in that: include: In a virtual simulation environment built based on digital twins, the flow data of people in each area within the target range is input into the target intelligent agent model to obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model; Based on the target action set, an optimal congestion relief strategy is obtained, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area; The target agent model is determined by the following steps: Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions; Based on a first action, a second action, and a third action, the agent model is trained to obtain a trained target agent model, wherein the first action is any action in the action space, the second action is an action determined based on demonstration data, and the third action is an action determined based on the agent model; Wherein, before training the agent model based on the first action, the second action and the third action, the method further includes: determining the first action when the exploration probability is greater than 0 and less than a preset exploration weight; determining the second action when the exploration probability is greater than or equal to the preset exploration weight and less than a dynamic influence factor, wherein the dynamic influence factor is a parameter selected during each round of training of the agent model; determining the third action when the exploration probability is greater than or equal to the dynamic influence factor and less than or equal to 1; Wherein, the dynamic impact factor is determined by the following formula: in, is the dynamic impact factor, The preset exploration weights are: is the preset hyperparameter, is the training round of the agent model, is the total number of training rounds for the agent model, For the The average reward of training times.

2. The method for determining a congestion relief strategy based on reinforcement learning and digital twins according to claim 1, characterized in that: The state space is: S t ={S 1,t , S 2,t , S 3,t , ..., S i,t },S t represents the set of people in each area at time t, S i,t It is determined by the following formula: in, represents the number of people in the ith area at time t, N i represents the i-th, It represents the number of people in the jth road section in the i-th area at time t.

3. The method for determining a congestion relief strategy based on reinforcement learning and digital twins according to claim 1, characterized in that: The crowd diversion action includes at least one of the following: issuing electronic vouchers to terminals located in each of the areas, the electronic vouchers being used to indicate consumption areas; providing a route guidance tool in each of said areas; Obstacle guiding means are provided in each of the areas.

4. The method for determining a congestion relief strategy based on reinforcement learning and digital twins according to claim 1, characterized in that: The reward value function is: in, is the reward value function, represents the congestion degree of each area in the time period t to t+w, w represents the action interval time, represents the number of people in the nth area who reach the crowding threshold. represents the number of people in the nth area at the mth moment, Represents the reward of the nth area at the mth moment.

5. A device for determining a congestion relief strategy based on reinforcement learning and digital twins, characterized in that: include: A first determination module is used to input the crowd flow data of each area within the target range into the target intelligent agent model in a virtual simulation environment built based on the digital twin, and obtain a target action set at the current moment, where the target action set is the optimal action sequence at the current moment output by the target intelligent agent model; A second determination module is used to obtain an optimal congestion relief strategy based on the target action set, where the optimal congestion relief strategy is a strategy that minimizes the average congestion degree of each area; The target agent model is determined by the following steps: Based on the state space, action space and reward value function in the simulation cycle, an intelligent agent model is constructed, wherein the state space is a set of the number of people in each area, the action space is a set of crowd diversion actions performed in each area, and the reward value function is used to indicate the congestion degree of each area within the action interval time, and the action interval time is the time difference of performing two adjacent crowd diversion actions; Based on a first action, a second action, and a third action, the agent model is trained to obtain a trained target agent model, wherein the first action is any action in the action space, the second action is an action determined based on demonstration data, and the third action is an action determined based on the agent model; The second determination module is further configured to determine the first action when the exploration probability is greater than 0 and less than a preset exploration weight; determine the second action when the exploration probability is greater than or equal to the preset exploration weight and less than a dynamic influence factor, wherein the dynamic influence factor is a parameter selected during each round of training of the agent model; and determine the third action when the exploration probability is greater than or equal to the dynamic influence factor and less than or equal to 1; Wherein, the dynamic impact factor is determined by the following formula: in, is the dynamic impact factor, The preset exploration weights are: is the preset hyperparameter, is the training round of the agent model, is the total number of training rounds for the agent model, For the The average reward of training times.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for determining a congestion relief strategy based on reinforcement learning and digital twins as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining a congestion relief strategy based on reinforcement learning and digital twins as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for determining a congestion relief strategy based on reinforcement learning and digital twins as described in any one of claims 1 to 4 is implemented.

Images