Target capture method in SUMO urban environment based on deep reinforcement learning

Abstract

The invention discloses a deep reinforcement learning-based SUMO urban environment target capture method. The method comprises the following steps of constructing an urban environment with a high simulation degree; collecting regional information; designing a state space, an action space and a reward function; training a DQN network of each chasing vehicle; and training a region division network according to the collected region information. The SUMO urban environment is closer to a real pursuit scene between a pursuit vehicle and a target vehicle in the urban environment, the environment comprises elements such as lane lines, traffic lights and obstacles, and dynamic traffic characteristics of multiple lanes and multiple intersections can be comprehensively simulated; according to the target capturing method provided by the invention, a complex problem is simplified, the surrounding of the target is divided into a plurality of sub-regions, the chasing vehicles are divided into different cooperative groups, and each group focuses on capturing tasks in the region in charge of the group, so that the cooperative capturing efficiency of the plurality of chasing vehicles on the target vehicle is improved.

Description

Technical Field

[0001] This invention belongs to the field of deep reinforcement learning, and mainly relates to a target capture method in a SUMO urban environment based on deep reinforcement learning. Background Technology

[0002] The rapid pace of global urbanization has brought numerous conveniences, but it has also given rise to new security risks. Factors such as dense populations, heavy traffic, and complex infrastructure have exacerbated urban safety issues. However, traditional response models often rely on fixed rules and human intervention, making it difficult to react quickly to constantly changing environments. In complex urban environments, responding to emergencies requires more intelligent solutions. Intelligent systems based on Deep Reinforcement Learning (DRL) can optimize response strategies through real-time environmental perception and analysis, thereby enhancing the ability to cope with complex scenarios.

[0003] Traditional target acquisition algorithms are generally divided into deterministic methods and heuristic methods. Deterministic methods are based on rigorous mathematical models and rules, relying on precise data and prior information, and typically provide reliable results in simple scenarios, with stable targets, or minimal interference. Examples include pursuit curve analysis and game theory. In contrast, heuristic methods rely on experience and intuition, offering greater flexibility and suitability for complex scenarios. For instance, they can be used to explore the collective behavior of intelligent agent swarms based on local observations through simulation. However, these methods may lack stability and reliability in certain situations. It should be noted that both deterministic and heuristic methods may face challenges such as inaccurate target localization, difficulty in motion prediction, or complex deformation modeling when dealing with complex scenarios involving target occlusion, rapid movement, or non-rigid deformation.

[0004] Traditional target acquisition algorithms often perform poorly in complex urban environments. Deep reinforcement learning, however, combines the advantages of deep learning in high-dimensional data processing with the intelligent decision-making capabilities of reinforcement learning, offering a novel optimization solution for the target acquisition problem. Leveraging the powerful representational capabilities of deep neural networks, deep reinforcement learning can efficiently fit key components such as state-value functions, action-value functions, policies, and models, and optimize the acquisition strategy through end-to-end learning. In robot navigation and cooperative control, deep reinforcement learning has been successfully applied to implement pursuit and escape strategies among multiple agents, improving the cooperative performance of the system in complex environments. Furthermore, in the gaming domain, deep reinforcement learning algorithms have achieved remarkable game intelligence performance in simulating attack and defense scenarios in virtual environments, demonstrating its powerful real-time decision-making capabilities.

[0005] However, existing technologies often have the following drawbacks in solving the target capture problem in urban environments: 1) Urban environments are often not complex enough, lacking elements such as lane lines, traffic lights, and obstacles; 2) Adding elements such as lane lines, traffic lights, and obstacles to urban environments will restrict the movement path of the pursuing vehicle, while the dense interactive dynamics will significantly increase the complexity of strategy decision-making. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention uses a SUMO simulation environment that incorporates elements such as lane lines, traffic lights, and obstacles to simulate a more realistic urban environment. Furthermore, to improve the collaborative capture efficiency of multiple pursuing vehicles in a target vehicle, a target capture method based on deep reinforcement learning in a SUMO urban environment is provided. This method uses an automatic grouping mechanism to divide pursuing vehicles into several groups, each responsible for tasks in different areas, and dynamically adjusts the allocation of group members based on real-time environmental changes.

[0007] The technical solution to achieve the purpose of this invention is: a target capture method based on deep reinforcement learning in a SUMO urban environment, comprising the following steps:

[0008] S1: Construct an urban environment that includes lane lines, traffic lights, and obstacles;

[0009] S2: Collect area information;

[0010] S3: Design the state space, action space, and reward function;

[0011] S4: Train the DQN network for each pursuit vehicle;

[0012] S5: Train the region segmentation network based on the collected regional information.

[0013] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0014] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0015] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] Compared with the prior art, the significant advantages of this invention are:

[0017] 1) The SUMO urban environment used in this invention is closer to the real chase scenario between the pursuing vehicle and the target vehicle in an urban environment. This environment includes elements such as lane lines, traffic lights and obstacles, which can fully simulate the dynamic traffic characteristics of multiple lanes and multiple intersections, providing a reliable simulation basis for subsequent algorithm verification.

[0018] 2) The target capture method proposed in this invention simplifies the complex problem. This method divides the area around the target into several sub-regions and assigns the pursuit vehicles to different cooperative groups. Each group focuses on the capture task in its assigned area, thereby improving the efficiency of the coordinated capture of the target vehicle by multiple pursuit vehicles. Attached Figure Description

[0019] Figure 1 This invention relates to the SUMO urban environment.

[0020] Figure 2 This is the DQN training process of the present invention.

[0021] Figure 3 The diagram shows the training framework and network structure of the collaborative capture DQN algorithm based on region information of this invention.

[0022] Figure 4 This is the region partitioning network of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] To make the technical solution of the present invention clearer, a target capture method based on deep reinforcement learning in a SUMO urban environment according to the present invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and do not limit the present invention. The present invention is specifically implemented according to the following steps:

[0025] S1: Construct a highly realistic urban environment:

[0026] To better simulate real-world urban traffic scenarios, this invention uses SUMO to construct a highly realistic urban environment. This environment includes elements such as lane lines, traffic lights, and obstacles. Figure 1 As shown, a multi-lane, multi-intersection urban network was generated based on SUMO, aiming to provide a realistic simulation background for the subsequent capture task. In the simulation, red vehicles represent the pursuer, initially positioned at the top right vertex; blue vehicles represent the escaping target, initially positioned at the bottom left vertex; and yellow vehicles represent obstacle vehicles. The pursuer's task is to approach and capture the escaping target. The pursuer is trained through multi-agent reinforcement learning to make decisions at intersections; while the target's movement path is randomly generated.

[0027] S2: Data collection area information:

[0028] This invention first divides the task area based on the target's current location, with the number of areas set to the number of intersections around the target; then, it collects data from area R. i The probability B of the target appearing i Potential threat value (number of vehicles obstructing the area) W i The number of agents currently assigned, C i And the pursuit vehicle P j Move to area R i The cost d ij Assume region R i Task requirements using T i It can be calculated using the following formula:

[0029] T i =α1B i +α2W i -α3C i

[0030] Among them, B i Indicates that the target is in region R i The probability of W appearing in i C represents the potential threat value of the area. i This represents the number of agents currently assigned to this region, where α1, α2, and α3 are the task parameter weights.

[0031] Based on the task requirements assessment results T i The allocation of target regions for pursuit vehicles can be achieved through a region partitioning network, and the allocation of target regions for pursuit vehicles can be achieved by minimizing the task cost:

[0032]

[0033] Where, x ij ∈{0,1} represents the pursuer P j Is it assigned to region R? i d ij P, the pursuer j Move to area R i The cost (path length or time) is denoted by n, where n represents the task space being divided into n regions, and m represents the total number of pursuit vehicles.

[0034] S3: Design the state space, action space, and reward function, specifically including the following steps:

[0035] S31: The state space is the primary source of an agent's perception of its environment. To ensure that the agent can accurately understand its traffic environment, the state is divided into two parts: the global state and the local state.

[0036] The global state describes the information of the entire city simulation environment, represented as:

[0037] S={pursuer_pos,evader_pos,background_veh,step}

[0038] Where, pursuer_pos contains lane and position information for all pursuing vehicles, evader_pos contains lane and position information for the target vehicle, background_veh contains information about obstacle vehicles, and step is the current simulation time step.

[0039] The local state describes the environmental information of a single pursuit vehicle from a local perspective, and is represented as:

[0040] s i ={ego_pos,target_pos,region_pos,traffic_state,step}

[0041] Here, ego_pos represents the location information of the currently pursued vehicle, target_pos represents the location information of the currently targeted vehicle, region_pos represents the relevant information of all regions, and traffic_state represents the relevant information of traffic flow within a local area.

[0042] S32: The action space defines all possible actions that the agent can take. In an urban traffic simulation environment, the action space is typically related to the agent's movement behavior on the road. This invention defines actions as the following basic behaviors: go straight, turn left, and turn right; the agent only makes choices at intersections.

[0043] In specific scenarios, the action space can be further expanded, for example, by adding actions such as "changing lanes" or "decelerating". However, to simplify the problem and reduce training difficulty, this invention initially sets three basic actions. Each action represents a decision that the agent can make in a traffic environment and corresponds to the actual driving rules of lanes in SUMO.

[0044] S33: The reward function plays a central role in reinforcement learning tasks, determining the agent's behavioral direction and driving its learning to achieve the expected goals. The reward function design of this invention combines the interaction process between the pursuing vehicle and the target vehicle, the capture behavior, the region segmentation behavior, and the time factor, aiming to incentivize the pursuer to efficiently complete the capture task. The specific reward function formula is as follows:

[0045]

[0046] in, Indicates the pursuit vehicle P i The reward value; This time-based penalty serves to encourage pursuers to complete their mission as quickly as possible; c This refers to the capture reward; when the pursuer successfully captures the escapee, a significant positive reward is given to incentivize the capture behavior. This indicates the distance bonus between the pursuit vehicle and the target area; a bonus is awarded when the distance between the pursuer and the target area decreases.

[0047] S4: Train the DQN network for each pursuit vehicle, which includes the following steps:

[0048] S41: The status, actions, rewards, and next status data generated by the interaction between the target vehicle and the pursuing vehicle in the urban environment are stored in the experience pool module, thereby reducing the correlation between samples and improving the sample utilization rate through the experience replay mechanism.

[0049] S42: Training the DQN network. Each pursuit vehicle independently possesses a DQN network for learning and optimizing the capture strategy. Through a distributed, independent learning mechanism, the pursuit vehicle can dynamically select the optimal action based on the current state input to complete the cooperative capture task of the target vehicle.

[0050] DQN introduces two important components to traditional Q-learning: a neural network and an experience pool. The main role of the neural network is to receive the current state, calculate the value of the corresponding action, and then select the action with the highest value as the output. However, it's important to note that parameter updates depend on the error between the current Q-value and the predicted Q-value. This means that current parameter updates will affect subsequent training data; for example, if the action corresponding to the maximum Q-value is a leftward movement, then the training samples will be influenced by samples to the left. This dependency can lead to loops during training, causing the neural network to get stuck in local minima or even fail to converge. To address this issue, DQN employs a dual-network architecture: an online network and a target network. The online network continuously updates the parameters and calculates the Q-estimate, while the target network keeps the parameters frozen and updates them only once at regular intervals to calculate the target Q-value. Simultaneously, the experience pool is used to store historical samples, from which samples are randomly drawn during subsequent learning, thereby breaking down correlations between data and improving training efficiency.

[0051] Furthermore, DQN employs a time difference (TD) based loss function, combined with gradient descent to optimize neural network parameters, enabling the network to gradually approximate the true Q-value function. Its loss function is:

[0052] L(θ)=[yQ(s t ,a;θ)]2

[0053] Where y = max a Q(s t+1 ,a;θ - ) is the target value of Q, s t s represents the state at the current time step t. t+1 The state at the next time step is represented by 'a', the action chosen by the pursuing vehicle is represented by 'θ', and 'θ' represents the state at the next time step. - This represents the parameters of the online network and the target network in DQN; when optimizing the loss function L(θ), the parameters θ from the previous iteration are temporarily fixed. - .

[0054] The training process of DQN is as follows: Figure 2 As shown, the target network Q(s) t ,a;θ - ) and online network Q(s t The structures (a, θ) are the same, but the parameters of the target network are updated only after every N steps, so that θ - =θ. This design ensures that the target Q value remains constant over a period of time, reducing the correlation between the current Q value and the target Q value to some extent and improving the stability of the algorithm.

[0055] The structure of the DQN network is as follows: Figure 3 As shown at the bottom. The input layer receives the environmental state vector perceived by the pursuing vehicle, containing key information such as the pursuing vehicle's own position, the relative position of the target vehicle, and the region location. The first hidden layer consists of three parallel fully connected layers, containing 16, 16, and 6 hidden units respectively, and uses the ReLU activation function to achieve a non-linear mapping, representing the position information of the pursuing and target vehicles, traffic information, and region information, respectively. This parallel design can effectively separate specific features in the input state, thereby improving the efficiency and accuracy of feature extraction. After feature separation, these feature vectors are concatenated and then further fused through several fully connected layers. The output layer consists of a fully connected layer, outputting the Q-values ​​of three actions, corresponding to the action value estimates of turning left, going straight, and turning right, respectively. To further ensure the accuracy of action selection, the output is normalized by Softmax, and the action with the largest Q-value is selected as the optimal decision in the current state.

[0056] S5: Train a region segmentation network based on the collected region information:

[0057] S51: Set a grouping scheme based on the rule that the number of regions is equal to the number of intersections surrounding the target. Assume the task space is divided into n regions R1, R2, ..., R... n There are a total of m pursuit vehicles P1, P2, ..., P mA static grouping scheme can be represented as G = [G1, G2, ..., G...]. n ],in Indicates allocation to region R i The pursuit vehicle must meet the following constraints:

[0058]

[0059] S52: Input the region information into the region partitioning network for training, such as... Figure 4 As shown. The input to the region partitioning network is the observation state information of the pursued vehicle i. Extract this state information to obtain the target's location in region R. i The probability of B appearing in i Potential threat value W within the region i The number of agents C currently assigned to this region i And indicate that the pursuer P j Move to area R i The cost d ij These features comprehensively reflect the dynamic environmental information of the target area and the potential complexity of task execution.

[0060] The network structure consists of two parallel branches, each processing different types of feature information. The first branch will process B... i W i And the number of pursuit vehicles C i As input, the data passes through a fully connected layer with 16 hidden units, and is then processed by a ReLU activation function to perform a non-linear mapping of features, extracting the latent relationships between them. The second branch will then... ij The input is also processed through a fully connected layer with 16 hidden units for feature mapping. The feature vectors from the two branches are fused in the second layer to form a comprehensive feature representation, which is then processed through a fully connected layer with three outputs for decision processing. The output layer uses the Softmax function to normalize the decision result into a probability distribution, and finally uses the Argmax operation to select the optimal target region.

[0061] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention.

Claims

1. A target acquisition method based on deep reinforcement learning in a SUMO urban environment, characterized in that, Includes the following steps: S1: Construct an urban environment that includes lane lines, traffic lights, and obstacles; S2: Collect area information; S3: Design the state space, action space, and reward function; S4: Train the DQN network for each pursuit vehicle; S5: Train the region segmentation network based on the collected regional information.

2. The method according to claim 1, characterized in that, In step S1, an urban environment containing lane lines, traffic lights, and obstacles is constructed. A multi-lane, multi-intersection urban network is generated based on SUMO. The task of the pursuer is to approach and capture the escape target. The pursuer is trained through multi-agent reinforcement learning to make decisions at intersections. The target's movement route is generated randomly.

3. The method according to claim 1, characterized in that, In step S2, the area information is collected, specifically as follows: First, the task area is divided based on the target's current location, with the number of areas set to the number of intersections around the target; then, the data collection area R is performed. i The probability B of the target appearing i Potential threat value W within the region i The number of agents currently assigned, C i And the pursuit vehicle P j Move to area R i The cost d ij Assume region R i Task requirements using T i It can be calculated using the following formula: T i =α1B i +α2W i -α3C i Among them, B i Indicates that the target is in region R i The probability of W appearing in i C represents the potential threat value of the area. i This represents the number of agents currently assigned to this region, where α1, α2, and α3 are the task parameter weights. Based on the task requirements assessment results T i The allocation of target regions for pursuit vehicles is achieved through a region partitioning network, and the allocation of target regions for pursuit vehicles is realized by minimizing the task cost: Where, x ij ∈{0,1} represents the pursuing vehicle P j Is it assigned to region R? i d ij Indicates the pursuit vehicle P j Move to area R i The cost is denoted by n, where n represents the task space being divided into n regions, and m represents the total number of pursuit vehicles.

4. The method according to claim 1, characterized in that, Step S3 involves designing the state space, action space, and reward function, specifically including the following steps: S31: The state space is the main source of an agent's perception of the environment; the state is divided into two parts: global state and local state. The global state describes the information of the entire city simulation environment, represented as: S={pursuer_pos,evader_pos,background_veh,step} in, pursuer_pos contains lane and position information for all pursuing vehicles, evader_pos contains lane and position information for the target vehicle, background_veh contains information about obstacle vehicles, and step is the current simulation time step. The local state describes the environmental information of a single pursuit vehicle from a local perspective, and is represented as: s i ={ego_pos,target_pos,region_pos,traffic_state,step} Among them, ego_pos represents the location information of the currently pursued vehicle, target_pos represents the location information of the currently targeted vehicle, region_pos represents the relevant information of all regions, and traffic_state represents the relevant information of traffic flow within a local area; S32: The action space defines all possible actions that the agent can take; in the urban traffic simulation environment, the action space is related to the agent's movement behavior on the road; the actions are defined as the following basic behaviors: go straight, turn left, turn right, and the agent only makes choices at intersections; S33: The reward function is designed by combining the interaction between the pursuing vehicle and the target vehicle, the capture behavior, the area division behavior, and the time factor, aiming to incentivize the pursuer to complete the capture task efficiently; the specific reward function formula is as follows: r pi ＝r si +r c +r di Where, r pi Indicates the pursuit vehicle P i The reward value; r si This time-based penalty serves to encourage pursuers to complete their mission as quickly as possible; c This refers to the capture reward; when the pursuer successfully captures the escapee, a significant positive reward is given to incentivize the capture behavior. di This indicates the distance bonus between the pursuit vehicle and the target area; a bonus is awarded when the distance between the pursuer and the target area decreases.

5. The method according to claim 1, characterized in that, In step S4, the DQN network for each pursuit vehicle is trained, which specifically includes the following steps: S41: Store the status, actions, rewards, and next status data generated by the interaction between the target vehicle and the pursuing vehicle in the urban environment in the experience pool module; S42: Training the DQN network; Each pursuit vehicle has its own independent DQN network for learning and optimizing the capture strategy; Through a distributed independent learning mechanism, the pursuit vehicle can dynamically select the optimal action based on the current state input to complete the cooperative capture task of the target vehicle.

6. The method according to claim 1, characterized in that, DQN introduces two components on top of Q-learning: a neural network and an experience pool. The neural network receives the current state, calculates the value of the corresponding action, and then selects the action with the highest value as the output. Parameter updates depend on the error between the current Q-value and the predicted Q-value. DQN adopts a dual-network architecture: an online network and a target network. The online network continuously updates the parameters and calculates the Q-estimate, while the target network keeps the parameters frozen and updates them only once at certain time intervals to calculate the target Q-value. The experience pool is used to store historical samples, from which samples are randomly drawn during subsequent learning. Furthermore, DQN employs a time-difference-based loss function, combined with gradient descent to optimize neural network parameters, enabling the network to gradually approximate the true Q-value function. Its loss function is: L(θ)=[yQ(s t ,a;θ)] 2 Where y = max a Q(s t+1 ,a;θ - ) is the target value of Q, s t s represents the state at the current time step t. t+1 The state at the next time step is represented by 'a', the action chosen by the pursuing vehicle is represented by 'θ', and 'θ' represents the state at the next time step. - This represents the parameters of the online network and the target network in DQN; when optimizing the loss function L(θ), the parameters θ from the previous iteration are temporarily fixed. - ; Target network Q(s) t ,a;θ - ) and online network Q(s t ,a; The structure is the same as θ), but the parameters of the target network are updated only after every N steps, so that θ - =θ; The input layer of the DQN network receives the environmental state vector perceived by the pursuing vehicle, which includes the position of the pursuing vehicle itself, the relative position of the target vehicle, and the region position. The first hidden layer consists of three parallel fully connected layers, which contain 16, 16, and 6 hidden units respectively, and are combined with the ReLU activation function to achieve nonlinear mapping, which respectively represent the position information of the pursuing and target vehicles, traffic information, and region information. After feature separation, these feature vectors are concatenated and then further fused through several fully connected layers. The output layer consists of a fully connected layer that outputs the Q-values ​​of three actions, corresponding to the action value estimates of turning left, going straight, and turning right, respectively. After the output is normalized by Softmax, the action with the largest Q-value is selected as the optimal decision in the current state.

7. The method according to claim 1, characterized in that, In step S5, a region segmentation network is trained based on the collected region information, specifically as follows: S51: Set a grouping scheme based on the rule that the number of regions is equal to the number of intersections around the target; assume the task space is divided into n regions R1, R2, ..., R n There are a total of m pursuit vehicles P1, P2, ..., P m The static grouping scheme can be represented as G = [G1, G2, ..., G...]. n ],in Indicates allocation to region R i The pursuit vehicle must meet the following constraints: S52: Input the region information into the region partitioning network for training. The input of the region partitioning network is the observation state information of the pursuing vehicle i. Extract this state information to obtain the target's location in region R. i The probability of B appearing in i Potential threat value W within the region i The number of agents C currently assigned to this region i And indicate that the pursuer P j Move to area R i The cost d ij ; The network structure consists of two parallel branches, which are used to process different types of feature information respectively; The first Branch B i W i And the number of pursuit vehicles C i As input, the data passes through a fully connected layer with 16 hidden units, and is then processed by a ReLU activation function to perform a non-linear mapping of features, extracting the latent relationships between features; the second branch then... ij The input is also processed through a fully connected layer with 16 hidden units for feature mapping; the feature vectors output from the two branches are fused in the second layer to form a comprehensive feature representation, which is then processed through a fully connected layer with 3 outputs for decision processing; the output layer uses the Softmax function to normalize the decision result into a probability distribution, and finally selects the optimal target region through the Argmax operation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-7.

Images