Road section non-motor vehicle behavior simulation method based on data driving

Abstract

The invention relates to a road section non-motor vehicle behavior simulation method based on data driving, non-motor vehicle behavior simulation is realized by using an intelligent agent model, and the acquisition process of the intelligent agent model comprises the following steps: initializing and controlling a non-motor vehicle to move in a simulation environment and constructing the intelligent agent model of a simulation track; taking the difference between the real trajectory and a simulation trajectory generated by the agent model as a basic reward of the agent; and based on the basic reward, under the efficiency constraint, the comfort constraint and the security constraint, iteratively updating the agent model through data driving. Compared with the prior art, the data driving method provided by the invention can capture motion process differences and interaction modes of heterogeneous non-motor vehicles, the provided reward function can accurately capture behavior action characteristics of different non-motor vehicles, and the used Lagrange method can accurately limit unreasonable behaviors of the intelligent agent.

Description

Technical Field

[0001] The invention relates to the technical field of traffic flow simulation, and in particular to a data-driven method for simulating the behavior of non-motor vehicles on a road section. Background Art

[0002] Autonomous driving technology has now entered the stage of large-scale testing and demonstration applications. However, safety remains a major challenge facing high-level autonomous driving vehicles before large-scale commercial applications. In existing technologies, microscopic traffic flow simulation technology is considered to be an economical, efficient and controllable technical means because it can provide a reliable background traffic flow environment for autonomous driving system safety testing and the construction of related platforms. However, in order to meet the authenticity requirements of virtual testing, how to reproduce various test scenarios based on traffic flow simulation methods with high fidelity and ensure the accuracy of individual behavior simulation during the scenario construction process has become an important prerequisite for whether traffic flow simulation technology can effectively support virtual testing and one of the key issues of continuous concern.

[0003] Non-motor vehicles are an important part of urban transportation, and to achieve a high degree of autonomous driving, mixed traffic environments are one of the scenarios that autonomous driving algorithms must deal with. In reality, due to the heterogeneity of traffic individuals and unstructured trajectories, non-motor vehicles have a very high degree of freedom of movement. Their behaviors such as overtaking, crossing the line and occupying the motor vehicle lane have brought severe challenges to the planning and decision-making of autonomous driving vehicles. It is urgent to accurately simulate these behaviors through traffic flow simulation technology so that they can be accurately reproduced in the test environment.

[0004] The existing data-driven simulation methods still have some applicability issues for simulating non-motor vehicle behavior on road sections, including:

[0005] (1) Existing methods homogenize the interactive behaviors of all cyclists, while non-motor vehicle interference behaviors such as crossing the line, overtaking and cutting in line have significant heterogeneous characteristics of individual cyclist behaviors;

[0006] (2) The agent behavior model in existing methods lacks reasonable constraints, resulting in large differences between the simulated behavior and the actual behavior, thus distorting the behavior;

[0007] (3) There is a lack of simulation models that can realistically reproduce the interference behaviors of non-motor vehicles on the road, such as overtaking, crossing the line and occupying the motor vehicle lane.

[0008] Therefore, the existing data-driven simulation methods for non-motor vehicle interaction behaviors still cannot meet the authenticity requirements of virtual testing of autonomous vehicles, and it is still necessary to propose a data-driven simulation method that can accurately simulate road interference behaviors. Summary of the invention

[0009] The purpose of the present invention is to provide a data-driven road section non-motor vehicle behavior simulation method in order to overcome the defects of the above-mentioned prior art, so as to solve or partially solve the problems of not fully considering the heterogeneity of individual non-motor vehicle behaviors, large differences between simulated behaviors and actual behaviors, and the addition of constraints affecting reinforcement learning.

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] The present invention provides a data-driven road section non-motor vehicle behavior simulation method, which uses an intelligent agent model to realize non-motor vehicle behavior simulation. The acquisition process of the intelligent agent model includes the following steps:

[0012] Initialize an intelligent agent model that controls the movement of non-motor vehicles in a simulation environment and constructs a simulation trajectory;

[0013] The difference between the real trajectory and the simulated trajectory generated by the agent model is used as the basic reward of the agent;

[0014] Based on the basic reward, the agent model is updated through data-driven iteration under efficiency constraints, comfort constraints and safety constraints.

[0015] As an optimal technical solution, during the construction of the intelligent agent model, the state space of the intelligent agent includes the expected speed of the vehicle, the distance between the vehicle and the curb, the speed of the vehicle, the relative position of the non-motor vehicle closest to the vehicle in multiple directions, the relative speed of the non-motor vehicle closest to the vehicle in multiple directions, the relative position of the motor vehicle closest to the front and rear directions on the motor vehicle lane, and the relative speed of the motor vehicle closest to the front and rear directions on the motor vehicle lane. The action space of the intelligent agent includes lateral acceleration and longitudinal acceleration.

[0016] As a preferred technical solution, in the agent model, the calculation process of the agent's basic reward includes the following steps:

[0017] Get the state-action pairs of the agent and use the discriminator to calculate the similarity between the real trajectory and the simulated trajectory generated by the agent model;

[0018] Calculate the coordinate distance difference between the real trajectory and the simulated trajectory generated by the agent model;

[0019] The similarity between the simulation trajectories and the coordinate distance difference are combined as the basic reward for the agent.

[0020] As a preferred technical solution, during the training process of the discriminator, the loss function is:

[0021] L=E π [log(D(s,a))]+E πE[log(1-D(s,a))]

[0022] Where L is the loss function, E[·] represents the expected value, π represents the data from the simulation trajectory, πE represents the data from the real trajectory, D represents the discriminator, and (s, a) represents the state-action pair.

[0023] As a preferred technical solution, the coordinate distance difference is calculated using the following formula:

[0024]

[0025] in, is the coordinate distance difference between the real trajectory and the simulated trajectory generated by the agent model, represents the coordinates of the real trajectory, Represents the coordinates of the simulation trajectory.

[0026] As a preferred technical solution, based on the basic reward, under efficiency constraints, comfort constraints and safety constraints, the process of iteratively updating the agent model through data-driven includes the following steps:

[0027] Adding constraint items corresponding to the efficiency constraint and the comfort constraint to the basic reward of the agent to obtain an enhanced reward of the agent;

[0028] Establishing a Lagrange multiplier that controls the weight of the safety constraint;

[0029] Based on the enhanced agent's reward and the Lagrange multiplier, the data-driven agent model is iteratively updated through proximal strategy optimization.

[0030] As a preferred technical solution, the update target of the Lagrange multiplier is:

[0031]

[0032] Among them, g(θ) represents the constraint condition, R s represents a safety constraint, represents the sum of the safety penalties during the total simulation time T, R limit It represents the limit that the safety constraint is expected not to exceed, L(θ) represents the optimization target, f(θ) represents the optimization target of the original reward, λ is the Lagrange multiplier, and the model parameter θ is updated by gradient ascent, and λ is updated by gradient descent.

[0033] As a preferred technical solution, the efficiency constraint, comfort constraint and safety constraint are:

[0034]

[0035] Among them, R e, R c are the constraints corresponding to efficiency constraint and comfort constraint, R s represents the safety constraint, v t is the current speed, v mean is the average speed of the vehicle, STD(v) is the standard deviation of the speeds of all vehicles, and a t is the acceleration at the current moment, a mean is the average acceleration of all cars, STD(a) is the standard deviation of acceleration of all cars, r ell is the distance from the point on the ellipse to the center, and r is the distance between other vehicles and the cyclist.

[0036] As a preferred technical solution, the optimization goal of the agent model is:

[0037]

[0038] Among them, J(θ) is the optimization objective function of the intelligent model, (s t ,a t ) is a state-action pair, π θ (a t |s t ) is the current strategy, π θold (a t |s t ) is the old strategy, A θold is the advantage value of the old strategy, ε is the clipping coefficient, and clip() represents the clipping function.

[0039] As a preferred technical solution, the process of implementing data-driven iterative updating of the intelligent agent model through proximal strategy optimization includes the following steps:

[0040] Initialize the network parameters θ of the Actor network, Critic network, Cost-Critic network, and discriminator respectively 0 ,ω 0 , μ 0 , Initialize the Lagrange multiplier λ 0 ;

[0041] Initialize the experience replay pool M;

[0042] Use the Actor network to control the target car in the simulation environment, run the simulation, and run the trajectory (s) per step. t ,a t ,r t ,c t ,s t+1 ) is stored in the experience replay pool M;

[0043] Randomly sample N groups of data (st ,a t ,r t ,c t ,s t+1 );

[0044] Update the discriminator parameters arrive

[0045] Update the Critic network parameters ω i to i+1 , calculate the TD-error of Cost and update the Cost-Critic parameter μ i to μ i+1 ;

[0046] Update the Lagrange multiplier λ i to λ i+1 ;

[0047] Update Actor network parameters θ i to θ i+1 ;

[0048] Clear the experience replay pool M.

[0049] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0050] (1) Fully consider the heterogeneity of individual behaviors of non-motor vehicles: The present invention iteratively updates the intelligent agent model through data-driven, which can capture various complex interference interaction features from real data, and can realistically reproduce interference behaviors such as non-motor vehicles overtaking on the road section, crossing the line and occupying the motor vehicle lane, etc., which meets the high authenticity requirements of the background traffic environment for autonomous driving virtual testing, and thus has the possibility of accurately reproducing the entire interactive process of various interference behaviors, providing a basis for improving the accuracy of traffic flow simulation and supporting the authenticity of autonomous driving virtual test scenarios.

[0051] (2) The difference between simulated behavior and actual behavior is small: The present invention calculates the difference between the real trajectory and the simulated trajectory generated by the intelligent agent model as the reward of the intelligent agent, which can capture the behavioral interactions and heterogeneous characteristics of the movements of different non-motor vehicle individuals, thereby accurately reproducing the individual differences in behaviors such as overtaking, crossing the line and occupying the motor vehicle lane, and solving the problem of individual behavior homogenization in existing micro-simulation models.

[0052] (3) The addition of constraints has little impact on reinforcement learning: The present invention establishes a Lagrange multiplier that controls the proportion of safety constraints. The Lagrange method-based safety constraint addition method can accurately limit the unreasonable behavior of the intelligent agent, making the individual behavior process simulated by the intelligent agent more reasonable, thereby solving the problems of unreasonable constraints and excessive constraints interfering with behavior learning in traditional constraint addition methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of a data-driven road section non-motor vehicle behavior simulation method in an embodiment;

[0054] Figure 2 Generate a trajectory result diagram for simulation in the embodiment;

[0055] Figure 3 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] Example 1

[0058] In response to the problems existing in the aforementioned prior art, the present embodiment provides a data-driven method for simulating the behavior of non-motor vehicles on road sections, which can realistically reproduce the interference behaviors of non-motor vehicles on urban roads, such as overtaking, cutting in, crossing the line and encroaching on motor vehicle lanes, in a traffic flow simulation and autonomous driving virtual test environment, thereby providing accurate simulation trajectories for virtual testing.

[0059] To this end, this embodiment is based on a generative adversarial imitation learning framework to train an intelligent agent model and learn non-motor vehicle behavior data segments contained in real data, thereby overcoming the homogenization of behavior simulation and uncontrollable behavior in existing methods.

[0060] Step 1: Initialize the intelligent agent model, which controls the non-motor vehicle to run in the simulation environment and obtain the simulation trajectory.

[0061] The non-motor vehicle behavior model is trained by single-agent training, that is, the agent only controls one target simulated individual in the simulation environment, and other traffic participants operate according to real trajectory records to construct a real surrounding traffic environment.

[0062] First, a real track segment is selected from the actual data set and used as training data. Then, the simulation environment generates the target simulation car and the environment car according to the same initial state and runs the simulation. In each track segment, the target simulation car runs the same length as the real segment to obtain the simulation trajectory. After each segment is completed, the next track segment is selected until the entire data set is traversed.

[0063] Taking the road separated by markings as an example, the agent control model is modeled using a multi-layer perceptron, and the agent receives the current state and outputs actions to control the operation of the simulated vehicle. Here, the state space S of the designed agent model includes the expected speed v of the vehicle de , distance from the curb y, speed v x ,v y , the relative position Δx of the nearest non-motor vehicle in the four directions of left front, left rear, right front, and right rear b ,Δy b , relative speed Δv xb ,Δv yb , the relative position Δx of the closest vehicle in the front and rear directions on the motorway v ,Δy v , relative speed Δv xv ,Δv yv The agent outputs the horizontal and vertical acceleration according to the current observed state, and the action space A is a x ,a y .

[0064] Step 2: Compare the similarity between the real trajectory and the simulated trajectory, and calculate the reward based on the difference between the two, so as to provide guidance for the state update of the intelligent agent.

[0065] The design of the reward needs to accurately evaluate the difference between the simulated trajectory and the real trajectory. The rewards used in this embodiment include basic rewards and behavioral constraints.

[0066] Among them, the basic rewards mainly include behavior similarity and trajectory distance error.

[0067] Behavior similarity mainly evaluates the similarity between the state-action pairs generated by the simulation trajectory and the state-action pairs of the real trajectory. The core idea is to gradually make the simulation trajectory approach the real trajectory through the adversarial update between the agent and the discriminator. Set a discriminator to score the real trajectory and the simulation trajectory. Its input is a set of state-action pairs and outputs a score of 0-1. The higher the score, the higher the realism. Use the real trajectory and the generated simulation trajectory to train the discriminator so that it gives high scores to the real trajectory and low scores to the simulation trajectory. Update the parameters of the discriminator through gradient descent. The loss function of the discriminator can be described as:

[0068] L=Eπ [log(D(s,a))]+E πE [log(1-D(s,a))]

[0069] Where E[·] represents the expected value, π represents the data from the simulation trajectory, πE represents the data from the real trajectory, D represents the set discriminator, and (s, a) represents the state-action pair.

[0070] After the discriminator is updated, it can score the simulated trajectory, and the score obtained is the similarity R between the simulated trajectory and the real trajectory. G =D(s,a).

[0071] This embodiment uses the trajectory position difference to provide accurate action feedback for different trajectories. The distance difference between the trajectory point and the actual trajectory at each moment is calculated as another position reward R p , the calculation formula is as follows:

[0072]

[0073] In the formula, represents the coordinates of the real trajectory, Represents the coordinates of the simulation trajectory.

[0074] Behavior constraints are used to limit unreasonable behaviors. This embodiment calculates efficiency, comfort, and safety constraints to guide the behavior of the agent. The specific calculation formula is as follows:

[0075]

[0076] In the formula, R e is the efficiency constraint, R c is the comfort constraint, R s is a safety constraint, w e ,w c is the adjustment coefficient, v t is the current speed, v mean is the average speed of the car, STD(v) is the standard deviation of the speeds of all cars, and a t is the acceleration at the current moment, a mean is the average acceleration of all cars, STD(a) is the standard deviation of acceleration of all cars, r ell is the distance from the point on the ellipse to the center, and r is the distance between other vehicles and the cyclist.

[0077] Step 3: Based on the calculated rewards, the reinforcement learning algorithm is used to update the agent model.

[0078] In the calculation method of rewards, efficiency and comfort constraints are added as reward enhancement. The specific calculation method is:

[0079] R=Rbasic -w e ×R e -w c ×R c

[0080] Safety constraints are added in a Lagrangian manner, and the Lagrangian multiplier is used to control the weight of the safety constraints. The Lagrangian multiplier is dynamically adjusted according to the safety constraints of each run. The optimization objective is as follows:

[0081]

[0082] In the formula, g(θ) represents the constraint condition, represents the sum of the safety penalties in the simulation, R limit represents the limit that the safety constraint is not expected to exceed, L(θ) represents the optimization target, f(θ) represents the optimization target of the original reward, and λ is the Lagrange multiplier. The model parameters θ are updated by gradient ascent, and λ is updated by gradient descent.

[0083] The learning algorithm uses proximal policy optimization to update the agent policy model. Its optimization goal can be expressed as:

[0084]

[0085] In the formula, π θ (a t |s t ) refers to the current strategy, π θold (a t |s t ) refers to the old strategy, A θold refers to the advantage value of the old strategy, and ε refers to the clipping factor (set to 0.2 as recommended).

[0086] Table 1 Agent model update steps

[0087]

[0088] The effectiveness of this method will be illustrated by a practical example below.

[0089] The data in this embodiment selected a section of non-motorized lane-separated road on Caoyang Road in Shanghai as the data collection location. The total length of this section of road is about 110m, including two 3.5m motor vehicle lanes and a 2.8m non-motor vehicle lane, and adjacent lanes are separated by markings. Investigators installed cameras high up in buildings near the data collection location, and photographed the traffic conditions of the section from a high-altitude perspective during peak hours (16:00-17:00). The trajectory data of all motor vehicles and non-motor vehicles on the road section were extracted, with a time interval of 0.12s and a trajectory accuracy of 0.1m. The research object of this article is mainly non-motor vehicles, so the final research area is the non-motor vehicle lane and the adjacent motor vehicle lane. The acquired data includes trajectory coordinates, speed, acceleration, and vehicle type.

[0090] Step 1: Initialize the intelligent agent model, which controls the non-motor vehicle to run in the simulation environment and obtain the simulation trajectory.

[0091] First, the data is organized into segments of 80 simulation steps (9.6s) each. A real trajectory segment is extracted from the segment data set. The simulation environment is initialized, and the target simulation vehicle and the environment vehicle are generated according to the same initial state as the real one, including trajectory coordinates, speed, acceleration, and vehicle type.

[0092] Initialize the intelligent agent model. The intelligent agent control model uses a multi-layer perceptron. The designed intelligent agent model state space S includes the expected speed v of the vehicle de , distance from the curb y, speed v x ,v y , the relative position Δx of the nearest non-motor vehicle in the four directions of left front, left rear, right front, and right rear b ,Δy b , relative speed Δv xb ,Δv yb , the relative position Δx of the closest vehicle in the front and rear directions on the motorway v ,Δy v , relative speed Δv xv ,Δv yv The agent outputs the horizontal and vertical acceleration according to the current observed state, and the action space A is a x ,a y .

[0093] The target simulated vehicle in the environment is controlled by the agent model, and the simulation step length corresponding to the real trajectory length is run to obtain the corresponding simulation trajectory. After each segment is completed, the next trajectory segment is selected until the entire data set is traversed.

[0094] Step 2: Compare the similarity between the actual trajectory and the simulated trajectory, and calculate the difference between the two as a reward to provide guidance for the agent update.

[0095] Calculate the behavior similarity score. Set a discriminator to score the real trajectory and the simulated trajectory. Its input is a set of state-action pairs and the output is a score of 0-1. The higher the score, the higher the realism. Use the real trajectory and the generated simulated trajectory to train the discriminator so that it gives high scores to the real trajectory and low scores to the simulated trajectory. Update the parameters of the discriminator through gradient descent. The loss function of the discriminator can be described as:

[0096] L=E π [log(D(s,a))]+E πE [log(1-D(s,a))]

[0097] Where E[·] represents the expected value, π represents the data from the simulation trajectory, πE represents the data from the real trajectory, D represents the set discriminator, and (s, a) represents the state-action pair.

[0098] After the discriminator is updated, it can score the simulated trajectory, and the score obtained is the similarity R between the simulated trajectory and the real trajectory. G =D(s,a).

[0099] Calculate the trajectory position difference. Calculate the distance difference between the trajectory point and the true trajectory at each moment as another position reward R p , the calculation formula is as follows:

[0100]

[0101] In the formula, represents the coordinates of the real trajectory, Represents the coordinates of the simulation trajectory.

[0102] Calculate behavioral constraints. This embodiment calculates efficiency, comfort, and safety constraints to guide the behavior of the agent. The specific calculation formula is as follows:

[0103]

[0104]

[0105] In the formula, R e is the efficiency constraint, R c is the comfort constraint, R s is a safety constraint, w e ,w c is the adjustment coefficient, v t is the current speed, v mean is the average speed of the car, STD(v) is the standard deviation of the speeds of all cars, and a t is the acceleration at the current moment, a meanis the average acceleration of all cars, STD(a) is the standard deviation of acceleration of all cars, r ell is the distance from the point on the ellipse to the center, and r is the distance between other vehicles and the cyclist.

[0106] Step 3: Use the reinforcement learning algorithm to update the agent model based on the calculated reward.

[0107] In the calculation method of rewards, efficiency and comfort constraints are added as reward enhancement. The specific calculation method is:

[0108] R=R basic -w e ×R e -w c ×R c

[0109] Safety constraints are added in a Lagrangian way, and the Lagrangian multiplier is used to control the weight of safety constraints. The optimization objective is as follows:

[0110]

[0111] In the formula, g(θ) represents the constraint condition, represents the sum of the safety penalties in the simulation, R limit represents the limit that the safety constraint is not expected to exceed, L(θ) represents the optimization target, f(θ) represents the optimization target of the original reward, and λ is the Lagrange multiplier. The model parameters θ are updated by gradient ascent, and λ is updated by gradient descent.

[0112] The learning algorithm uses proximal policy optimization to update the agent policy model. Its optimization goal can be expressed as:

[0113]

[0114] In the formula, π θ (a t |s t ) refers to the current strategy, π θold (a t |s t ) refers to the old strategy, A θold refers to the Advantage of the old strategy, and ε refers to the clipping factor (set to 0.2 as recommended).

[0115] The specific steps of overall update are shown in Table 1.

[0116] Step 4: Evaluation of simulation results.

[0117] The model is trained on the training set and the model performance is tested on the test set. This embodiment uses the average distance error per step as an indicator to measure the performance of the test set model. The calculation method is as follows:

[0118]

[0119] In the formula, T is the total duration of the simulation run, is the position of the actual trajectory point at a certain moment, is the position of the simulation trajectory point at a certain moment.

[0120] Finally, the test results show that after running 80 simulation steps (9.6s), the average distance error per step of the model is 1.63m, which meets the accuracy requirements of data-driven simulation.

[0121] In terms of simulation accuracy at the behavioral layer. The KL divergence of the lateral and longitudinal accelerations between the trajectory generated by the simulation model and the real trajectory is 0.304 and 0.083 respectively. The percentage errors of the number of times non-motor vehicles overtake other non-motor vehicles and the number of times non-motor vehicles cross the vehicle-non-motor vehicle separation line are 0.8% and 5.4% respectively. The percentage error between the number of dangerous vehicle behaviors and the real trajectory is 13.3%. The simulation results show that the method used can accurately reproduce the characteristics of individual interactive behaviors of non-motor vehicles.

[0122] In terms of trajectory layer simulation accuracy. The overtaken vehicle is taken as the origin, and the horizontal and vertical relative positions of the overtaking vehicle and the overtaken vehicle are taken as the horizontal and vertical coordinates, and the number of occurrences of each corresponding relative position trajectory point is counted. First, both the simulated trajectory and the real trajectory can simultaneously observe the behavioral characteristics of most cyclists overtaking from the left. Secondly, from the perspective of the safe distance for overtaking, the overtaken vehicle is about 2.5m longitudinally and about 0.6m laterally, and overtaking vehicles are rarely observed. This range is basically consistent with the safe distance counted from the actual data. This result shows that the intelligent agent maintains a reasonable safe overtaking space and can realistically reproduce the overtaking behavior of non-motor vehicles.

[0123] Figure 2 A set of comparison cases between real trajectories and simulated trajectories is provided, where (a) is the real trajectory and (b) is the simulation model running trajectory, describing a scenario in which two electric bicycles invade the adjacent motor vehicle lane and weave in front of the motor vehicle in order to overtake the slower electric bicycle in front, thereby interfering with the motor vehicle. The simulation scenario and the real scenario have the same initial state, and the positions of the traffic participants are the positions of the simulated individual and the real individual in the scenario at the time of crossing the line. Figure 2 It can be seen that the method proposed in this embodiment successfully reproduced a series of interference behaviors of the non-motor vehicle crossing the motor vehicle-non-motor vehicle separation line, overtaking the motor vehicle and completing overtaking, and generated a trajectory close to the real one. In view of this, the non-motor vehicle interference behavior simulation method proposed in this embodiment can generate real and reliable interference behaviors in a virtual test environment, thereby strongly supporting the construction of a real and reliable virtual test environment for autonomous driving and the construction of related technologies.

[0124] In summary, this method can capture the differences in the motion process and interaction patterns of heterogeneous non-motor vehicles, the reward function can accurately capture the behavioral characteristics of different non-motor vehicles, and the Lagrangian method used can accurately restrict the unreasonable behavior of the intelligent agent. This method has good applicability for non-motor vehicle behavior simulation, and the relevant improvements have effectively improved the simulation effect. It can realistically reproduce the interference behaviors of non-motor vehicles such as overtaking, crossing the line and occupying the motor vehicle lane, and has obvious advantages over existing methods in simulation accuracy, behavior similarity, and interference event reproduction accuracy.

[0125] Example 2

[0126] Based on Example 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the data-driven road section non-motor vehicle behavior simulation method as described in Example 1.

[0127] like Figure 3 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0128] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0129] Example 3

[0130] Based on Example 1, this embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the data-driven road section non-motor vehicle behavior simulation method as described in Example 1.

[0131] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0132] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data-driven road section non-motor vehicle behavior simulation method, characterized in that: The non-motor vehicle behavior simulation is realized by using an intelligent agent model, and the acquisition process of the intelligent agent model includes the following steps: Initialize an intelligent agent model that controls the movement of non-motor vehicles in a simulation environment and constructs a simulation trajectory; The difference between the real trajectory and the simulated trajectory generated by the agent model is used as the basic reward of the agent; Based on the basic reward, the agent model is updated through data-driven iteration under efficiency constraints, comfort constraints and safety constraints.

2. A data-driven road section non-motor vehicle behavior simulation method according to claim 1, characterized in that: During the construction of the intelligent agent model, the state space of the intelligent agent includes the expected speed of the vehicle, the distance between the vehicle and the curb, the speed of the vehicle, the relative position of the non-motor vehicle closest to the vehicle in multiple directions, the relative speed of the non-motor vehicle closest to the vehicle in multiple directions, the relative position of the motor vehicle closest to the front and rear directions on the motor vehicle lane, and the relative speed of the motor vehicle closest to the front and rear directions on the motor vehicle lane. The action space of the intelligent agent includes lateral acceleration and longitudinal acceleration.

3. The data-driven road section non-motor vehicle behavior simulation method according to claim 1, characterized in that: In the above-mentioned agent model, the calculation process of the agent's basic reward includes the following steps: Get the state-action pairs of the agent and use the discriminator to calculate the similarity between the real trajectory and the simulated trajectory generated by the agent model; Calculate the coordinate distance difference between the real trajectory and the simulated trajectory generated by the agent model; The similarity between the simulation trajectories and the coordinate distance difference are combined as the basic reward for the agent.

4. A data-driven road section non-motor vehicle behavior simulation method according to claim 3, characterized in that: During the training process of the discriminator, the loss function is: L and E π [log(D(s,a))]+E vE [log(1-D(s,a))] Where L is the loss function, E[·] represents the expected value, π represents the data from the simulation trajectory, πE represents the data from the real trajectory, D represents the discriminator, and (s, a) represents the state-action pair.

5. The data-driven road section non-motor vehicle behavior simulation method according to claim 3, characterized in that: The coordinate distance difference is calculated using the following formula: in, is the coordinate distance difference between the real trajectory and the simulated trajectory generated by the agent model, represents the coordinates of the real trajectory, Represents the coordinates of the simulation trajectory.

6. The method for simulating the behavior of non-motor vehicles on a road section based on data drive according to claim 1, characterized in that: Based on the basic reward, under the efficiency constraint, the comfort constraint and the safety constraint, the process of updating the agent model through data-driven iteration includes the following steps: Adding constraint items corresponding to the efficiency constraint and the comfort constraint to the basic reward of the agent to obtain an enhanced reward of the agent; Establishing a Lagrange multiplier that controls the weight of the safety constraint; Based on the enhanced agent's reward and the Lagrange multiplier, the data-driven agent model is iteratively updated through proximal strategy optimization.

7. A data-driven road section non-motor vehicle behavior simulation method according to claim 6, characterized in that: The efficiency constraint, comfort constraint and safety constraint are: Among them, R e , R c are the constraints corresponding to efficiency constraint and comfort constraint, R s represents the safety constraint, v t is the current speed, v mean is the average speed of the vehicle, STD(v) is the standard deviation of the speeds of all vehicles, and a t is the acceleration at the current moment, a mean is the average acceleration of all cars, STD(a) is the standard deviation of acceleration of all cars, r ell is the distance from the point on the ellipse to the center, and r is the distance between other vehicles and the cyclist.

8. The method for simulating the behavior of non-motor vehicles on a road section based on data drive according to claim 6, characterized in that: The optimization goal of the agent model is: Among them, J(θ) is the optimization objective function of the intelligent agent model, (s t ,a t ) is a state-action pair, π θ (a t |s t ) is the current strategy, π θold (a t |s t ) is the old strategy, A θold is the advantage value of the old strategy, ε is the clipping coefficient, and clip() represents the clipping function.

9. The data-driven road section non-motor vehicle behavior simulation method according to claim 6, characterized in that: The process of implementing data-driven iterative updating of the intelligent agent model through proximal strategy optimization includes the following steps: Initialize the network parameters of Actor network, Critic network, Cost-Critic network, and discriminator respectively: θ0, ω0, μ0, Initialize the Lagrange multiplier λ0; Initialize the experience replay pool M; Use the Actor network to control the target car in the simulation environment, run the simulation, and run the trajectory (s) per step. t ,a t ,r t ,c t ,s t+1 ) is stored in the experience replay pool M; Randomly sample N groups of data (s t ,a t ,r t ,c t ,s t+1 ); Update the discriminator parameters arrive Update the Critic network parameters ω i to i+1 , calculate the TD-error of Cost and update the Cost-Critic parameter μ i to μ i+1 ; Update the Lagrange multiplier λ i to λ i+1 ; Update Actor network parameters θ i to θ i+1 ; Clear the experience replay pool M.

10. The method for simulating the behavior of non-motor vehicles on a road section based on data drive according to claim 9, characterized in that: The update target of the Lagrange multiplier is: Among them, g(θ) represents the constraint condition, R s represents a safety constraint, represents the sum of the safety penalties during the total simulation time T, R limit It represents the limit that the safety constraint is expected not to exceed, L(θ) represents the optimization target, f(θ) represents the optimization target of the original reward, λ is the Lagrange multiplier, and the model parameter θ is updated by gradient ascent, and λ is updated by gradient descent.

Images