A vehicle trajectory optimization method based on imitation learning and related device

Abstract

The present application discloses a vehicle trajectory optimization method based on imitation learning and a related device, the method comprising: obtaining initial trajectory data of a target vehicle; determining a target flow direction of the target vehicle; based on the initial trajectory data, screening a target lane from at least one lane corresponding to the target flow direction; if the initial trajectory data meets a correction condition, extracting lane information of the target lane from the configuration information of the intersection, and based on the initial trajectory data, obtaining current position information and current motion information of the target vehicle and its surrounding vehicles; inputting the current position information, current motion information and lane information of the target lane of the target vehicle and its surrounding vehicles into a generative adversarial imitation learning model to obtain target trajectory data determined by the generative adversarial imitation learning model; and correcting the initial trajectory data based on the target trajectory data.

Description

Technical Field

[0001] The present application relates to the field of intelligent traffic management technology, and in particular to a vehicle trajectory optimization method based on imitation learning and related devices. Background Art

[0002] With the acceleration of urbanization, traffic pressure and safety challenges are becoming increasingly severe. Intelligent Transportation Systems (ITS), as a key technology to meet these challenges, are developing rapidly. ITS integrates advanced information technology, data communication transmission technology and computer technology to achieve real-time monitoring, efficient management and scientific guidance of traffic flow. As an important part of ITS, holographic trajectory construction technology can fully capture and analyze the dynamic behavior of vehicles at intersections, which is crucial to improving the optimization and prediction capabilities of traffic flow.

[0003] At present, the main technical solution for building complex holographic trajectories at intersections relies on radar-visual fusion technology. This technology combines the advantages of radar and visual detection equipment to obtain characteristic information such as the vehicle's position, speed, direction of movement, appearance, and license plate. However, in actual applications, due to factors such as line of sight obstruction, lighting changes, and distance limitations, visual detection equipment often cannot continuously or accurately capture vehicle trajectories, resulting in incomplete trajectory information.

[0004] To solve these problems, the current technical solution adopts a trajectory correction method. The core of this method is to set a preset trajectory, that is, to set a series of expected driving paths and speeds for the vehicle based on road design and traffic rules. In actual applications, the system first obtains the actual driving data of the vehicle through radar and visual detection equipment, and then compares and analyzes this data with the preset trajectory. When it is found that the actual trajectory deviates from the preset trajectory, the system will use an algorithm to correct the trajectory to simulate and restore the actual driving state of the vehicle at the intersection.

[0005] However, although the trajectory correction method has alleviated the detection error to a certain extent, it still has limitations. On the one hand, the setting of the preset trajectory depends on the road design and traffic rules, and lacks in-depth learning and understanding of vehicle behavior. Therefore, in complex traffic environments, especially when there is heavy traffic and sudden traffic incidents, the prediction accuracy of the preset trajectory may be limited. On the other hand, the trajectory correction method mainly relies on the algorithm to compare and analyze the actual trajectory with the preset trajectory, and lacks the ability to predict the vehicle's driving intention and path in real time. This limits the ability of holographic trajectory construction technology to deal with sudden traffic incidents and complex traffic scenarios. Summary of the invention

[0006] In view of the above problems, this application provides a vehicle trajectory optimization method and related devices based on imitation learning to achieve the purpose of better responding to sudden traffic events and complex traffic scenarios. The specific scheme is as follows:

[0007] The first aspect of the present application provides a vehicle trajectory optimization method based on imitation learning, comprising:

[0008] Obtaining initial trajectory data of the target vehicle;

[0009] Determining a target flow direction of the target vehicle;

[0010] Based on the initial trajectory data, selecting a target lane from at least one lane corresponding to the target flow direction;

[0011] If the initial trajectory data satisfies the correction condition, extracting the lane information of the target lane from the configuration information of the intersection, and obtaining the current position information and current motion information of the target vehicle and its surrounding vehicles based on the initial trajectory data;

[0012] Inputting the current position information, current motion information of the target vehicle and its surrounding vehicles, and the lane information of the target lane into a generative adversarial imitation learning model to obtain target trajectory data determined by the generative adversarial imitation learning model;

[0013] Based on the target trajectory data, the initial trajectory data is corrected.

[0014] In a possible implementation, determining the target flow direction of the target vehicle includes:

[0015] If the target vehicle has locked the flow direction, the locked flow direction of the target vehicle is used as the target flow direction of the target vehicle;

[0016] If the target vehicle has an unlocked flow direction, and the flow direction of the entrance lane of the target vehicle is configured as a single flow direction, the single flow direction is used as the target flow direction of the target vehicle;

[0017] If the target vehicle has an unlocked flow direction, and the flow direction of the entrance lane of the target vehicle is configured as multiple flow directions, then by comparing the vehicle flow corresponding to each of the multiple flow directions, the flow direction with the largest vehicle flow is selected from the multiple flow directions as the target flow direction of the target vehicle;

[0018] If the traffic volume corresponding to each flow direction in the multiple flow directions is consistent, a temporary target point is determined based on the historical trajectory of the target vehicle. If the temporary target point is located in one of the multiple flow directions, the flow direction in the multiple flow directions containing the temporary target point is used as the target flow direction of the target vehicle.

[0019] In a possible implementation, selecting a target lane from at least one lane corresponding to the target flow direction based on the initial trajectory data includes:

[0020] Determining the latest trajectory heading angle of the target vehicle based on the current trajectory point of the target vehicle in the initial trajectory data;

[0021] Obtaining an exit track heading angle corresponding to a target flow direction of the target vehicle;

[0022] Determine an average angle between the latest trajectory heading angle and the exit path heading angle;

[0023] An angle between a target point of each lane in at least one lane corresponding to the target flow direction and a current trajectory point of the target vehicle is determined, and a lane having a minimum difference between the angle and the average angle is taken as a target lane.

[0024] In a possible implementation, whether the initial trajectory data satisfies the correction condition is determined by:

[0025] Determine a first distance between a first initial trajectory point in the initial trajectory data that enters the target lane and a target point in the target lane;

[0026] Determine a second distance between the current trajectory point of the target vehicle and the first initial trajectory point in the initial trajectory data;

[0027] If the target flow direction is a left turn flow direction, and if the ratio of the second distance to the first distance is not less than a left turn threshold, then the correction condition is satisfied;

[0028] If the target flow direction is a right turn flow direction, and if the ratio of the second distance to the first distance is not less than a right turn threshold, then the correction condition is satisfied;

[0029] If the target flow direction is a straight flow direction, if the ratio of the second distance to the first distance is not less than a straight threshold, and the target vehicle fails in visual tracking, then a correction condition is satisfied;

[0030] If the target flow direction is a U-turn flow direction, and if the target vehicle fails in visual tracking, the correction condition is met.

[0031] In a possible implementation, the generative adversarial imitation learning model is obtained based on adversarial network training, and the adversarial network includes a generator and a judge;

[0032] The process of training the generative adversarial imitation learning model based on the adversarial network includes:

[0033] Acquire a vehicle driving trajectory at an intersection, and determine an expert state-action pair based on the vehicle driving trajectory at the intersection;

[0034] At the present moment , according to the setting of course distribution set Sampling out vehicles, as multiple sample vehicles;

[0035] Acquire sample information of each of the multiple sample vehicles at the current position; the sample information includes: position information and motion information of the sample vehicle at the current position, position information and motion information of surrounding sample vehicles, and lane information of the target sample lane;

[0036] Processing the sample information corresponding to each sample vehicle according to the current strategy of the generator to generate a trajectory of each sample vehicle;

[0037] Determining a penalty value corresponding to the trajectory of each of the sample vehicles;

[0038] Based on the judgement for each state-action pair in the trajectory of each sample vehicle Scoring is performed to generate a reward value for each of the sample vehicles; the reward value is determined in the following manner:

[0039]

[0040] in, Representation Judger In the parameters Next, according to the state action The value obtained, represents the penalty value corresponding to the trajectory of the sample vehicle;

[0041] Keeping the parameters of the evaluator unchanged, updating the strategy parameters of the generator based on the trust region optimization method; updating the strategy parameters of the generator based on the trust region optimization method includes: solving the following constrained optimization problem:

[0042] ;

[0043] in, Representation strategy Parameters; express expectations; Indicated in The current strategy adopted at any moment, which is based on the old parameters defined; Indicates new strategy; Indicates that the current strategy is Observe the conditions at all times Take action The probability of Indicates that the new strategy Observe the conditions at all times Take action The probability of Indicates the current strategy Under observation conditions The probability distribution of taking actions under Indicates new strategy Under observation conditions The probability distribution of taking actions under express and The KL (Kullback-Leibler) divergence between; represents the step size parameter, which is used to control the maximum change of the strategy in each optimization step; represents the advantage function, which is used to measure the observation condition Take action The expected action value With observer Estimated state value expectation degree of difference between represents the behavior taken by the sample vehicle according to the strategy;

[0044] The advantage function is estimated by the following generalized advantage estimation method:

[0045]

[0046] in, represents the discount rate; It is a parameter between 0 and 1, used to balance the TD (Temporal Difference) error The weight of represents the reward value determined by the judge; and Respectively expressed in Moment and The expected value of the state at the moment;

[0047] Keep the strategy parameters of the generator unchanged, and update the judgment parameters of the judge based on the state-action pairs of the expert and the state-action pairs generated by the new strategy of the generator. ; The judging parameters of the judge The update is performed by the following objective function:

[0048]

[0049] in, represents the expert strategy, represents the new strategy, Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state probability; Indicates that based on the current strategy in state Take action probability; Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state probability; Indicates that based on the expert strategy in the state Take action probability; yes A simplified expression of In the parameters Next, according to the state action The resulting value.

[0050] In a possible implementation, determining the penalty value corresponding to the trajectory of each sample vehicle includes:

[0051] Through the penalty function , determine the penalty value corresponding to the trajectory of each sample vehicle;

[0052] in, represents the minimum distance between any two sample vehicles, 1 indicates the collision penalty value, Indicates the shortest distance between the sample vehicle and the edge of the road, , Indicates the closest distance between the sample vehicle and the left edge of the road, Indicates the closest distance between the sample vehicle and the right edge of the road, Represents the distance penalty value, Indicates that the vehicle kinematic constraints are not met. represents the constraint penalty value, Indicates the emergency brake penalty value. Indicates acceleration.

[0053] In a possible implementation, the collision penalty value is determined in the following manner:

[0054] Extracting the first n consecutive position points from the trajectory of the sample vehicle;

[0055] For each position point among the first n consecutive position points, if the sample vehicle does not collide with surrounding vehicles before moving to the position point, but collides with surrounding vehicles at the position point, the position point is marked as abnormal; if the sample vehicle does not collide with surrounding vehicles before moving to the position point, and does not collide with surrounding vehicles at the position point, the position point is marked as a candidate;

[0056] If there is a position point marked as a candidate among each of the first n consecutive position points, the position point arranged last among the position points marked as candidates is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain a collision penalty value;

[0057] If the first n consecutive position points are all marked as abnormal, the first position point among the first n consecutive position points is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain the collision penalty value.

[0058] On the other hand, the present application provides a vehicle trajectory optimization device based on imitation learning, comprising:

[0059] A first acquisition module is used to obtain initial trajectory data of the target vehicle;

[0060] A first determination module, used to determine the target flow direction of the target vehicle;

[0061] a screening module, configured to screen a target lane from at least one lane corresponding to the target flow direction based on the initial trajectory data;

[0062] A second acquisition module is used to extract the lane information of the target lane from the configuration information of the intersection if the initial trajectory data meets the correction condition, and obtain the current position information and current motion information of the target vehicle and its surrounding vehicles based on the initial trajectory data;

[0063] A second determination module is used to input the current position information, current motion information of the target vehicle and its surrounding vehicles and the lane information of the target lane into the generative adversarial imitation learning model to obtain the target trajectory data determined by the generative adversarial imitation learning model;

[0064] A correction module is used to correct the initial trajectory data based on the target trajectory data.

[0065] A third aspect of the present application provides an electronic device, including:

[0066] The memory is used to store computer programs;

[0067] The processor is used to execute the computer program so that the electronic device can implement the vehicle trajectory optimization method based on imitation learning as described in any one of the above.

[0068] A fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the vehicle trajectory optimization method based on imitation learning as described in any one of the above.

[0069] In this application, by training the generative adversarial imitation learning model, the generative adversarial imitation learning model can learn how to predict the future driving intention and path of the vehicle based on the current position, speed, direction of movement and dynamic changes of surrounding vehicles. Therefore, the current position information, current movement information and lane information of the target vehicle and its surrounding vehicles are input into the generative adversarial imitation learning model, and the generative adversarial imitation learning model can generate more accurate and reliable target trajectory data. Therefore, the target trajectory data can be used to intelligently correct the incomplete or deviated vehicle trajectory determined by the radar equipment and the visual detection equipment to deal with problems such as line of sight obstruction, illumination change and distance too far, so as to achieve continuous and accurate capture of the vehicle motion state in the intersection. In addition, by using the generative adversarial imitation learning model to deeply learn and understand vehicle behavior, it is possible to abandon the dependence on the preset trajectory, realize accurate prediction of the vehicle's driving intention and path, and better deal with sudden traffic events and complex traffic scenes. In addition, by generating adversarial imitation learning models, the data fusion process in complex traffic scenes can be optimized, such as the accuracy and adaptability of radar-visual fusion in traffic congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0071] Figure 1 A schematic diagram of a process flow of a vehicle trajectory optimization method based on imitation learning provided in Example 1 of the present application;

[0072] Figure 2 A schematic diagram of a process for guiding vehicle movement provided in this application;

[0073] Figure 3A schematic diagram of another implementation scenario of a vehicle trajectory optimization method based on imitation learning provided in this application;

[0074] Figure 4 A schematic diagram of the training process of a generator and a judge provided in this application;

[0075] Figure 5 A schematic diagram of another implementation scenario of a vehicle trajectory optimization method based on imitation learning provided in this application;

[0076] Figure 6 A schematic diagram of the structure of a vehicle trajectory optimization device based on imitation learning provided in this application. DETAILED DESCRIPTION

[0077] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0078] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0079] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0080] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0081] Reference Figure 1 , is a flow chart of a vehicle trajectory optimization method based on imitation learning provided in Example 1 of the present application, such as Figure 1 As shown, the method may include but is not limited to the following steps:

[0082] Step S101: Obtain initial trajectory data of the target vehicle.

[0083] In this embodiment, the radar device can be used to accurately capture the dynamic data such as the position information, speed and movement direction of the target vehicle, and the visual detection device can be used to capture the appearance characteristics (such as model, color, etc.) and unique identification information such as the license plate of the target vehicle. Based on the dynamic data such as the position information, speed and movement direction of the target vehicle and the appearance characteristics (such as model, color, etc.) and license plate of the target vehicle, the initial trajectory data of the target vehicle is generated.

[0084] Step S102: Determine the target flow direction of the target vehicle.

[0085] The target flow direction can be understood as the direction or path that the target vehicle may or is expected to travel in the future. Determining the target flow direction can help to more accurately screen and locate the lanes that the target vehicle may travel.

[0086] Step S103: based on the initial trajectory data, select a target lane from at least one lane corresponding to the target flow direction.

[0087] After determining the target flow direction of the target vehicle, the lane most likely to be driven by the target vehicle can be screened out from multiple lanes corresponding to the target flow direction according to the initial trajectory data, i.e., the target lane. This step helps to further narrow the driving range of the target vehicle and facilitates subsequent analysis and processing.

[0088] Step S104: If the initial trajectory data meets the correction condition, the lane information of the target lane is extracted from the configuration information of the intersection, and the current position information and current motion information of the target vehicle and its surrounding vehicles are obtained based on the initial trajectory data.

[0089] When the initial trajectory data meets certain correction conditions, lane information related to the target lane can be extracted from the configuration information of the intersection, such as lane number, lane corresponding flow direction, exit target point, etc.

[0090] In this embodiment, the current position information and current motion information of the target vehicle can be extracted from the initial trajectory data. Based on the initial trajectory data, the surrounding vehicles of the target vehicle can be determined. The current position information and current motion information of the surrounding vehicles can be determined through the trajectory data of the surrounding vehicles.

[0091] The current motion information of the target vehicle may include but is not limited to: at least one of the current speed, current acceleration and current vehicle turning angle of the target vehicle. Accordingly, the current motion information of the surrounding vehicles of the target vehicle may also include but is not limited to: at least one of the current speed, current acceleration and current vehicle turning angle of the surrounding vehicles.

[0092] If the initial trajectory data does not meet the correction condition, the initial trajectory data can be used to output the position in the initial trajectory data. The output position can be used to guide the target vehicle to move.

[0093] Step S105: input the current position information, current motion information of the target vehicle and its surrounding vehicles, and the lane information of the target lane into the generative adversarial imitation learning model to obtain the target trajectory data determined by the generative adversarial imitation learning model.

[0094] In this embodiment, the generative adversarial imitation learning model can be trained based on the actual vehicle driving trajectory at the intersection, and the strategy parameters of the generative adversarial imitation learning can be updated so that the generative adversarial imitation learning model can imitate the expert strategy (i.e., the strategy corresponding to the actual vehicle behavior), deeply learn and understand the vehicle behavior, so that the generative adversarial imitation learning model can generate action parameters similar to the actual vehicle behavior, and then obtain the target trajectory data, ensuring that the target trajectory data is more reasonable.

[0095] In this embodiment, the vehicle driving trajectories at the intersection can be grouped by flow direction to obtain the vehicle driving trajectories corresponding to the left turn flow, straight flow, right turn flow and U-turn flow direction, and the vehicle driving trajectories corresponding to the left turn flow, straight flow, right turn flow and U-turn flow direction are trained simultaneously, so that the generative adversarial imitation learning model can simultaneously imitate and learn the expert strategies of different flow directions.

[0096] Step S106: correcting the initial trajectory data based on the target trajectory data.

[0097] In this embodiment, the target trajectory data may replace the initial trajectory data to complete the correction of the initial trajectory data.

[0098] Alternatively, the initial trajectory data may be updated based on the target trajectory data to complete the correction.

[0099] After the initial trajectory data is corrected, the target vehicle can be guided to move based on the corrected trajectory data. In the process of guiding the target vehicle to move, it can be determined whether to continue to generate target trajectory data. For example, Figure 2 As shown, after the vehicle moves and outputs the position after movement, it can be determined based on the position after movement whether it reaches the effective area of ​​the radar device and the visual detection device. If so, the step of generating target trajectory data can be ended. If not, the position after movement can be used as the new current position information, and the new current motion information of the target vehicle can be obtained from the target trajectory data, and the new current position information and new current motion information of the surrounding vehicles can be obtained, and the step of generating target trajectory data can be continued.

[0100] In this embodiment, by training the generative adversarial imitation learning model, the generative adversarial imitation learning model can learn how to predict the future driving intention and path of the vehicle based on the current position, speed, direction of movement and dynamic changes of surrounding vehicles. Therefore, the current position information, current movement information and lane information of the target vehicle and its surrounding vehicles are input into the generative adversarial imitation learning model, and the generative adversarial imitation learning model can generate more accurate and reliable target trajectory data. Therefore, the target trajectory data can be used to intelligently correct the incomplete or deviated vehicle trajectory determined by the radar equipment and the visual detection equipment to deal with problems such as line of sight obstruction, illumination change and too far distance, so as to achieve continuous and accurate capture of the vehicle motion state in the intersection. In addition, by using the generative adversarial imitation learning model to deeply learn and understand vehicle behavior, it is possible to abandon the dependence on the preset trajectory, realize accurate prediction of the vehicle's driving intention and path, and better deal with sudden traffic events and complex traffic scenes. In addition, by generating adversarial imitation learning models, the data fusion process in complex traffic scenes can be optimized, such as the accuracy and adaptability of radar-visual fusion in traffic congestion.

[0101] As another optional embodiment of the present application, a vehicle trajectory optimization method based on imitation learning is provided in Embodiment 2 of the present application. This embodiment is mainly an implementation method of step S102 in the above embodiment 1. Step S102 may include but is not limited to the following steps:

[0102] Step S1021: If the target vehicle has locked its flow direction, the locked flow direction of the target vehicle is used as the target flow direction of the target vehicle.

[0103] In this embodiment, if the target vehicle's driving direction has been determined and is not allowed to change due to its driving path or traffic signal control before entering the intersection, it can be determined that the target vehicle has locked its direction.

[0104] In most cases, vehicles in the through flow direction may be "locked" when approaching an intersection, because through vehicles usually do not need to make a direction choice before the intersection, but instead continue directly in their current lane.

[0105] Step S1022: If the flow direction of the target vehicle is not locked, and the flow direction of the entrance lane of the target vehicle is configured as a single flow direction, the single flow direction is used as the target flow direction of the target vehicle.

[0106] One-way flow can be understood as: the entrance lane only allows vehicles in a single direction to travel, such as only allowing left turns or only allowing right turns.

[0107] Step S1023: If the target vehicle has not locked its flow direction, and the entrance lane flow direction of the target vehicle is configured as multiple flow directions, then by comparing the vehicle flow corresponding to each flow direction in the multiple flow directions, the flow direction with the largest vehicle flow is selected from the multiple flow directions as the target flow direction of the target vehicle.

[0108] If multiple flow directions can include left-turn flow and straight-ahead flow, the number of vehicles turning left and going straight in the intersection can be monitored to determine the current flow direction of the vehicle.

[0109] If the number of left-turning vehicles is greater than the number of straight-moving vehicles, the left-turning flow is used as the target flow direction of the target vehicle.

[0110] If the number of left-turning vehicles is less than the number of straight-moving vehicles, the straight-moving direction is used as the target flow direction of the target vehicle.

[0111] If multiple flow directions can include right-turn flow and straight-ahead flow, the flow direction can be determined by monitoring the number of right-turn and straight-ahead vehicles in the intersection.

[0112] If the number of right-turn flow vehicles is greater than the number of straight-moving flow vehicles or the number of straight-moving flow vehicles is 0, the right-turn flow direction may be used as the target flow direction of the target vehicle.

[0113] Step S1024: If the vehicle flows corresponding to the multiple flow directions are consistent, a temporary target point is determined based on the historical trajectory of the target vehicle; if the temporary target point is located in one of the multiple flow directions, the flow direction in the multiple flow directions containing the temporary target point is used as the target flow direction of the target vehicle.

[0114] In this embodiment, when the traffic flow in each direction of a multi-direction lane is consistent, selecting a temporary target point based on the historical trajectory of the target vehicle to determine the target flow direction can more accurately reflect the vehicle's true intentions and driving habits, and can improve the accuracy of the target flow direction.

[0115] As another optional embodiment of the present application, a vehicle trajectory optimization method based on imitation learning is provided in Embodiment 3 of the present application. This embodiment is mainly an implementation method of step S103 in the above embodiment 1. Step S103 may include but is not limited to the following steps:

[0116] Step S1031: determining the latest trajectory heading angle of the target vehicle based on the current trajectory point of the target vehicle in the initial trajectory data.

[0117] In this embodiment, the current trajectory point of the target vehicle in the initial trajectory data can be and the original trajectory point N intervals from the current trajectory point of the target vehicle, and calculate the latest trajectory heading angle of the target vehicle Wherein, N can be set as needed and is not limited in this application. For example, N can be 3.

[0118] Step S1032: Obtain an exit course heading angle corresponding to the target flow direction of the target vehicle.

[0119] Step S1033: determine the average angle between the latest trajectory heading angle and the exit path heading angle.

[0120] In this embodiment, the average angle can be calculated by the following relationship:

[0121]

[0122] represents the average angle, Indicates the latest track heading angle, Indicates the exit course heading angle.

[0123] Step S1034, determining the angle between the target point of each lane in at least one lane corresponding to the target flow direction and the current trajectory point of the target vehicle, and taking the lane with the smallest difference between the angle and the average angle as the target lane.

[0124] like Figure 3 As shown, the target point of each lane in at least one lane corresponding to the target flow direction and the current trajectory point of the target vehicle can be connected, and the angle between the target point of each lane in at least one lane corresponding to the target flow direction and the current trajectory point of the target vehicle can be calculated, and the lane with the smallest difference between the angle and the average angle is taken as the target lane.

[0125] As another optional embodiment of the present application, a vehicle trajectory optimization method based on imitation learning is provided in Embodiment 4 of the present application. This embodiment is mainly an implementation method of a method for determining whether the initial trajectory data in Embodiment 1 meets the correction condition. Whether the initial trajectory data meets the correction condition can be determined by the following method:

[0126] Step S11, determining a first distance between a first initial trajectory point in the initial trajectory data that enters the target lane and a target point in the target lane.

[0127] In this embodiment, the coordinates of the first initial trajectory point in the intersection entering the target lane in the initial trajectory data may be used. and the coordinates of the target point of the target lane , calculate the distance between the first initial trajectory point and the target point of the target lane (expressed as ),Will As the first distance.

[0128] Step S12: determining a second distance between the current trajectory point of the target vehicle and the first initial trajectory point in the initial trajectory data.

[0129] In this embodiment, the coordinates of the current trajectory point of the target vehicle in the initial trajectory data can be used. and the coordinates of the first initial trajectory point , calculate the distance between the current trajectory point of the target vehicle and the first initial trajectory point (expressed as ),Will As the second distance.

[0130] Step S13: If the target flow direction is a left turn flow direction, and if the ratio of the second distance to the first distance is not less than a left turn threshold, the correction condition is satisfied.

[0131] In this embodiment, it can be compared and the left turn threshold (expressed as ) size, if Not less than , it is determined that the correction condition is met.

[0132] Step S14: If the target flow direction is a right turn flow direction, and if the ratio of the second distance to the first distance is not less than a right turn threshold, the correction condition is satisfied.

[0133] In this embodiment, it can be compared and the right turn threshold (expressed as ) size, if Not less than , it is determined that the correction condition is met.

[0134] Step S15: If the target flow direction is a straight flow direction, if the ratio of the second distance to the first distance is not less than a straight threshold, and the target vehicle fails in visual tracking, then the correction condition is met.

[0135] In this embodiment, it can be compared and the straight-line threshold (expressed as ) size, if Not less than , the target vehicle can be marked as locked flow state. When it is further determined that the target vehicle fails in visual tracking, it is determined that the correction condition is met.

[0136] Step S16: If the target flow direction is a U-turn flow direction and if the target vehicle fails in visual tracking, a correction condition is met.

[0137] Except for the situation where vehicles in the straight flow direction may lock the flow direction before meeting the correction conditions, vehicles in the left-turn flow direction and the right-turn flow direction may lock the flow direction while meeting the correction conditions. Vehicles in the U-turn flow direction may lock the flow direction while meeting the correction conditions after tracking fails.

[0138] In this embodiment, by setting the thresholds of different flow directions ( , , ), and based on the actual driving distance of the vehicle ( ) and the reference distance ( ) to determine whether it has entered the "untrusted area" (i.e., dynamic blind spot). Compared with the method that relies on the physical calibration of radar equipment and visual detection equipment to determine the range of the blind spot, it is more flexible and can adapt to changes in different environments and intersections. When a vehicle enters a dynamic blind spot, the system automatically enters the correction mode and performs intelligent correction on the vehicle trajectory in the blind spot. This helps to reduce trajectory errors and misjudgments caused by blind spots. By dynamically demarcating blind spots and making intelligent corrections, this method improves the robustness of the system, enabling it to more accurately track and predict vehicle trajectories. In addition, the radar-vision fusion process can be improved so that radar and visual information can be more effectively combined to improve the accuracy and reliability of overall perception.

[0139] As another optional embodiment of the present application, a vehicle trajectory optimization method based on imitation learning is provided in Example 5 of the present application. This embodiment is mainly an implementation method of the generative adversarial imitation learning model described in the above Example 1. The generative adversarial imitation learning model can be obtained based on adversarial network training. The adversarial network can include a generator and a judge.

[0140] In this embodiment, the discount rate during the training process can be determined , batch size , number of training sessions , course learning process, maximum number of vehicles for training and strategies For example, you can choose two training stages, in which the first training stage uses a discount rate of 0.95 and a batch size of 10,000 observation-action pairs. , the number of training iterations is 1000; and every 200 iterations, 10 vehicles are added to the environment to conduct a curriculum learning process, which improves learning efficiency and performance by gradually increasing the difficulty of the task. In the second training phase (fine-tuning phase), the discount rate is 0.99 and the batch size is 40,000 observation-action pairs. , the training number is 200 iterations, and 100 vehicles are used for training. Strategy The action that the vehicle should perform can be selected based on the current environment state and policy parameters.

[0141] The discount rate can be understood as a coefficient between 0 and 1 used to measure the current value of future rewards in scenarios such as reinforcement learning. It determines the degree of importance attached to future rewards. A higher discount rate places more emphasis on future rewards, while a lower discount rate places more emphasis on current rewards.

[0142] An observation-action pair can be understood as a combination of an observation received by the vehicle at the current moment (an observation is information obtained from the environment, which describes the current state or part of the state of the environment) and the action taken subsequently.

[0143] In this embodiment, a penalty function may be defined according to vehicle kinematic constraints and anti-collision rules.

[0144] In this embodiment, the strategy parameters of the generator can be initialized , the judging parameters of the judge (By updating the judgment parameters , the critic can learn to evaluate whether the state-action pair is generated by the expert strategy or the generator strategy), step size parameter (used to control the maximum change of the strategy in each optimization step to ensure that the strategy update is both effective and stable. This parameter is a key hyperparameter for adjusting the stability of the algorithm and the learning progress), course distribution set (i.e., a concept in curriculum learning that defines the distribution of task difficulty that an agent will face during training. In a multi-agent (i.e., vehicle) environment, the curriculum distribution set It can be defined as a time-varying distribution that gradually increases the number of agents controlled by the policy, thereby gradually increasing the difficulty of training. This approach helps agents gradually learn to handle more complex interaction scenarios and avoid facing overly complex decision-making problems at the beginning.

[0145] In the PS-GAIL (Parameter Sharing Generative Adversarial Imitation Learning) method, the policy parameters Can be a shared policy parameter In a multi-agent environment, all agents share the same set of policy parameters. This means that the agents will take the same policy actions when given an input. By sharing policy parameters, agents are able to learn driving strategies that interact with other agents in complex traffic scenarios. Sharing policy parameters helps reduce model complexity, improve training efficiency, and promote collaborative behavior between agents.

[0146] The process of training the generative adversarial imitation learning model based on the adversarial network includes:

[0147] Step S31, obtaining the vehicle driving trajectory at the intersection, and determining the state-action pair of the expert based on the vehicle driving trajectory at the intersection.

[0148] In this embodiment, the vehicle driving trajectories at the intersection can be collected by using drones, mobile phone positioning data, etc., and the trajectories that are too short and discontinuous can be eliminated to obtain the remaining vehicle driving trajectories.

[0149] In this embodiment, the remaining vehicle driving trajectories can be grouped into flow directions to accommodate different driving intentions such as left turn, straight ahead, and right turn. Based on the expert strategy, discrete trajectory points are extracted from each vehicle driving trajectory corresponding to each flow direction as an expert state-action pair. , forming an expert behavior set . For different driving intentions, training is performed simultaneously. Indicates state (e.g., position, speed, etc.), Indicates an action (such as acceleration, deceleration, turning, etc.).

[0150] Step S32: At the current moment , according to the setting of course distribution set Sampling out vehicles as multiple sample vehicles.

[0151] Step S33, obtaining sample information of each of the multiple sample vehicles at the current position; the sample information includes: position information and motion information of the sample vehicle at the current position and position information and motion information of surrounding sample vehicles and lane information of the target sample lane.

[0152] The position information and motion information can refer to the relevant introduction to the current position information and current motion information of the target vehicle in the aforementioned embodiment, which will not be repeated here.

[0153] Step S34: Process the sample information corresponding to each sample vehicle according to the current strategy of the generator to generate the trajectory of each sample vehicle.

[0154] Step S35: Determine the penalty value corresponding to the trajectory of each sample vehicle.

[0155] In this embodiment, if the behavior of the vehicle violates the rules, a penalty value may be generated.

[0156] Step S36: Based on the evaluator, for each state-action pair in the trajectory of each sample vehicle Scoring is performed to generate a reward value for each of the sample vehicles; the reward value is determined in the following manner:

[0157]

[0158] in, Representation Judger In the parameters Next, according to the state action The value obtained, Represents the penalty value corresponding to the trajectory of the sample vehicle.

[0159] When a vehicle's behavior violates the rules, This will create a positive value, thus reducing the total reward. This way, when making decisions, the vehicle will tend to avoid actions that will result in penalties.

[0160] Step S37, keeping the parameters of the evaluator unchanged, and updating the strategy parameters of the generator based on the trust region optimization method; updating the strategy parameters of the generator based on the trust region optimization method includes: solving the following constrained optimization problem:

[0161] ;

[0162] in, Representation strategy Parameters; express expectations; Indicated in The current strategy adopted at any moment, which is based on the old parameters defined; Indicates new strategy; Indicates that the current strategy is Observe the conditions at all times Take action The probability of Indicates that the new strategy Observe the conditions at all times Take action The probability of Indicates the current strategy Under observation conditions The probability distribution of taking actions under Indicates new strategy Under observation conditions The probability distribution of taking actions under express and The KL (Kullback-Leibler) divergence between; represents the step size parameter, which is used to control the maximum change of the strategy in each optimization step; represents the advantage function, which is used to measure the observation condition Take action The expected action value With observer Estimated state value expectation the degree of difference between Indicates the action that the vehicle should perform based on the current strategy of the generator;

[0163] The advantage function is estimated by the following generalized advantage estimation method:

[0164]

[0165] in, represents the discount rate; It is a parameter between 0 and 1, used to balance the TD (Temporal Difference) error The weight of represents the reward value determined by the judge; and Respectively expressed in Moment and The expected value of the state at the moment;

[0166] Step S38: Keep the strategy parameters of the generator unchanged, and update the evaluation parameters of the evaluator based on the state-action pairs of the expert and the state-action pairs generated by the new strategy of the generator. ; The judging parameters of the judge The update is performed by the following objective function:

[0167]

[0168] in, represents the expert strategy, represents the new strategy, Indicates that the policy is being executed Time state action pair The probability of being accessed, i.e., the occupancy measure; Indicated in strategy Next time In state The probability of Indicates that based on the current strategy in state Take action The probability of Indicates that the policy is being executed Time state action pair The probability of being accessed, i.e., the occupancy measure; Indicated in strategy Next time In state The probability of Indicates that based on the expert strategy in the state Take action The probability of yes A simplified expression of In the parameters Next, according to the state action The resulting value.

[0169] If the training end condition is not met, steps S31-S37 can be continued; if the training end condition is met, the training can be terminated, and the generator in the adversarial network can be used as a generative adversarial imitation learning model. The training end condition can be set as needed and is not limited in this application. For example, the training end condition may include reaching the number of iterations or satisfying the objective function requirements (e.g., the judge indicator ). Now combined Figure 4 , the iterative execution process of steps S31-S37 is described as follows: Figure 4 As shown, on the basis of determining the training process, defining the penalty function, and initializing various parameters, the vehicle driving trajectories at the intersection are obtained, the vehicle driving trajectories are grouped by flow direction, and the corresponding expert behavior set is obtained. The vehicles are sampled according to the setting of the course distribution set; after obtaining the sample information of each of the multiple sample vehicles at the current position, the sample information corresponding to each sample vehicle is processed according to the current strategy of the generator, and after the trajectory of each sample vehicle is generated, it can be evaluated whether the training number is reached or the objective function requirements are met. If the training number is not reached or the objective function requirements are not met, the evaluator can be for each state-action pair in the trajectory of each sample vehicle. Scoring is performed to generate a reward value for each of the sample vehicles; keeping the parameters of the judge unchanged, updating the strategy parameters of the generator based on the trust region optimization method, wherein the reward value generated by the judge can be used in the trust region optimization method; after updating the strategy parameters of the generator, the generator learns a new strategy; keeping the strategy parameters of the generator unchanged, updating the judgment parameters of the judge based on the state-action pairs of the expert and the state-action pairs generated by the new strategy of the generator , the judging parameters of the judge Updated by the objective function.

[0170] If the number of training times is reached or the objective function requirement is met, the training ends.

[0171] As another optional embodiment of the present application, a vehicle trajectory optimization method based on imitation learning is provided in Embodiment 6 of the present application. This embodiment is mainly an implementation method of step S35 in the above-mentioned embodiment 5. Step S35 may include but is not limited to the following steps:

[0172] Step S351: Penalty function , determine the penalty value corresponding to the trajectory of each sample vehicle.

[0173] in, represents the minimum distance between any two sample vehicles, 1 indicates the collision penalty value, Indicates the shortest distance between the sample vehicle and the edge of the road, , Indicates the closest distance between the sample vehicle and the left edge of the road, Indicates the closest distance between the sample vehicle and the right edge of the road, Represents the distance penalty value, Indicates that the vehicle kinematic constraints are not met. represents the constraint penalty value, Indicates the emergency brake penalty value. Indicates acceleration.

[0174] Understandably, There may be one penalty value (ie, one of a collision penalty value, a distance penalty value, a constraint penalty value, and an emergency brake penalty value) or multiple penalty values ​​(ie, at least two of a collision penalty value, a distance penalty value, a constraint penalty value, and an emergency brake penalty value).

[0175] As another optional embodiment of the present application, a vehicle trajectory optimization method based on imitation learning is provided in Embodiment 7 of the present application. This embodiment is mainly an implementation method of the collision penalty value in the above-mentioned Embodiment 6. The collision penalty value can be determined in the following manner:

[0176] Step S41: extract the front track from the track of the sample vehicle. consecutive locations.

[0177] In this embodiment, the The size of is set, and there is no limit in this application. For example, Can be set to 6.

[0178] forward The continuous position points can be expressed as .

[0179] Step S42: For each position point in the continuous position points, if the sample vehicle does not collide with the surrounding vehicles before moving to the position point, but collides with the surrounding vehicles at the position point, the position point is marked as abnormal; if the sample vehicle does not collide with the surrounding vehicles before moving to the position point, and does not collide with the surrounding vehicles at the position point, the position point is marked as a candidate.

[0180] For example, for , if the sample vehicle moves to There was no collision with the surrounding vehicles before, but the sample vehicle If it collides with the surrounding vehicles, it can Marked as abnormal, that is, not moving in this round. There was no collision with the surrounding vehicles before, but the sample vehicle There is no collision with the surrounding vehicles. Marked as candidate, that is, this round can go to .

[0181] Step S43: If the There is a position point marked as a candidate among each of the consecutive position points, the position point arranged last among the position points marked as candidates is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain the collision penalty value.

[0182] For example, if , middle is the position point marked as a candidate, is a candidate point marked as anomaly, then as the new current position of the sample vehicle, and The corresponding penalty values ​​are accumulated to obtain the collision penalty value.

[0183] Step S44: If the The consecutive position points are marked as abnormal. The first position point among the consecutive position points is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain the collision penalty value.

[0184] For example, if , are marked as abnormal and can be as the new current position of the sample vehicle, and The corresponding penalty values ​​are accumulated to obtain the collision penalty value.

[0185] If the previous The consecutive position points are marked as abnormal. The first position point among the consecutive position points is used as the new current position of the sample vehicle to avoid the inability to determine the new current position, which affects the model training.

[0186] In this embodiment, by optimizing the anti-collision judgment logic, unnecessary calculations can be reduced, thereby speeding up the model training speed. Although the calculation process is simplified, by accumulating the penalty values ​​corresponding to the position points marked as abnormal, the collision risk faced by the sample vehicle in the trajectory can be quantified. This quantitative evaluation helps to understand the safety of vehicle behavior more accurately. For example, Figure 5 As shown, after the generative adversarial imitation learning model is trained by the collision penalty value, the target trajectory data generated by the generative adversarial imitation learning model can avoid collisions between the target vehicle and other vehicles after correcting the initial trajectory data.

[0187] Next, the vehicle trajectory optimization device based on imitation learning provided in the present application is introduced. The vehicle trajectory optimization device based on imitation learning introduced below and the vehicle trajectory optimization method based on imitation learning introduced above can refer to each other.

[0188] Reference Figure 6 The vehicle trajectory optimization device based on imitation learning includes: a first acquisition module 100, a first determination module 200, a screening module 300, a second acquisition module 400, a second determination module 500 and a correction module 600.

[0189] The first obtaining module 100 is used to obtain initial trajectory data of the target vehicle.

[0190] The first determination module 200 is used to determine the target flow direction of the target vehicle.

[0191] The screening module 300 is used to screen a target lane from at least one lane corresponding to the target flow direction based on the initial trajectory data.

[0192] The second acquisition module 400 is used to extract the lane information of the target lane from the configuration information of the intersection if the initial trajectory data meets the correction condition, and obtain the current position information and current motion information of the target vehicle and its surrounding vehicles based on the initial trajectory data.

[0193] The second determination module 500 is used to input the current position information, current motion information of the target vehicle and its surrounding vehicles and the lane information of the target lane into the generative adversarial imitation learning model to obtain the target trajectory data determined by the generative adversarial imitation learning model.

[0194] The correction module 600 is used to correct the initial trajectory data based on the target trajectory data.

[0195] The first determination module 200 may be specifically used for:

[0196] If the target vehicle has locked the flow direction, the locked flow direction of the target vehicle is used as the target flow direction of the target vehicle;

[0197] If the target vehicle has an unlocked flow direction, and the flow direction of the entrance lane of the target vehicle is configured as a single flow direction, the single flow direction is used as the target flow direction of the target vehicle;

[0198] If the target vehicle has an unlocked flow direction, and the flow direction of the entrance lane of the target vehicle is configured as multiple flow directions, then by comparing the vehicle flow corresponding to each of the multiple flow directions, the flow direction with the largest vehicle flow is selected from the multiple flow directions as the target flow direction of the target vehicle;

[0199] If the traffic volume corresponding to each flow direction in the multiple flow directions is consistent, a temporary target point is determined based on the historical trajectory of the target vehicle. If the temporary target point is located in one of the multiple flow directions, the flow direction in the multiple flow directions containing the temporary target point is used as the target flow direction of the target vehicle.

[0200] The screening module can be used to:

[0201] Determining the latest trajectory heading angle of the target vehicle based on the current trajectory point of the target vehicle in the initial trajectory data;

[0202] Obtaining an exit track heading angle corresponding to a target flow direction of the target vehicle;

[0203] Determine an average angle between the latest trajectory heading angle and the exit path heading angle;

[0204] An angle between a target point of each lane in at least one lane corresponding to the target flow direction and a current trajectory point of the target vehicle is determined, and a lane having a minimum difference between the angle and the average angle is taken as a target lane.

[0205] The device may also include:

[0206] The judgment module is used to:

[0207] Determine a first distance between a first initial trajectory point in the initial trajectory data that enters the target lane and a target point in the target lane;

[0208] Determine a second distance between the current trajectory point of the target vehicle and the first initial trajectory point in the initial trajectory data;

[0209] If the target flow direction is a left turn flow direction, and if the ratio of the second distance to the first distance is not less than a left turn threshold, then the correction condition is satisfied;

[0210] If the target flow direction is a right turn flow direction, and if the ratio of the second distance to the first distance is not less than a right turn threshold, then the correction condition is satisfied;

[0211] If the target flow direction is a straight flow direction, if the ratio of the second distance to the first distance is not less than a straight threshold, and the target vehicle fails in visual tracking, then a correction condition is satisfied;

[0212] If the target flow direction is a U-turn flow direction, and if the target vehicle fails in visual tracking, the correction condition is met.

[0213] The device may further include: a training module.

[0214] The generative adversarial imitation learning model is obtained based on adversarial network training, and the adversarial network includes a generator and a judge;

[0215] Training modules for:

[0216] Acquire a vehicle driving trajectory at an intersection, and determine an expert state-action pair based on the vehicle driving trajectory at the intersection;

[0217] At the present moment , according to the setting of course distribution set Sampling out vehicles, as multiple sample vehicles;

[0218] Acquire sample information of each of the multiple sample vehicles at the current position; the sample information includes: position information and motion information of the sample vehicle at the current position, position information and motion information of surrounding sample vehicles, and lane information of the target sample lane;

[0219] Processing the sample information corresponding to each sample vehicle according to the current strategy of the generator to generate a trajectory of each sample vehicle;

[0220] Determining a penalty value corresponding to the trajectory of each of the sample vehicles;

[0221] Based on the judgement for each state-action pair in the trajectory of each sample vehicle Scoring is performed to generate a reward value for each of the sample vehicles; the reward value is determined in the following manner:

[0222]

[0223] in, Representation Judger In the parameters Next, according to the state action The value obtained, represents the penalty value corresponding to the trajectory of the sample vehicle;

[0224] Keeping the parameters of the evaluator unchanged, updating the strategy parameters of the generator based on the trust region optimization method; updating the strategy parameters of the generator based on the trust region optimization method includes: solving the following constrained optimization problem:

[0225] ;

[0226] in, Representation strategy Parameters; express expectations; Indicated in The current strategy adopted at any moment, which is based on the old parameters defined; Indicates new strategy; Indicates that the current strategy is Observe the conditions at all times Take action probability; Indicates that the new strategy Observe the conditions at all times Take action probability; Indicates the current strategy Under observation conditions The probability distribution of taking actions under Indicates new strategy Under observation conditions The probability distribution of taking actions under express and The KL (Kullback-Leibler) divergence between; represents the step size parameter, which is used to control the maximum change of the strategy in each optimization step; represents the advantage function, which is used to measure the observation condition Take action The expected action value With observer Estimated state value expectation degree of difference between represents the behavior taken by the sample vehicle according to the strategy;

[0227] The advantage function is estimated by the following generalized advantage estimation method:

[0228]

[0229] in, represents the discount rate; It is a parameter between 0 and 1, used to balance the TD (Temporal Difference) error The weight of represents the reward value determined by the judge; and Respectively expressed in Moment and The expected value of the state at the moment;

[0230] Keep the strategy parameters of the generator unchanged, and update the judgment parameters of the judge based on the state-action pairs of the expert and the state-action pairs generated by the new strategy of the generator. ; The judging parameters of the judge The update is performed by the following objective function:

[0231]

[0232] in, represents the expert strategy, represents the new strategy, Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state The probability of Indicates that based on the current strategy in state Take action The probability of Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state The probability of Indicates that based on the expert strategy in the state Take action probability; yes A simplified expression of In the parameters Next, according to the state action The resulting value.

[0233] The training module determines the penalty value corresponding to the trajectory of each sample vehicle, which may include:

[0234] Through the penalty function , determine the penalty value corresponding to the trajectory of each sample vehicle;

[0235] in, represents the minimum distance between any two sample vehicles, 1 indicates the collision penalty value, Indicates the shortest distance between the sample vehicle and the edge of the road, , Indicates the closest distance between the sample vehicle and the left edge of the road, Indicates the closest distance between the sample vehicle and the right edge of the road, Represents the distance penalty value, Indicates that the vehicle kinematic constraints are not met. represents the constraint penalty value, Indicates the emergency brake penalty value. Indicates acceleration.

[0236] The collision penalty value can be determined by the following method:

[0237] Extract the previous consecutive position points;

[0238] Regarding the aforementioned For each position point among the consecutive position points, if the sample vehicle did not collide with the surrounding vehicles before moving to the position point, but collided with the surrounding vehicles at the position point, the position point is marked as abnormal; if the sample vehicle did not collide with the surrounding vehicles before moving to the position point, and did not collide with the surrounding vehicles at the position point, the position point is marked as a candidate;

[0239] If the previous There is a position point marked as a candidate among each of the continuous position points, the position point arranged last among the position points marked as candidates is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain a collision penalty value;

[0240] If the previous The consecutive position points are marked as abnormal. The first position point among the consecutive position points is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain the collision penalty value.

[0241] In another embodiment of the present application, an electronic device is provided, including:

[0242] The memory is used to store computer programs;

[0243] The processor is used to execute the computer program so that the electronic device can implement the vehicle trajectory optimization method based on imitation learning as introduced in any one of embodiments 1-7.

[0244] In another embodiment of the present application, a computer storage medium is provided, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the vehicle trajectory optimization method based on imitation learning as described in any one of Examples 1-7.

[0245] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0246] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0247] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0248] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A vehicle trajectory optimization method based on imitation learning, characterized in that: include: Obtaining initial trajectory data of the target vehicle; Determining a target flow direction of the target vehicle; Based on the initial trajectory data, selecting a target lane from at least one lane corresponding to the target flow direction; If the initial trajectory data satisfies the correction condition, extracting the lane information of the target lane from the configuration information of the intersection, and obtaining the current position information and current motion information of the target vehicle and its surrounding vehicles based on the initial trajectory data; Inputting the current position information, current motion information of the target vehicle and its surrounding vehicles, and the lane information of the target lane into a generative adversarial imitation learning model to obtain target trajectory data determined by the generative adversarial imitation learning model; The generative adversarial imitation learning model is trained based on the actual vehicle driving trajectory at the intersection. The generative adversarial imitation learning model is used to imitate the expert strategy, generate action parameters similar to the actual vehicle behavior, and obtain target trajectory data; the generative adversarial imitation learning model is trained based on an adversarial network, and the adversarial network includes a generator and a judge; The process of training the generative adversarial imitation learning model based on the adversarial network includes: obtaining the vehicle driving trajectory at the intersection, and determining the state-action pair of the expert based on the vehicle driving trajectory at the intersection; at the current moment , according to the setting of course distribution set Sampling out vehicles as multiple sample vehicles; obtaining sample information of each of the multiple sample vehicles at the current position; the sample information includes: the position information and motion information of the sample vehicle at the current position and the position information and motion information of the surrounding sample vehicles and the lane information of the target sample lane; processing the sample information corresponding to each sample vehicle according to the current strategy of the generator to generate the trajectory of each sample vehicle; determining the penalty value corresponding to the trajectory of each sample vehicle; and determining the penalty value corresponding to each state-action pair in the trajectory of each sample vehicle based on the judgement. Scoring is performed to generate a reward value for each of the sample vehicles; the reward value is determined in the following manner: Among them, it means that the judge has parameters Next, according to the state action The value obtained, Represents the penalty value corresponding to the trajectory of the sample vehicle; keeps the parameters of the evaluator unchanged, and updates the strategy parameters of the generator based on the trust region optimization method; updates the strategy parameters of the generator based on the trust region optimization method, including: solving the following constrained optimization problem: ； in, Representation strategy Parameters; express expectations; Indicated in The current strategy adopted at any moment, which is based on the old parameters defined; Indicates new strategy; Indicates that the current strategy is Observe the conditions at all times Take action probability; Indicates that the new strategy Observe the conditions at all times Take action probability; Indicates the current strategy Under observation conditions The probability distribution of taking actions under Indicates new strategy Under observation conditions The probability distribution of taking actions under express and The KL (Kullback-Leibler) divergence between; represents the step size parameter, which is used to control the maximum change of the strategy in each optimization step; represents the advantage function, which is used to measure the observation condition Take action The expected action value With observer Estimated state value expectation degree of difference between represents the behavior taken by the sample vehicle according to the strategy; the advantage function is estimated by the following generalized advantage estimation method: in, represents the discount rate; It is a parameter between 0 and 1, used to balance the TD (Temporal Difference) error The weight of represents the reward value determined by the judge; and Respectively expressed in Moment and Keep the strategy parameters of the generator unchanged, and update the evaluation parameters of the judge based on the state-action pairs of the expert and the state-action pairs generated by the new strategy of the generator. ; The judging parameters of the judge The update is performed by the following objective function: in, represents the expert strategy, represents the new strategy, Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state probability; Indicates that based on the current strategy in state Take action probability; Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state probability; Indicates that based on the expert strategy in the state Take action probability; yes A simplified expression of In the parameters Next, according to the state action The value obtained; Based on the target trajectory data, the initial trajectory data is corrected.

2. The vehicle trajectory optimization method based on imitation learning according to claim 1 is characterized in that: Determining a target flow direction of the target vehicle includes: If the target vehicle has locked the flow direction, the locked flow direction of the target vehicle is used as the target flow direction of the target vehicle; If the target vehicle has an unlocked flow direction, and the flow direction of the entrance lane of the target vehicle is configured as a single flow direction, the single flow direction is used as the target flow direction of the target vehicle; If the target vehicle has an unlocked flow direction, and the flow direction of the entrance lane of the target vehicle is configured as multiple flow directions, then by comparing the vehicle flow corresponding to each of the multiple flow directions, the flow direction with the largest vehicle flow is selected from the multiple flow directions as the target flow direction of the target vehicle; If the traffic volume corresponding to each flow direction in the multiple flow directions is consistent, a temporary target point is determined based on the historical trajectory of the target vehicle. If the temporary target point is located in one of the multiple flow directions, the flow direction in the multiple flow directions containing the temporary target point is used as the target flow direction of the target vehicle.

3. The vehicle trajectory optimization method based on imitation learning according to claim 1 is characterized in that: Based on the initial trajectory data, selecting a target lane from at least one lane corresponding to the target flow direction includes: Determining the latest trajectory heading angle of the target vehicle based on the current trajectory point of the target vehicle in the initial trajectory data; Obtaining an exit track heading angle corresponding to a target flow direction of the target vehicle; Determine an average angle between the latest trajectory heading angle and the exit path heading angle; An angle between a target point of each lane in at least one lane corresponding to the target flow direction and a current trajectory point of the target vehicle is determined, and a lane having a minimum difference between the angle and the average angle is taken as a target lane.

4. The vehicle trajectory optimization method based on imitation learning according to claim 1 is characterized in that: The initial trajectory data meets the correction condition by judging in the following way: Determine a first distance between a first initial trajectory point in the initial trajectory data that enters the target lane and a target point in the target lane; Determine a second distance between the current trajectory point of the target vehicle and the first initial trajectory point in the initial trajectory data; If the target flow direction is a left turn flow direction, and if the ratio of the second distance to the first distance is not less than a left turn threshold, then the correction condition is satisfied; If the target flow direction is a right turn flow direction, and if the ratio of the second distance to the first distance is not less than a right turn threshold, then the correction condition is satisfied; If the target flow direction is a straight flow direction, if the ratio of the second distance to the first distance is not less than a straight threshold, and the target vehicle fails in visual tracking, then a correction condition is satisfied; If the target flow direction is a U-turn flow direction, and if the target vehicle fails in visual tracking, the correction condition is met.

5. The vehicle trajectory optimization method based on imitation learning according to claim 1 is characterized in that: Determining the penalty value corresponding to the trajectory of each of the sample vehicles includes: Through the penalty function , determine the penalty value corresponding to the trajectory of each sample vehicle; wherein, represents the minimum distance between any two sample vehicles, 1 indicates the collision penalty value, Indicates the shortest distance between the sample vehicle and the edge of the road, , Indicates the closest distance between the sample vehicle and the left edge of the road, Indicates the closest distance between the sample vehicle and the right edge of the road, Represents the distance penalty value, Indicates that the vehicle kinematic constraints are not met. represents the constraint penalty value, Indicates the emergency brake penalty value. Indicates acceleration.

6. The vehicle trajectory optimization method based on imitation learning according to claim 5 is characterized in that: The collision penalty value is determined by: Extracting the first n consecutive position points from the trajectory of the sample vehicle; For each position point among the first n consecutive position points, if the sample vehicle does not collide with surrounding vehicles before moving to the position point, but collides with surrounding vehicles at the position point, the position point is marked as abnormal; If the sample vehicle does not collide with surrounding vehicles before moving to the position point, and does not collide with surrounding vehicles at the position point, marking the position point as a candidate; If there is a position point marked as a candidate among each of the first n consecutive position points, the position point arranged last among the position points marked as candidates is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain a collision penalty value; If the first n consecutive position points are all marked as abnormal, the first position point among the first n consecutive position points is used as the new current position of the sample vehicle, and the penalty value corresponding to the position point marked as abnormal is determined, and the penalty values ​​corresponding to the position points marked as abnormal are accumulated to obtain the collision penalty value.

7. A vehicle trajectory optimization device based on imitation learning, characterized in that: include: A first acquisition module is used to obtain initial trajectory data of the target vehicle; A first determination module, used to determine the target flow direction of the target vehicle; a screening module, configured to screen a target lane from at least one lane corresponding to the target flow direction based on the initial trajectory data; A second acquisition module is used to extract the lane information of the target lane from the configuration information of the intersection if the initial trajectory data meets the correction condition, and obtain the current position information and current motion information of the target vehicle and its surrounding vehicles based on the initial trajectory data; A second determination module is used to input the current position information, current motion information of the target vehicle and its surrounding vehicles and the lane information of the target lane into the generative adversarial imitation learning model to obtain the target trajectory data determined by the generative adversarial imitation learning model; The generative adversarial imitation learning model is trained based on the actual vehicle driving trajectory at the intersection. The generative adversarial imitation learning model is used to imitate the expert strategy, generate action parameters similar to the actual vehicle behavior, and obtain target trajectory data; the generative adversarial imitation learning model is trained based on an adversarial network, and the adversarial network includes a generator and a judge; The process of training the generative adversarial imitation learning model based on the adversarial network includes: obtaining the vehicle driving trajectory at the intersection, and determining the state-action pair of the expert based on the vehicle driving trajectory at the intersection; at the current moment , according to the setting of course distribution set Sampling out vehicles as multiple sample vehicles; obtaining sample information of each of the multiple sample vehicles at the current position; the sample information includes: the position information and motion information of the sample vehicle at the current position and the position information and motion information of the surrounding sample vehicles and the lane information of the target sample lane; processing the sample information corresponding to each sample vehicle according to the current strategy of the generator to generate the trajectory of each sample vehicle; determining the penalty value corresponding to the trajectory of each sample vehicle; and determining the penalty value corresponding to each state-action pair in the trajectory of each sample vehicle based on the judgement. Scoring is performed to generate a reward value for each of the sample vehicles; the reward value is determined in the following manner: in, Representation Judger In the parameters Next, according to the state action The value obtained, Represents the penalty value corresponding to the trajectory of the sample vehicle; keeps the parameters of the evaluator unchanged, and updates the strategy parameters of the generator based on the trust region optimization method; updates the strategy parameters of the generator based on the trust region optimization method, including: solving the following constrained optimization problem: ； in, Representation strategy Parameters; express expectations; Indicated in The current strategy adopted at any moment, which is based on the old parameters defined; Indicates new strategy; Indicates that the current strategy is Observe the conditions at all times Take action probability; Indicates that the new strategy Observe the conditions at all times Take action probability; Indicates the current strategy Under observation conditions The probability distribution of taking actions under Indicates new strategy Under observation conditions The probability distribution of taking actions under express and The KL (Kullback-Leibler) divergence between; represents the step size parameter, which is used to control the maximum change of the strategy in each optimization step; represents the advantage function, which is used to measure the observation condition Take action The expected action value With observer Estimated state value expectation degree of difference between represents the behavior taken by the sample vehicle according to the strategy; the advantage function is estimated by the following generalized advantage estimation method: in, represents the discount rate; It is a parameter between 0 and 1, used to balance the TD (Temporal Difference) error The weight of represents the reward value determined by the judge; and Respectively expressed in Moment and Keep the strategy parameters of the generator unchanged, and update the evaluation parameters of the judge based on the state-action pairs of the expert and the state-action pairs generated by the new strategy of the generator. ; The judging parameters of the judge The update is performed by the following objective function: in, represents the expert strategy, represents the new strategy, Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state probability; Indicates that based on the current strategy in state Take action probability; Indicates that the policy is being executed Time state action pair The probability of being visited; Indicated in strategy Next time In state probability; Indicates that based on the expert strategy in the state Take action probability; yes A simplified expression of In the parameters Next, according to the state action The value obtained; A correction module is used to correct the initial trajectory data based on the target trajectory data.

8. An electronic device, characterized in that: include: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the vehicle trajectory optimization method based on imitation learning as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the vehicle trajectory optimization method based on imitation learning as described in any one of claims 1 to 6.

Images