Driving decision determination method and device

Abstract

The present application relates to a driving decision determination method and device, which is applied to the field of vehicle control technology. The method includes: real-time collection of sensor data, determination of driving state information of each vehicle, and extraction of road network information; determination of the scene type at the initial moment, and determination of the reward function corresponding to the scene type at the initial moment according to the mapping relationship between the scene type and the reward function; inputting the driving state information at the Nth moment after the initial moment into the reward function to obtain a reward value; using the road network information at the initial moment, the driving state information at the initial moment and the Nth moment after the initial moment as state input data, processing the state input data and the reward value using a multi-agent reinforcement learning algorithm to determine the driving decision; sending the driving decision to the corresponding vehicle respectively; using the N+1th moment after the initial moment as the initial moment, returning to extract the road network information, until stopping the collection of sensor data. Improve the efficiency and coordination of the overall traffic flow.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a driving decision determination method and device. Background Art

[0002] With the rapid development of the automotive industry, intelligent network technology and autonomous driving have become important directions for innovation in the transportation system. Intelligent networked vehicles currently use vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications to achieve information sharing and real-time decision-making, thereby improving the safety and efficiency of road traffic. In daily driving, vehicle driving decisions and traffic flow management are crucial.

[0003] Traditional driving decisions often rely on the vehicle's own sensor data and the driver's experience. The vehicle monitors the surrounding environment in real time through sensors, and makes acceleration, deceleration or lane changes based on preset rules and algorithms. However, this method is limited to single-vehicle intelligence and relies on the driver or the on-board system to make local decisions, making it difficult to fully grasp the dynamic changes in the overall traffic flow. In addition, since the driving strategies of different vehicles may not be coordinated with each other, it is easy to cause low traffic efficiency and safety hazards. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a driving decision determination method, device, electronic device and storage medium.

[0005] According to a first aspect of the present application, a driving decision determination method is provided, comprising:

[0006] Collect sensor data within the coverage area of ​​the road-end device in real time, and determine the driving status information of each vehicle within the coverage area of ​​the road-end device based on the sensor data;

[0007] Extracting road network information from the driving state information at the initial moment, and performing scene recognition based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment;

[0008] Determine the reward function corresponding to the scene type at the initial moment according to the mapping relationship between the scene type and the reward function in the pre-built scene library;

[0009] Input the driving state information at the Nth moment after the initial moment into the reward function to obtain a reward value; N is a positive integer;

[0010] The road network information at the initial moment and the driving state information at the initial moment and N moments after the initial moment are used as state input data, and the state input data and the reward value are processed by a multi-agent reinforcement learning algorithm to determine the driving decision corresponding to each vehicle;

[0011] Sending the driving decisions corresponding to the respective vehicles to the corresponding vehicles respectively, so that the respective vehicles drive according to the corresponding driving decisions;

[0012] The N+1th moment after the initial moment is taken as the initial moment, and the step of extracting the road network information from the driving status information at the initial moment is returned until the collection of the driving status information of each vehicle within the coverage area of ​​the road end device is stopped.

[0013] Optionally, the scene types in the scene library include: traffic light intersection scene, non-traffic light intersection scene and ramp merging scene;

[0014] The function value of the traffic light intersection reward function corresponding to the traffic light intersection scenario is determined by whether the vehicle passes through the intersection according to the indicator light and the speed of the vehicle when passing through the intersection;

[0015] The function value of the non-traffic light intersection reward function corresponding to the non-traffic light intersection scenario is determined according to the length of time the vehicle stays at the non-traffic light intersection;

[0016] The function value of the ramp merging reward function corresponding to the ramp merging scenario is determined according to whether the vehicle collides after merging, the distance between the vehicle and other vehicles, and the speed and acceleration at the time of merging.

[0017] Optionally, inputting the driving state information at the Nth moment after the initial moment into the reward function to obtain the reward value includes:

[0018] If the scene type at the initial moment is a ramp merging scene, determining the distance between the vehicle and other vehicles after merging, and the speed and acceleration of the vehicle when merging, based on the driving state information at the Nth moment after the initial moment;

[0019] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0020] R=ω c r c +ω s r s +ω a r a +ω h r h ;

[0021] Among them, ω c ,ω s ,ω a ,ω h is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r his the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a The acceleration bonus value is determined based on the acceleration of the vehicle when merging.

[0022] Optionally, inputting the driving state information at the Nth moment after the initial moment into the reward function to obtain the reward value includes:

[0023] If the scene type at the initial moment is a ramp merging scene, determine the distance between the vehicle and other vehicles after merging, the speed and acceleration of the vehicle when merging, and the merging cost of the vehicle when merging according to the driving state information at the Nth moment after the initial moment;

[0024] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0025] R=ω c r c +ω s r s +ω a r a +ω h r h +ω m r m ;

[0026] Among them, ω c ,ω s ,ω a ,ω h ,ω m is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a is the acceleration reward value determined based on the acceleration of the vehicle when merging, r m is the merging cost reward value determined based on the merging cost of the vehicle when merging, r m Less than 0.

[0027] Optionally, the method for determining the combined cost reward value includes:

[0028] According to the following formula:

[0029] Determine the combined cost reward value r m ;

[0030] Wherein, A7 is a preset constant greater than 0, x is the distance traveled by the vehicle in the merging area, and L is the length of the merging area.

[0031] Optionally, the method for determining the speed bonus value includes:

[0032] According to the following formula:

[0033]

[0034] Determine the speed bonus value r s ;

[0035] Among them, A2 and A3 are preset constants greater than 0, v t is the speed of the vehicle at time t, t is the Nth time after the initial time, v max and v min The maximum and minimum speeds are preset.

[0036] Optionally, the method for determining the safety bonus value includes:

[0037] According to the following formula:

[0038]

[0039] Determine the safety reward value r c ;

[0040] Wherein, A1 is a preset constant greater than 0.

[0041] Optionally, the method for determining the distance reward value comprises:

[0042] According to the following formula:

[0043]

[0044] Determine the distance reward value r h ;

[0045] Among them, A6 is a preset constant greater than 0, d headway represents the distance between vehicles, t h Indicates the preset time threshold.

[0046] Optionally, the method for determining the distance reward value includes:

[0047] Determining the traffic density according to the road network information at the Nth moment after the initial moment;

[0048] Determine the safe distance based on traffic density;

[0049] If the distance between the vehicle and other vehicles after the merging is greater than the safety distance, setting the distance reward function value to a first positive reward value;

[0050] If the distance between the vehicle and other vehicles after the merging is less than or equal to the safety distance, the distance reward function value is set to the first negative reward value.

[0051] According to a second aspect of the present application, a driving decision determination device is provided, comprising:

[0052] A driving status information determination module, used to collect sensor data within the coverage area of ​​the road terminal equipment in real time, and determine the driving status information of each vehicle within the coverage area of ​​the road terminal equipment according to the sensor data;

[0053] A scene type determination module is used to extract road network information from the driving state information at the initial moment, and perform scene recognition based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment;

[0054] A reward function determination module, used to determine the reward function corresponding to the scene type at the initial moment according to the mapping relationship between the scene type and the reward function in the pre-built scene library;

[0055] A reward value determination module, used to input the driving state information at the Nth moment after the initial moment into the reward function to obtain a reward value; N is a positive integer;

[0056] A multi-agent reinforcement learning module is used to use the road network information at the initial moment and the driving state information at the initial moment and N moments after the initial moment as state input data, and use the multi-agent reinforcement learning algorithm to process the state input data and the reward value to determine the driving decision corresponding to each vehicle;

[0057] A driving decision sending module, used for sending the driving decisions corresponding to the respective vehicles to the corresponding vehicles, so that the respective vehicles drive according to the corresponding driving decisions;

[0058] The initial moment update module is used to determine that when the driving status information determination module has not stopped collecting sensor data within the coverage range of the road-side device, take the N+1th moment after the initial moment as the initial moment and return to the scene type determination module; when it is determined that the driving status information determination module has stopped collecting sensor data within the coverage range of the road-side device, end the process.

[0059] Optionally, the scene types in the scene library include: traffic light intersection scene, non-traffic light intersection scene and ramp merging scene;

[0060] The function value of the traffic light intersection reward function corresponding to the traffic light intersection scenario is determined by whether the vehicle passes through the intersection according to the indicator light and the speed of the vehicle when passing through the intersection;

[0061] The function value of the non-traffic light intersection reward function corresponding to the non-traffic light intersection scenario is determined according to the length of time the vehicle stays at the non-traffic light intersection;

[0062] The function value of the ramp merging reward function corresponding to the ramp merging scenario is determined according to whether the vehicle collides after merging, the distance between the vehicle and other vehicles, and the speed and acceleration at the time of merging.

[0063] Optionally, the reward value determination module is specifically configured to determine, if the scene type at the initial moment is a ramp merging scene, the distance between the vehicle after merging and other vehicles, and the speed and acceleration of the vehicle when merging according to the driving state information at the Nth moment after the initial moment;

[0064] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0065] R=ω c r c +ω s r s +ω a r a +ω h r h ;

[0066] Among them, ω c ,ω s ,ω a ,ω h is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a The acceleration bonus value is determined based on the acceleration of the vehicle when merging.

[0067] Optionally, the reward value determination module is specifically configured to determine, if the scene type at the initial moment is a ramp merging scene, the distance between the vehicle after merging and other vehicles, the speed and acceleration of the vehicle when merging, and the merging cost of the vehicle when merging according to the driving state information at the Nth moment after the initial moment;

[0068] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0069] R=ωc r c +ω s r s +ω a r a +ω h r h +ω m r m ;

[0070] Among them, ω c ,ω s ,ω a ,ω h ,ω m is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a is the acceleration reward value determined based on the acceleration of the vehicle when merging, r m is the merging cost reward value determined based on the merging cost of the vehicle when merging, r m Less than 0.

[0071] Optionally, the reward value determination module is specifically configured to determine the combined cost reward value in the following manner:

[0072] According to the following formula:

[0073] Determine the combined cost reward value r m ;

[0074] Wherein, A7 is a preset constant greater than 0, x is the distance traveled by the vehicle in the merging area, and L is the length of the merging area.

[0075] Optionally, the method for determining the speed bonus value comprises:

[0076] According to the following formula:

[0077]

[0078] Determine the speed bonus value r s ;

[0079] Among them, A2 and A3 are preset constants greater than 0, v t is the speed of the vehicle at time t, t is the Nth time after the initial time, v max and v min The maximum and minimum speeds are preset.

[0080] Optionally, the reward value determination module is specifically configured to determine the safety reward value in the following manner:

[0081] According to the following formula:

[0082]

[0083] Determine the safety reward value r c ;

[0084] Wherein, A1 is a preset constant greater than 0.

[0085] Optionally, the reward value determination module is specifically configured to determine the distance reward value in the following manner:

[0086] According to the following formula:

[0087]

[0088] Determine the distance reward value r h ;

[0089] Among them, A6 is a preset constant greater than 0, d headway represents the distance between vehicles, t h Indicates the preset time threshold.

[0090] Optionally, the reward value determination module is specifically configured to determine the distance reward value in the following manner:

[0091] Determining the traffic density according to the road network information at the Nth moment after the initial moment;

[0092] Determine the safe distance based on traffic density;

[0093] If the distance between the vehicle and other vehicles after the merging is greater than the safety distance, setting the distance reward function value to a first positive reward value;

[0094] If the distance between the vehicle and other vehicles after the merging is less than or equal to the safety distance, the distance reward function value is set to the first negative reward value.

[0095] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, wherein the processor is configured to execute a computer program stored in a memory, wherein the computer program implements the method described in the first aspect when executed by the processor.

[0096] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0097] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method described in the first aspect.

[0098] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0099] The road-side equipment collects sensor data within its coverage area in real time, and determines the driving status information of each vehicle based on the sensor data. The road network information is extracted from the driving status information at the initial moment, and the scene recognition is performed based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment. By integrating multi-source data through a multi-agent reinforcement learning algorithm, global traffic perception can be provided, making up for the limitations of single-vehicle intelligence, so that vehicles can make more reasonable decisions based on the overall traffic status, achieve global optimal effects, achieve effective coordination between vehicles, and improve the efficiency and coordination of the overall traffic flow. The pre-built scenario library can provide the algorithm with historical information and experience accumulation, helping the algorithm to use different reward functions to calculate the reward value when facing different scenarios, which is more flexible and widely applicable. For example, the reward function corresponding to the scene type at the initial moment can be selected, thereby improving the accuracy and effectiveness of driving decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0101] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0102] Figure 1 A flow chart of a driving decision determination method in an embodiment of the present application;

[0103] Figure 2 A schematic diagram of data processing by a road-side device in an embodiment of the present application;

[0104] Figure 3 A schematic diagram of scene recognition in an embodiment of the present application;

[0105] Figure 4 A schematic diagram of the structure of a driving decision determination device in an embodiment of the present application;

[0106] Figure 5A schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0107] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0108] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0109] Most existing driving decision-making methods are based on local information perceived by the vehicle's own sensors, which makes it difficult to achieve global optimization. In addition, they are usually based on fixed scene rules and lack the ability to adapt to diverse scenes. This results in the system being unable to provide effective and flexible decision-making suggestions when facing different traffic environments.

[0110] In the embodiment of the present application, by deploying intelligent networked road-side equipment (RSU), using its various sensors (such as cameras, radars, etc.) and communication modules, the driving status information of vehicles within the coverage range of the road-side equipment is perceived in real time. At the same time, by combining scene recognition technology with multi-agent reinforcement learning algorithm, the road-side equipment can analyze the interaction behavior of multiple vehicles and generate driving decisions corresponding to each vehicle. Through V2X communication technology, the driving decision is transmitted to the on-board unit (OBU) to provide the vehicle with optimized driving decisions within the range of the road-side equipment. By making full use of the perception, computing and communication capabilities of the road-side equipment and integrating multi-source data, the overall traffic flow can be monitored and analyzed in real time, forward-looking decision-making suggestions can be provided, the global traffic flow can be optimized, and more comprehensive and real-time environmental information sharing can be provided. In addition, by combining scene recognition technology with multi-vehicle reinforcement learning methods, by real-time identification and analysis of traffic scenes, driving decision suggestions for specific environments can be provided to vehicles, thereby improving the effectiveness and safety of driving decisions.

[0111] See also Figure 1 , Figure 1 A flowchart of a driving decision determination method in an embodiment of the present application may include the following steps:

[0112] Step S102, collecting sensor data within the coverage area of ​​the road-side device in real time, and determining the driving status information of each vehicle within the coverage area of ​​the road-side device according to the sensor data.

[0113] Roadside equipment includes cameras, radars, computing units, and communication modules (such as V2X communication). Cameras are used to capture real-time images and video streams of vehicles, and provide lane information and vehicle location data. Radars are used to detect vehicle speed, acceleration, relative distance, and other data, and provide accurate physical status information. Communication modules are used to achieve two-way communication between roadside equipment and vehicles, collect sensor data from the vehicle, and transmit driving decisions generated by the roadside equipment to the vehicle in real time. The process of roadside equipment processing data is as follows: Figure 2 shown.

[0114] The sensors deployed in the roadside equipment can collect sensor data within the coverage area of ​​the roadside equipment in real time and determine the driving status information of the vehicle based on the sensor data. The driving status information of the vehicle includes: the precise position, speed, acceleration and lane information of each vehicle. The speed and acceleration are used to determine the movement status of the vehicle. The lane information is used to identify the lane and lane change situation of each vehicle.

[0115] Step S104, extracting road network information from the driving status information at the initial moment, and performing scene recognition based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment.

[0116] In the embodiment of the present application, a static scene represented by road network information can be constructed based on the data collected by the road-side equipment, and the scene recognition algorithm can be used to provide input for the selection of the reward function of the subsequent multi-agent reinforcement learning algorithm. The initial moment refers to the moment when the driving decision begins to be provided, which can be the moment when the sensor data is initially collected, or it can be any moment in the process of collecting sensor data. The road-side equipment will collect the vehicle's driving status information (such as position, speed, acceleration, lane information, etc.) and organize it to extract the road network information.

[0117] Road network information includes: road network structure, traffic sign information, vehicle location data and traffic flow information. Among them, the road network structure includes the length, width and connection relationship of each road section. Traffic sign information includes traffic signs, signal lights and their status in the road network, providing complete rule information for the scene. Vehicle location data integrates the real-time location of all vehicles and establishes a distribution map of vehicles in the road network. Traffic flow information can analyze the flow density of vehicles and identify potential congestion areas.

[0118] Using sensor data and road network information as input data, the scene type of a static scene can be identified through a scene recognition algorithm. Sensor data includes image data and radar data. Radar data contains the distance, speed, and relative motion state of the vehicle, usually represented in the form of a point cloud.

[0119] Scene recognition algorithms such as Figure 3As shown, the image can be pre-processed before being input into the convolutional neural network (CNN) model. The data pre-processing includes: denoising, color normalization, and size adjustment. The radar data can be processed by converting the point cloud data into a rasterized two-dimensional map to represent the spatial distribution of the vehicle.

[0120] The image is processed through multiple convolutional layers to extract low-level features (such as edges and textures), and then high-level features (such as object shape and structure). The raster map of radar data is also processed through convolutional layers to extract spatial features. The feature vectors extracted from the image and radar data are connected to form a high-dimensional feature representation. The fused features are further processed through the fully connected layer to extract more complex scene information.

[0121] The output layer can use the Softmax activation function to classify features and output the probability distribution of each scene type. The probability value of each scene type represents the confidence of the scene being recognized. For example, the scene types include: traffic light intersection, non-traffic light intersection and ramp merging. The traffic light intersection scene refers to the intersection scene with traffic lights, and the non-traffic light intersection scene refers to the intersection scene without traffic lights. The output scene type and its corresponding probability value can be 0.75 for "ramp merging", 0.15 for "non-traffic light intersection" and 0.10 for "traffic light intersection". The scene type with the highest probability is selected as the final recognition result.

[0122] After being processed by the convolutional neural network, images, radar data and road network information can effectively output the recognition results and probabilities of traffic scenes, providing real-time decision support for intelligent connected vehicles.

[0123] Step S106, determining the reward function corresponding to the scene type at the initial moment according to the mapping relationship between the scene type and the reward function in the pre-built scene library.

[0124] In order to cope with diverse traffic scenarios, this application builds a dynamic scenario library. The scenario library contains different types of traffic scenarios, such as traffic light intersections, non-traffic light intersections, and ramp merging scenarios, and designs corresponding reward functions for each scenario. As the road-side equipment's real-time perception of the traffic environment and scene recognition capabilities are enhanced, the scenario library can also be continuously supplemented and optimized to cope with the different scenarios identified in step S104.

[0125] Construction of scenario library: Whenever the road-side equipment recognizes a new traffic scenario, the characteristics, key behaviors and reward functions of the scenario can be recorded in the scenario library. Each scenario has a corresponding reward function and adjustment parameters. The reward function can be set according to the key behaviors of the scenario, so as to facilitate flexible call in the multi-agent reinforcement learning process.

[0126] As road-side equipment continuously perceives and analyzes the traffic environment and professionals define scenario reward functions, the scenario library can continuously add new scenarios and corresponding reward functions, thereby achieving a more precise definition of traffic scenarios.

[0127] As more data is collected, the content of the scenario library can be dynamically optimized by analyzing historical data, driving behavior, and changes in traffic flow. For example, if it is found that the reward function of a certain scenario is no longer suitable for the current traffic conditions, the reward function of the scenario can be adjusted based on the latest traffic data to better reflect the actual driving environment.

[0128] Optionally, the scene types in the scene library include: traffic light intersection scene, non-traffic light intersection scene and ramp merging scene.

[0129] The function value of the traffic light intersection reward function corresponding to the traffic light intersection scenario can be determined according to whether the vehicle passes through the intersection according to the indicator light and the speed of the vehicle when passing through the intersection.

[0130] The function value of the non-traffic light intersection reward function corresponding to the non-traffic light intersection scenario can be determined according to the length of time the vehicle stays at the non-traffic light intersection.

[0131] The function value of the ramp merging reward function corresponding to the ramp merging scenario can be determined according to whether the vehicle collides after merging, the distance between the vehicle and other vehicles, and the speed and acceleration when merging.

[0132] Step S108, input the driving state information at the Nth moment after the initial moment into the reward function to obtain a reward value; N is a positive integer.

[0133] As mentioned above, different scenarios correspond to different reward functions. For the ramp merging scenario, during the vehicle driving process, more attention is paid to whether the vehicle collides after merging, the distance between the vehicle and other vehicles, the speed and acceleration when merging, etc. Therefore, the input of the ramp merging reward function may include: whether the vehicle collides after merging, the distance between the vehicle and other vehicles, the speed and acceleration when merging, etc.

[0134] Optionally, if the scene type at the initial moment is a ramp merging scene, the distance between the vehicle and other vehicles after merging, and the speed and acceleration of the vehicle when merging are determined based on the driving status information at the Nth moment after the initial moment.

[0135] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0136] R=ω c r c +ω s r s +ωa r a +ω h r h ;

[0137] Among them, ω c ,ω s ,ω a ,ω h is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a The acceleration bonus value is determined based on the acceleration of the vehicle when merging.

[0138] In the embodiment of the present application, multiple influencing factors such as collision, speed, acceleration and distance are fully considered, making the decision-making process more comprehensive, thereby determining more accurate driving decisions. In addition, the weight coefficient can be adjusted according to the needs or risk level of ramp merging scenarios in different regions to flexibly adjust the influence of each evaluation. A safety reward value less than or equal to 0 can incentivize vehicles to avoid collisions, thereby effectively reducing the risk of accidents.

[0139] Optionally, if the scene type at the initial moment is a ramp merging scene, the distance between the vehicle and other vehicles after merging, the speed and acceleration of the vehicle at the merging, and the merging cost of the vehicle at the merging are determined based on the driving status information at the Nth moment after the initial moment.

[0140] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0141] R=ω c r c +ω s r s +ω a r a +ω h r h +ω m r m ;

[0142] Among them, ω c ,ω s ,ω a ,ω h ,ω m is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r his the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a is the acceleration reward value determined based on the acceleration of the vehicle when merging, r m is the merging cost reward value determined based on the merging cost of the vehicle when merging, r m Less than 0.

[0143] For ramp merging scenarios, in addition to considering multiple influencing factors such as collision, speed, acceleration and distance, the merging cost can also be considered. The safety reward value is less than or equal to 0, and the merging cost reward value is less than 0, which encourages vehicles to avoid collisions and stay in the merging area for too long, thereby effectively reducing the risk of accidents and improving merging efficiency.

[0144] Alternatively, the following formula can be used:

[0145]

[0146] Determine the safety reward value r c , A1 is a preset constant greater than 0.

[0147] Alternatively, the following formula can be used:

[0148] Determine the combined cost reward value r m ;

[0149] Wherein, A7 is a preset constant greater than 0, x is the distance traveled by the vehicle in the merging area, and L is the length of the merging area.

[0150] It can be seen that the longer the distance the vehicle travels in the merging area, the smaller the merging cost reward value. The shorter the distance the vehicle travels in the merging area, the larger the merging cost reward value, and the merging cost reward value is always less than 0.

[0151] Optionally, according to the following formula:

[0152]

[0153] Determine the speed bonus value r s ;

[0154] Among them, A2 and A3 are preset constants greater than 0, v t is the speed of the vehicle at time t, t is the Nth time after the initial time, v max and v min The maximum and minimum speeds are preset.

[0155] The above formula limits the speed to a reasonable range, and punishes speeding, which encourages vehicles to adjust their speed according to traffic flow and road conditions, thereby improving the safety and smoothness of merging.

[0156] Optionally, when the acceleration is less than or equal to the preset acceleration, the acceleration reward value may be a reward value greater than 0, and the smaller the acceleration, the greater the reward value. When the acceleration is greater than the preset acceleration, the acceleration reward value may be a reward value less than 0, and the greater the acceleration, the smaller the reward value, so as to improve the safety and smoothness of the merging.

[0157] Optionally, according to the following formula:

[0158]

[0159] Determine the distance reward value r h ;

[0160] Among them, A6 is a preset constant greater than 0, d headway represents the distance between vehicles, t h Indicates the preset time threshold.

[0161] The distance reward value can be determined based on the real-time vehicle distance and vehicle speed, encouraging vehicles to maintain an appropriate distance, thereby reducing the risk of rear-end collisions and improving overall traffic safety.

[0162] Optionally, the distance reward value may also be determined according to the following method:

[0163] The traffic density is determined based on the road network information at the Nth moment after the initial moment; the safe distance is determined based on the traffic density. The greater the traffic density, the smaller the safe distance. If the distance between the vehicle and other vehicles after merging is greater than the safe distance, the distance reward function value is set to the first positive reward value; if the distance between the vehicle and other vehicles after merging is less than or equal to the safe distance, the distance reward function value is set to the first negative reward value. In this way, the difficulty of merging at different traffic densities is fully considered, and vehicles are encouraged to maintain an appropriate distance when merging, thereby improving traffic safety.

[0164] If the scene type at the initial moment is a traffic light intersection scene, determine whether the vehicle passes through the intersection according to the indicator light based on the traffic status information at the Nth moment after the initial moment;

[0165] If the vehicle passes through the intersection according to the indicator light and the speed of the vehicle when passing through the intersection is less than or equal to the preset speed, the traffic light intersection reward value is set to a second positive reward value;

[0166] If the vehicle does not pass through the intersection according to the indicator light or the speed of the vehicle when passing through the intersection is greater than a preset speed, the traffic light intersection reward value is set to a second negative reward value.

[0167] If the scene type at the initial moment is a non-traffic light intersection scene, determine the length of time the vehicle stays at the non-traffic light intersection based on the traffic status information at the Nth moment after the initial moment;

[0168] If the vehicle stays at a non-traffic light intersection for a period of time that is less than or equal to a preset period of time, the traffic light intersection reward value is set to a third positive reward value;

[0169] If the vehicle stays at the non-traffic light intersection for longer than a preset time, the traffic light intersection reward value is set to a third negative reward value.

[0170] It should be noted that the driving status information at each moment may contain the driving status information of multiple vehicles, so the reward value corresponding to the behavior of each vehicle can be calculated, and the weight coefficient corresponding to the vehicle can be adjusted according to the importance of the vehicle in the scene. The reward values ​​of multiple vehicles are weighted and summed to obtain the final reward value.

[0171] Step S110, taking the road network information at the initial moment, and the driving state information at the initial moment and N moments after the initial moment as state input data, and using a multi-agent reinforcement learning algorithm to process the state input data and reward values, and determine the driving decisions corresponding to each vehicle.

[0172] The road network information at the initial moment is static scene data, including road network structure, traffic signs, signal light status, etc. The driving status information at the initial moment and N moments after the initial moment, that is, the driving status information at N+1 moments, constitutes dynamic scene data. Static scene data and dynamic scene data constitute the state input data of the multi-agent reinforcement learning algorithm. Combined with the reward value calculated in step S108, the driving decision corresponding to each vehicle can be obtained. For example, the output driving decisions include acceleration, deceleration, lane change, maintaining speed, etc. This process simultaneously considers the collaborative behavior of multiple vehicles to optimize the overall traffic flow.

[0173] Step S112, sending the driving decisions corresponding to each vehicle to the corresponding vehicle respectively, so that each vehicle drives according to the corresponding driving decision.

[0174] Roadside equipment can transmit the generated driving decisions to vehicles within the coverage area in real time through V2X communication technology, and the vehicle's on-board unit receives the driving decisions. For autonomous vehicles, the received driving decisions can be directly executed by the vehicle control system (such as acceleration, deceleration or lane change), while human-driven vehicles can provide driving decisions to the driver through the on-board unit, and the driver then makes corresponding operations based on the driving decisions.

[0175] Step S114, determining whether to stop collecting the driving status information of each vehicle within the coverage area of ​​the road end device.

[0176] If the road terminal device has not stopped collecting the driving status information of each vehicle within the coverage area of ​​the road terminal device, step S116 is executed; if the driving status information of each vehicle within the coverage area of ​​the road terminal device is stopped, the process ends.

[0177] Step S116, taking the N+1th moment after the initial moment as the initial moment, and returning to step S104.

[0178] The driving decision determination method of the embodiment of the present application collects sensor data within its coverage range in real time through the road-side equipment, and determines the driving status information of each vehicle based on the sensor data. The road network information is extracted from the driving status information at the initial moment, and the scene recognition is performed based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment. By integrating multi-source data through a multi-agent reinforcement learning algorithm, global traffic perception can be provided to make up for the limitations of single-vehicle intelligence, so that vehicles can make more reasonable decisions based on the overall traffic status, achieve global optimal effects, achieve effective coordination between vehicles, and improve the efficiency and synergy of the overall traffic flow. The pre-built scenario library can provide historical information and experience accumulation for the algorithm, helping the algorithm to use different reward functions to calculate the reward value when facing different scenarios, which is more flexible and widely applicable. For example, the reward function corresponding to the scene type at the initial moment can be selected, so as to improve the accuracy and effectiveness of driving decisions.

[0179] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0180] Corresponding to the above method embodiment, the present application embodiment also provides a driving decision determination device, see Figure 4 , the driving decision determination device 400 includes:

[0181] The driving state information determination module 402 is used to collect sensor data within the coverage area of ​​the road-side device in real time, and determine the driving state information of each vehicle within the coverage area of ​​the road-side device according to the sensor data;

[0182] A scene type determination module 404 is used to extract road network information from the driving state information at the initial moment, and perform scene recognition based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment;

[0183] A reward function determination module 406 is used to determine the reward function corresponding to the scene type at the initial moment according to the mapping relationship between the scene type and the reward function in the pre-built scene library;

[0184] The reward value determination module 408 is used to input the driving state information at the Nth moment after the initial moment into the reward function to obtain a reward value; N is a positive integer;

[0185] The multi-agent reinforcement learning module 410 is used to use the road network information at the initial time and the driving state information at the initial time and N times after the initial time as state input data, and use the multi-agent reinforcement learning algorithm to process the state input data and the reward value to determine the driving decision corresponding to each vehicle;

[0186] A driving decision sending module 412, used to send the driving decisions corresponding to each vehicle to the corresponding vehicle, so that each vehicle drives according to the corresponding driving decision;

[0187] The initial moment update module 414 is used to determine that when the driving status information determination module 402 has not stopped collecting sensor data within the coverage range of the road-side device, take the N+1th moment after the initial moment as the initial moment and return to the scene type determination module 404; when it is determined that the driving status information determination module 402 has stopped collecting sensor data within the coverage range of the road-side device, the process ends.

[0188] Optionally, the scene types in the scene library include: traffic light intersection scene, non-traffic light intersection scene and ramp merging scene;

[0189] The function value of the traffic light intersection reward function corresponding to the traffic light intersection scenario is determined by whether the vehicle passes through the intersection according to the indicator light and the speed of the vehicle when passing through the intersection;

[0190] The function value of the non-traffic light intersection reward function corresponding to the non-traffic light intersection scenario is determined according to the length of time the vehicle stays at the non-traffic light intersection;

[0191] The function value of the ramp merging reward function corresponding to the ramp merging scenario is determined according to whether the vehicle collides after merging, the distance between the vehicle and other vehicles, and the speed and acceleration at the time of merging.

[0192] Optionally, the reward value determination module 408 is specifically configured to determine, if the scene type at the initial moment is a ramp merging scene, the distance between the vehicle after merging and other vehicles, and the speed and acceleration of the vehicle when merging according to the driving state information at the Nth moment after the initial moment;

[0193] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0194] R=ωc r c +ω s r s +ω a r a +ω h r h ;

[0195] Among them, ω c ,ω s ,ω a ,ω h is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a The acceleration bonus value is determined based on the acceleration of the vehicle when merging.

[0196] Optionally, the reward value determination module 408 is specifically configured to determine, if the scene type at the initial moment is a ramp merging scene, the distance between the vehicle after merging and other vehicles, the speed and acceleration of the vehicle when merging, and the merging cost of the vehicle when merging according to the driving state information at the Nth moment after the initial moment;

[0197] The ramp merging reward value R is determined according to the following ramp merging reward function:

[0198] R=ω c r c +ω s r s +ω a r a +ω h r h +ω m r m ;

[0199] Among them, ω c ,ω s ,ω a ,ω h ,ω m is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r ais the acceleration reward value determined based on the acceleration of the vehicle when merging, r m is the merging cost reward value determined based on the merging cost of the vehicle when merging, r m Less than 0.

[0200] Optionally, the reward value determination module 408 is specifically configured to determine the combined cost reward value in the following manner:

[0201] According to the following formula:

[0202] Determine the combined cost reward value r m ;

[0203] Wherein, A7 is a preset constant greater than 0, x is the distance traveled by the vehicle in the merging area, and L is the length of the merging area.

[0204] Optionally, the reward value determination module 408 is specifically configured to determine the safety reward value in the following manner:

[0205] According to the following formula:

[0206]

[0207] Determine the safety reward value r c ;

[0208] Wherein, A1 is a preset constant greater than 0.

[0209] Optionally, the reward value determination module 408 is specifically configured to determine the distance reward value in the following manner:

[0210] According to the following formula:

[0211]

[0212] Determine the distance reward value r h ;

[0213] Among them, A6 is a preset constant greater than 0, d headway represents the distance between vehicles, t h Indicates the preset time threshold.

[0214] Optionally, the reward value determination module 408 is specifically configured to determine the distance reward value in the following manner:

[0215] Determine the traffic density based on the road network information at the Nth moment after the initial moment;

[0216] Determine the safe distance based on traffic density;

[0217] If the distance between the vehicle and other vehicles after merging is greater than the safe distance, the distance reward function value is set to the first positive reward value;

[0218] If the distance between the vehicle and other vehicles after merging is less than or equal to the safe distance, the distance reward function value is set to the first negative reward value.

[0219] The specific details of each module or unit in the above device have been described in detail in the corresponding method, so they will not be repeated here.

[0220] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0221] The present application also provides an electronic device, see Figure 5 , Figure 5 The electronic device of the embodiment of the present application includes: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504;

[0222] Memory 503, used for storing computer programs;

[0223] The processor 501 is used to implement the above-mentioned driving decision determination method when executing the program stored in the memory 503.

[0224] It should be noted that the communication bus 504 mentioned in the above electronic device can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 504 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0225] The communication interface 502 is used for communication between the electronic device and other devices.

[0226] The memory 503 may include a RAM (Random Access Memory), or may include a non-volatile memory, such as at least one disk memory. Optionally, the memory 503 may also be at least one storage device located away from the aforementioned processor.

[0227] The above-mentioned processor 501 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0228] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned driving decision determination method is implemented.

[0229] It should be noted that the computer-readable storage medium shown in the present application may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.

[0230] In an embodiment of the present application, a computer program product is also provided. When the computer program product is run on a computer, the computer executes the above-mentioned driving decision determination method.

[0231] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0232] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A driving decision determination method, characterized in that: include: Collect sensor data within the coverage area of ​​the road-end device in real time, and determine the driving status information of each vehicle within the coverage area of ​​the road-end device based on the sensor data; Extracting road network information from the driving state information at the initial moment, and performing scene recognition based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment; Determine the reward function corresponding to the scene type at the initial moment according to the mapping relationship between the scene type and the reward function in the pre-built scene library; Input the driving state information at the Nth moment after the initial moment into the reward function to obtain a reward value; N is a positive integer; The road network information at the initial moment and the driving state information at the initial moment and N moments after the initial moment are used as state input data, and the state input data and the reward value are processed by a multi-agent reinforcement learning algorithm to determine the driving decision corresponding to each vehicle; Sending the driving decisions corresponding to the respective vehicles to the corresponding vehicles respectively, so that the respective vehicles drive according to the corresponding driving decisions; The N+1th moment after the initial moment is taken as the initial moment, and the step of extracting the road network information from the driving status information at the initial moment is returned until the collection of sensor data within the coverage range of the road-end equipment is stopped.

2. The method according to claim 1, characterized in that The scene types in the scene library include: traffic light intersection scene, non-traffic light intersection scene and ramp merging scene; The function value of the traffic light intersection reward function corresponding to the traffic light intersection scenario is determined by whether the vehicle passes through the intersection according to the indicator light and the speed of the vehicle when passing through the intersection; The function value of the non-traffic light intersection reward function corresponding to the non-traffic light intersection scenario is determined according to the length of time the vehicle stays at the non-traffic light intersection; The function value of the ramp merging reward function corresponding to the ramp merging scenario is determined according to whether the vehicle collides after merging, the distance between the vehicle and other vehicles, and the speed and acceleration at the time of merging.

3. The method according to claim 2, characterized in that The step of inputting the driving state information at the Nth moment after the initial moment into the reward function to obtain the reward value comprises: If the scene type at the initial moment is a ramp merging scene, determining the distance between the vehicle and other vehicles after merging, and the speed and acceleration of the vehicle when merging, based on the driving state information at the Nth moment after the initial moment; The ramp merging reward value R is determined according to the following ramp merging reward function: R=ω c r c +oh s r s +oh a r a +oh h r h ; Among them, ω c ,ω s ,ω a ,ω h is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a The acceleration bonus value is determined based on the acceleration of the vehicle when merging.

4. The method according to claim 2, characterized in that: The step of inputting the driving state information at the Nth moment after the initial moment into the reward function to obtain the reward value comprises: If the scene type at the initial moment is a ramp merging scene, determine the distance between the vehicle and other vehicles after merging, the speed and acceleration of the vehicle when merging, and the merging cost of the vehicle when merging according to the driving state information at the Nth moment after the initial moment; The ramp merging reward value R is determined according to the following ramp merging reward function: R=ω c r c +oh s r s +oh a r a +oh h r h +oh m r m ; Among them, ω c ,ω s ,ω a ,ω h ,ω m is the weight coefficient, r c is the safety reward value determined based on whether a collision occurs after the merging, r c Less than or equal to 0, r h is the distance reward value determined based on the distance between the vehicle and other vehicles after merging, r s is the speed bonus value determined based on the vehicle’s speed at the time of merging, r a is the acceleration reward value determined based on the acceleration of the vehicle when merging, r m is the merging cost reward value determined based on the merging cost of the vehicle when merging, r m Less than 0.

5. The method according to claim 4, characterized in that Methods for determining the value of the combined cost award include: According to the following formula: Determine the combined cost reward value r m ; Wherein, A7 is a preset constant greater than 0, x is the distance traveled by the vehicle in the merging area, and L is the length of the merging area.

6. The method according to claim 3 or 4, characterized in that: Methods for determining speed bonus values ​​include: According to the following formula: Determine the speed bonus value r s ; Among them, A2 and A3 are preset constants greater than 0, v t is the speed of the vehicle at time t, t is the Nth time after the initial time, v max and v min The maximum and minimum speeds are preset.

7. The method according to claim 3 or 4, characterized in that: Methods for determining safety bonus values ​​include: According to the following formula: Determine the safety reward value r c ; Wherein, A1 is a preset constant greater than 0.

8. The method according to claim 3 or 4, characterized in that: Methods for determining distance bonus values ​​include: According to the following formula: Determine the distance reward value r h ; Among them, A6 is a preset constant greater than 0, d headway represents the distance between vehicles, t h Indicates the preset time threshold.

9. The method according to claim 3 or 4, characterized in that: Methods for determining distance bonus values ​​include: Determining the traffic density according to the road network information at the Nth moment after the initial moment; Determine the safe distance based on traffic density; If the distance between the vehicle and other vehicles after the merging is greater than the safety distance, setting the distance reward function value to a first positive reward value; If the distance between the vehicle and other vehicles after the merging is less than or equal to the safety distance, the distance reward function value is set to the first negative reward value.

10. A driving decision determination device, characterized in that: The device comprises: A driving status information determination module, used to collect sensor data within the coverage area of ​​the road terminal equipment in real time, and determine the driving status information of each vehicle within the coverage area of ​​the road terminal equipment according to the sensor data; A scene type determination module is used to extract road network information from the driving state information at the initial moment, and perform scene recognition based on the sensor data and road network information at the initial moment to determine the scene type at the initial moment; A reward function determination module, used to determine the reward function corresponding to the scene type at the initial moment according to the mapping relationship between the scene type and the reward function in the pre-built scene library; A reward value determination module, used to input the driving state information at the Nth moment after the initial moment into the reward function to obtain a reward value; N is a positive integer; A multi-agent reinforcement learning module is used to use the road network information at the initial moment and the driving state information at the initial moment and N moments after the initial moment as state input data, and use the multi-agent reinforcement learning algorithm to process the state input data and the reward value to determine the driving decision corresponding to each vehicle; A driving decision sending module, used for sending the driving decisions corresponding to the respective vehicles to the corresponding vehicles, so that the respective vehicles drive according to the corresponding driving decisions; The initial moment update module is used to determine that when the driving status information determination module has not stopped collecting sensor data within the coverage range of the road-side device, take the N+1th moment after the initial moment as the initial moment and return to the scene type determination module; when it is determined that the driving status information determination module has stopped collecting sensor data within the coverage range of the road-side device, end the process.

Images