Deep reinforcement learning safety decision method for automatic driving long tail scene

Abstract

The application relates to the technical field of intelligent driving, in particular to a deep reinforcement learning safety decision method for automatic driving long-tail scenes. The method comprises the following steps: matching a topological relationship graph in a long-tail knowledge graph to obtain a scene similarity, and determining whether there is an implicit risk element in the driving scene of the ego vehicle based on the scene similarity; in the case that it is determined that there is an implicit risk element in the driving scene of the ego vehicle, the attribute information of the implicit risk element is determined, and the spatial risk index is determined based on the attribute information of the ego vehicle and the implicit risk element; the scene complexity of the driving scene of the ego vehicle is determined based on the scene similarity and the spatial risk index; the attribute information of the ego vehicle, the attribute information of each road element and the scene complexity are input into a decision model to obtain a target action output by the decision model, and the ego vehicle is controlled based on the target action. The method can improve the safety of vehicle driving in complex scenes.

Description

Technical Field

[0001] This application relates to the technical field of intelligent driving, and in particular to a deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving. Background Technology

[0002] Autonomous vehicles are becoming a disruptive innovation and are gradually moving towards commercial applications. Real-world traffic scenarios are characterized by high diversity, complexity, and temporal dynamics, which pose new challenges to the safety of autonomous vehicles.

[0003] Among related technologies, autonomous driving technology can handle most routine scenarios and easily avoid the explicit risks they bring. Real-world traffic scenarios exhibit a typical long-tail distribution, characterized by a high frequency of routine scenarios and a low probability of extreme situations, unforeseen accidents, and complex scenarios caused by unpredictable human behavior, thus posing unknown implicit risks to vehicles. Most related technologies train intelligent driving decision-making models based on scenario information from actual road conditions, but their decision-making capabilities in complex scenarios remain insufficient.

[0004] Therefore, improving vehicle safety in complex driving scenarios is an urgent problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a deep reinforcement learning safety decision-making method for long-tail autonomous driving scenarios that can improve vehicle driving safety in complex scenarios and address the aforementioned technical problems.

[0006] Firstly, this application provides a deep reinforcement learning-based safety decision-making method for long-tail scenarios in autonomous driving, including:

[0007] The topological relationship graph is matched with the long-tail knowledge graph to obtain scene similarity, and the existence of implicit risk elements in the driving scene of the vehicle is determined based on the scene similarity. The topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene. The attribute information includes element category, position, direction of movement, and speed. The long-tail knowledge graph is constructed based on multiple sample scene information. The sample scene information includes sample elements, sample categories, and prior knowledge. The prior knowledge is the knowledge in the driving scene that includes the sample elements and sample categories.

[0008] If it is determined that there are implicit risk elements in the driving scenario of the vehicle, the attribute information of the implicit risk elements is determined, and a spatial risk index is determined based on the attribute information of the vehicle and the implicit risk elements respectively.

[0009] Based on the scene similarity and the spatial risk index, the scene complexity of the driving scene of the autonomous vehicle is determined;

[0010] The vehicle's attribute information, the attribute information of each road element, and the scene complexity are input into the decision model to obtain the target action output by the decision model, and the vehicle is controlled based on the target action. The target action includes a target lateral position sequence and a target longitudinal speed sequence.

[0011] The decision model is trained based on multiple scenario samples. Each scenario sample includes the attribute information of the target vehicle and each road element, as well as the scenario complexity of the target vehicle's driving scenario.

[0012] In one embodiment, the process of constructing the long-tail knowledge graph includes:

[0013] The sample elements, sample categories, and words in the prior knowledge are converted into feature word vectors, and a co-occurrence matrix is ​​constructed based on each feature word vector. The co-occurrence matrix represents the frequency of co-occurrence among the words.

[0014] The feature word vectors are trained as training samples, and the initial word vector embedding model is trained based on the training samples and the loss function. The co-occurrence matrix is ​​then updated according to the training results to obtain the updated co-occurrence matrix.

[0015] Based on the updated co-occurrence matrix and the feature word vectors, a long-tail knowledge graph is constructed.

[0016] In one embodiment, controlling the vehicle based on the target action includes:

[0017] The lateral position sequence is input into the path planning component to obtain a trajectory point sequence including multiple trajectory points and a target velocity sequence including multiple target velocities output by the path planning component.

[0018] The trajectory point sequence is input into the lateral controller to obtain the steering angle sequence output by the lateral controller, which includes multiple target steering angles;

[0019] The target velocity sequence is input into the longitudinal controller to obtain an acceleration sequence output by the longitudinal controller, which includes multiple target accelerations;

[0020] The vehicle is controlled based on the acceleration sequence and the steering angle sequence.

[0021] In one embodiment, the training process of the decision model includes:

[0022] The attribute information of each simulation element is obtained from the simulation space, and the scene complexity of the sample scene is determined based on the attribute information of each simulation element. The simulation elements include the target vehicle, each simulation scene element, and the simulation implicit risk element; the attribute information includes element category, position, direction of movement, and speed.

[0023] Based on the attribute information of each simulation element, a simulation state space is constructed. The simulation state space represents the relationship between the state information of any simulation element and time. The state information includes velocity, position, direction of motion, and longitudinal angle.

[0024] The attribute information of each simulation element in the simulation space at each acquisition time is used as a training sample and input into the initial decision model to obtain the decision action output by the initial decision model for the training sample. The target vehicle in the simulation state space is then controlled to execute the decision action, which includes a lateral position sequence and a longitudinal velocity sequence. During the execution of the decision action by the target vehicle in the simulation state space, the simulation state space updates the state information of each simulation element.

[0025] The reward value corresponding to the decision action is determined based on the reward function, and the value of the loss function of the initial decision model is also determined. The initial decision model is trained using multiple training samples, and the initial decision model is determined as the completed decision model under the condition that the reward value and the loss function of the initial decision model meet the preset rules. The reward function is constructed based on the scene complexity of the sample scene and the state information of each simulation element.

[0026] In one embodiment, the process of constructing the reward function includes:

[0027] A traffic efficiency reward item is constructed based on the actual speed and the ideal efficiency speed of the target vehicle, where the ideal efficiency speed is the maximum speed that can be achieved in this sample scenario.

[0028] A risk reward item is established based on the actual speed of the target vehicle, the safe speed, and the scenario complexity of the sample scenario. The safe speed is the highest speed at which the target vehicle will not collide with the simulated implicit risk element.

[0029] Based on the actual lateral position of the target vehicle and the lateral position sequence in the decision-making action, an arrival reward item is established;

[0030] Based on the state information of each simulation element, the collision reward item is determined;

[0031] A reward function is constructed based on the traffic efficiency reward, the risk reward, the arrival reward, and the collision reward.

[0032] In one embodiment, obtaining the attribute information of each simulation element from the simulation space includes:

[0033] Each simulation element is created in the simulation space, initial attribute information is configured for each simulation element, and random variables are set for each simulation element, wherein the random variables are used to randomly assign values ​​based on the initial attribute information.

[0034] Based on the initial attribute information and the random variables, the attribute information of each simulation element is obtained from the simulation space.

[0035] In one embodiment, the initial decision model includes an online value network and a target network with identical structures, and training the initial decision model using multiple training samples includes:

[0036] For each training sample, the training sample is input into the online value network to obtain the decision action output by the online value network, and the target vehicle in the simulation state space is controlled to execute the decision action. Based on the state information of each simulation element updated in the simulation state space, the reward value corresponding to the decision action and the latest state information of the target vehicle after executing the decision action are determined.

[0037] If the reward value is greater than a set threshold, the training sample, the reward value, the latest state information, and the decision action are combined to form an experience sample and placed into the experience pool.

[0038] Each of the aforementioned empirical samples is input into the online value network and the target network respectively to determine the target difference, which is the difference between the online value determined by the online value network and the target value determined by the target network;

[0039] The loss function of the online value network is updated with gradients based on the difference in the values, and after the loss function of the online value network is updated with gradients C times, the parameters of the online value network are copied to the target network.

[0040] In one embodiment, the preset rules include:

[0041] The curve of the reward value satisfies the first curve;

[0042] The curve of the value of the loss function satisfies the second curve.

[0043] Secondly, this application also provides a security decision-making device, which includes an implicit risk element determination module, a spatial risk index determination module, a scene complexity determination module, and a decision-making module, wherein:

[0044] An implicit risk element determination module is used to match a topological relationship graph in a long-tail knowledge graph to obtain scene similarity, and determine whether implicit risk elements exist in the driving scene of the vehicle based on the scene similarity. The topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene. The attribute information includes element category, position, direction of movement, and speed. The long-tail knowledge graph is constructed based on multiple sample scene information. The sample scene information includes sample elements, sample categories, and prior knowledge. The prior knowledge is knowledge in the driving scene that includes the sample elements and sample categories.

[0045] The spatial risk index determination module is used to determine the attribute information of the implicit risk element when it is determined that there is an implicit risk element in the driving scenario of the vehicle, and to determine the spatial risk index based on the attribute information of the vehicle and the implicit risk element respectively.

[0046] The scenario complexity determination module is used to determine the scenario complexity of the driving scenario of the vehicle based on the scenario similarity and the spatial risk index.

[0047] The decision module is used to input the attribute information of the vehicle, the attribute information of each road element, and the scene complexity into the decision model to obtain the target action output by the decision model, and control the vehicle based on the target action. The target action includes a target lateral position sequence and a target longitudinal speed sequence. The decision model is trained based on multiple scene samples, and each scene sample includes the attribute information of the target vehicle and each road element, as well as the scene complexity of the target vehicle's driving scenario.

[0048] Thirdly, this application also provides a vehicle including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method as described in any of the first aspects above.

[0049] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects above.

[0050] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the first aspects above.

[0051] The aforementioned deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving constructs a topological relationship graph based on the attribute information of the vehicle and other road elements in the driving scenario. Then, it matches the topological relationship graph from the long-tail knowledge graph to obtain the similarity between the vehicle's driving scenario and the scenarios with implicit risk elements in the long-tail knowledge graph, thereby determining whether there are implicit risk elements in the current driving scenario of the vehicle. When implicit risk elements are identified in the current driving scenario of the autonomous vehicle, a spatial risk index is determined by combining the attribute information of the autonomous vehicle and other road elements in the driving scenario, as well as the attribute information of the identified implicit risk elements. Then, the scenario complexity of the current driving scenario is determined based on the risk index. Furthermore, the attribute information of the autonomous vehicle and other road elements in the driving scenario, along with the scenario complexity, are input into a trained decision model to obtain the target action output by the decision model. The decision model is pre-trained and trained on multiple scenario samples containing implicit risk elements. Each scenario sample includes the attribute information of each road element, the controlled target vehicle, and the scenario complexity of the target vehicle's driving scenario. In other words, the decision model has the ability to make decisions about the target vehicle in a driving scenario with risk elements. Therefore, by controlling the autonomous vehicle according to the target action output by the decision model, implicit risk elements can be avoided, reducing the dangers posed by implicit risk elements in the autonomous vehicle's driving scenario. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a security decision-making method in one embodiment;

[0054] Figure 2 This is a schematic diagram of a typical long-tail driving scenario in one embodiment;

[0055] Figure 3 This is a schematic diagram of the construction process of a long-tail knowledge graph in one embodiment;

[0056] Figure 4 This is a schematic diagram illustrating the process of determining a real-time topology graph in one embodiment;

[0057] Figure 5 This is a schematic diagram of the process of controlling the vehicle in one embodiment;

[0058] Figure 6This is a flowchart illustrating the steps involved in training a decision model in one embodiment.

[0059] Figure 7 This is a schematic diagram illustrating the training process of the initial decision model in one embodiment;

[0060] Figure 8 This is a schematic diagram of the decision model training stage in one embodiment;

[0061] Figure 9 This is a structural block diagram of a security decision-making device in one embodiment;

[0062] Figure 10 This is a diagram of the internal structure of a vehicle in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] In one exemplary embodiment, such as Figure 1 As shown, a deep reinforcement learning-based safety decision-making method for long-tail scenarios in autonomous driving is provided. The method is illustrated using a vehicle as an example, but this does not constitute a limitation on the application scenarios of the method. The method provided in this application may specifically include the following steps 110-140, wherein:

[0065] Step 110: Match the topology graph in the long-tail knowledge graph to obtain scene similarity, and determine whether there are implicit risk elements in the driving scene of the vehicle based on the scene similarity; the topology graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene, and the attribute information includes element category, position, direction of movement and speed; the long-tail knowledge graph is constructed based on multiple sample scene information, and the sample scene information includes sample elements, sample categories and prior knowledge, and the prior knowledge is the knowledge in the driving scene including sample elements and sample categories.

[0066] In this embodiment, the vehicle is equipped with various types of sensors, such as distance sensors, vision sensors, and ultrasonic / LiDAR detectors. In the current driving scenario, the vehicle's attribute information can be directly acquired by the vehicle. Furthermore, the attribute information of each road element within the vehicle's field of vision can also be directly acquired, or obtained by processing relevant information. Road elements include, but are not limited to, vehicles traveling on the road, as well as pedestrians, animals, and other targets appearing on the road ahead of the vehicle's direction of travel. Element categories are specific classifications of road elements; for example, if a road element is a vehicle target, its element category could be one of trucks, motorcycles, or buses. A topology graph represents the relationships between various road elements in the current driving scenario. The steps for constructing a topology graph will be explained in detail later.

[0067] The sample elements include various targets that may appear on the road, and the sample labels may also include categories that do not frequently appear on the road; for example, sample labels may include a soccer ball, or a toy, etc. That is, the sample elements and sample labels in multiple sample scenarios cover both targets and categories that appear frequently on the road, as well as targets and categories that appear less frequently. Prior knowledge is knowledge resulting from human rationality or intuition. In this embodiment, prior knowledge specifically refers to knowledge related to the road driving scenario associated with the sample elements and sample labels. For example, for a target whose sample element is a toy and whose sample label is a soccer ball, a corresponding prior knowledge might be "a soccer ball usually does not appear alone on the road."

[0068] A topological relationship graph is constructed based on the current vehicle's attributes. This graph is then matched with a pre-constructed long-tail knowledge graph to determine the scene similarity between the current vehicle's driving scenario and related scenarios in the long-tail knowledge graph. This allows us to infer whether there are any implicit risk elements in the current vehicle's driving scenario.

[0069] Step 120: If it is determined that there are implicit risk elements in the driving scenario of the vehicle, determine the attribute information of the implicit risk elements, and determine the spatial risk index based on the attribute information of the vehicle and the implicit risk elements.

[0070] In this embodiment of the application, the spatial risk index characterizes the probability of a collision between the vehicle and an implicit risk element. The distance between the vehicle and the implicit risk element is determined by the attribute information of the vehicle and the implicit risk element. The spatial risk index is negatively correlated with the distance; that is, the smaller the distance between the vehicle and the implicit risk element, the greater the probability of a collision and the larger the spatial risk index; conversely, the larger the distance between the vehicle and the implicit risk element, the lower the probability of a collision and the smaller the spatial risk index.

[0071] Step 130: Determine the scenario complexity of the vehicle's driving scenario based on scenario similarity and spatial risk indicators.

[0072] In this embodiment, scene complexity represents the complexity of the vehicle's driving scenario, that is, the probability of a collision. Scene complexity is related to the scene similarity determined by the attribute information of other road elements in the scene, and is also directly related to the spatial risk index. The calculation logic of scene complexity will be explained in detail later.

[0073] Step 140: Input the vehicle's attribute information, the attribute information of each road element, and the scene complexity into the decision model to obtain the target action output by the decision model, and control the vehicle based on the target action. The target action includes the target lateral position sequence and the target longitudinal speed sequence. The decision model is trained based on multiple scene samples. Each scene sample includes the attribute information of the target vehicle and each road element, as well as the scene complexity of the target vehicle's driving scene.

[0074] In this embodiment, the decision model is pre-trained and trained on multiple scenario samples containing implicit risk elements. Each scenario sample includes the attribute information of each road element, the controlled target vehicle, and the scenario complexity of the target vehicle's driving scenario. In other words, the decision model has the ability to make decisions about the target vehicle in driving scenarios with risk elements. In the target action, the target speed sequence is used to indicate adjustments to the vehicle's longitudinal acceleration and longitudinal position; the lateral position sequence is used to indicate adjustments to the vehicle's lateral position. Therefore, after determining the scenario complexity of the vehicle's driving scenario, the acquired attribute information of the vehicle, the attribute information of each road element, and the scenario complexity are input into the decision model. The vehicle is then controlled according to the target action output by the decision model, thereby achieving avoidance of implicit risk elements.

[0075] In the aforementioned deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving, a topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scenario. Then, the topological relationship graph is matched from the long-tail knowledge graph to obtain the similarity between the driving scenario of the vehicle and the scenario with implicit risk elements in the long-tail knowledge graph, thereby determining whether there are implicit risk elements in the current driving scenario of the vehicle. When implicit risk elements are identified in the current driving scenario of the autonomous vehicle, a spatial risk index is determined by combining the attribute information of the autonomous vehicle and other road elements in the driving scenario, as well as the attribute information of the identified implicit risk elements. Based on this risk index, the scenario complexity of the current driving scenario is then determined. Furthermore, the attribute information of the autonomous vehicle and other road elements in the driving scenario, along with the scenario complexity, are input into a trained decision model to obtain the target action output by the decision model. The decision model is pre-trained and trained on multiple scenario samples containing implicit risk elements. Each scenario sample includes the attribute information of each road element, the controlled target vehicle, and the scenario complexity of the target vehicle's driving scenario. In other words, the decision model has the ability to make decisions about the target vehicle in driving scenarios with risk elements. Therefore, by controlling the autonomous vehicle according to the target action output by the decision model, implicit risk elements can be avoided, reducing the dangers posed by implicit risk elements in the autonomous vehicle's driving scenario.

[0076] In a typical long-tail driving scenario, such as Figure 2 As shown in the diagram, the vehicle travels in the direction of the arrow. A child is present in the vehicle's blind spot ahead of the truck, while a soccer ball is a suddenly appearing road element visible to the vehicle. The vehicle acquires the state information St of each road element (excluding the child) and sends it to the long-tail scenario implicit risk reasoning and identification module (model). This determines the implicit risks (scenario complexity and spatial risk indicators) of the current driving scenario. The state information St of each road element, along with the implicit risks, is then input into the DQN-IR (Safety Decision Model). The Safety Decision Model outputs the action At that the vehicle needs to perform. The trajectory planning module determines the required trajectory point sequence based on the action At. The PID control module generates a steering angle sequence and an acceleration sequence based on the trajectory point sequence, and then controls the vehicle's movement based on these sequences, thereby avoiding the implicit risk (the child).

[0077] In one embodiment, reference Figure 3 The construction process of a long-tail knowledge graph can specifically include steps 001-003, where:

[0078] Step 001: Convert the sample elements, sample categories, and words in prior knowledge into feature word vectors, and construct a co-occurrence matrix based on each feature word vector. The co-occurrence matrix represents the frequency of co-occurrence among each word.

[0079] Step 002: Use each feature word vector as training samples, train the initial word vector embedding model based on the training samples and loss function, and update the co-occurrence matrix according to the training results to obtain the updated co-occurrence matrix;

[0080] Step 003: Based on the updated co-occurrence matrix and the word vectors of each feature, construct a long-tail knowledge graph.

[0081] Specifically, a long-tail knowledge graph based on hash tables and the GloVe model is constructed. The hash function enables fast searching and matching of the long-tail knowledge graph. An unsupervised word embedding method based on GloVe is used to segment semantic words such as scene elements, label categories, and prior knowledge, resulting in multiple words. Each word is then converted into a feature vector representation. An initial co-occurrence matrix is ​​then constructed based on these feature vectors. The co-occurrence matrix records the number of times the feature vectors corresponding to any two words co-occur in the context. GloVe uses the co-occurrence matrix to capture the correlation between words.

[0082] Furthermore, each feature vector is used as a training sample to perform unsupervised training on the word embedding model. By training a GloVe-based word embedding model and optimizing the loss function to learn the relationships between word vectors, a fully trained word embedding model is obtained. During the training process, the initial co-occurrence matrix is ​​continuously updated as training samples are used, resulting in an updated contribution matrix after training is complete. The updated co-occurrence matrix more accurately represents the correlation between sample elements, sample categories, and words in prior knowledge.

[0083] Furthermore, the structure of the co-occurrence matrix can be characterized by formula (1):

[0084] The vocabulary size (number of words) is V, the words are represented by indices i and j, and the range of words is 1 to V, X ij w represents the number of times word i and word j appear together in the context window. i For the word vectors of the target vocabulary, It is the word vector of the context vocabulary, b i and w i and The bias term, f(X) ij ) is a weighting function.

[0085] The weighting function can be expressed as formula (2), where:

[0086]

[0087] Furthermore, based on the loss function J, the word vector w i , and bias term b i , Perform gradient updates; for the specific update functions, please refer to formulas (3)-(6), where:

[0088]

[0089] Among them, l r It is the learning rate, used to control the step size of each update. and Let J represent the loss function J with respect to w. i b i , The partial derivatives of .

[0090] Furthermore, in step 110, the structure of the topology diagram of the vehicle can be represented by formula (7).

[0091] G T ={N,E,F}, formula (7); where G T This is a real-time generated topology structure, where N and E are defined as nodes and edges, respectively, and F is the time frame. Specifically, N consists of a numeric ID (representing the identity of a road element), the element's class, and its position (x, y) and velocity (v). x v y Composition. Where x represents the horizontal position, y represents the vertical position, and v... x Let v be the lateral velocity. y This represents the longitudinal velocity.

[0092] The nodes in the topological graph can be represented by formula (8), and the structure of the edges can be represented by formula (9). N={id,class,x,y,v x ,v y}, Formula (8); E={dist,direction,d,δ}, Formula (9); where, δ = arctan(dx, dy); d is the Euclidean distance, and the spatial relationship between two nodes is determined by the angle δ. dist depends on d and is defined as "near", "normal", or "far". The direction is defined as "front", "back", "left", "right", "left front", or "right front".

[0093] Furthermore, scene similarity includes the similarity of edge structures and nodes between the topological graph and the long-tail knowledge graph; where scene similarity is S. G The similarity of nodes is expressed by formula (10); S is the similarity of nodes. N The similarity of the edge structure is expressed by formula (11); S is the similarity of the edge structure. E , which is expressed by formula (12).

[0094] in: Formula (10);

[0095] S G =W N *S N +W E *S E , formula (12).

[0096] Specifically, w i and These are the feature vectors corresponding to nodes in the topological relationship graph and the long-tail knowledge graph, respectively, E. ij and W represents the edges in the topological graph and the long-tailed knowledge graph, respectively. N and W E These represent the weights of knowledge similarity (node ​​similarity) and edge structure similarity, respectively.

[0097] Furthermore, in step 130, the scene complexity is IR, and the scene complexity can be expressed by formula (13).

[0098] IR=W G *S G +W space *Space risk Formula (13). Where W G and W space These are the weights assigned to scene similarity and spatial risk indicators, respectively; Space risk As a spatial risk indicator, the closer the vehicle is to the risk element, the greater the spatial risk indicator value.

[0099] Designing hash functions can support fast searching and matching of long-tail knowledge graphs. The process of finding a specific key in the hash table can be represented by formulas (14) and (15), where:

[0100] G i ,value i =H[h(key)], formula (14);

[0101] H={(G1,key1,value1),(G2,key2,value2),…}, formula (15).

[0102] Where H is a hash table, h is a hash function, key is the key to be queried, G represents the knowledge graph associated with the key, and value is the attention value associated with the key.

[0103] like Figure 4 The diagram illustrates a process for determining a real-time topology graph within a driving scenario. (Refer to...) Figure 2 In the example of a driving scenario, based on the attribute information of each road element in the current driving scenario of the vehicle, where children are not visible to the vehicle, an initial topological relationship graph of each road element is first generated. Then, based on the initial topological relationship graph, it is matched with a long-tail knowledge graph to infer that there is an implicit risk element (child) in the current driving scenario, and then a real-time topological relationship graph structure (topological relationship graph) is constructed.

[0104] Furthermore, referring to Figure 5 In step 140, the vehicle is controlled based on the target action, which may specifically include steps 141 and 144, wherein:

[0105] Step 141: Input the target action into the path planning component to obtain the trajectory point sequence including multiple trajectory points and the target velocity sequence including multiple target velocities output by the path planning component.

[0106] Specifically, the target actions output by the decision model include a target lateral position sequence and a target longitudinal velocity sequence. However, in reality, if the vehicle executes the target actions, it may experience significant changes in direction and swaying, or even sudden braking or acceleration, which can easily lead to loss of control and instability. In this embodiment, the target actions determined by the decision model are input into the path planning component. The path planning component uses cubic interpolation to smooth the target lateral position sequence and the target longitudinal velocity sequence, thereby obtaining the trajectory point sequence and the target velocity sequence.

[0107] The common knowledge of cubic polynomial interpolation can be expressed as formulas (16) and (17), where:

[0108] P(t) = at 3 +bt 2 +ct+d, formula (16);

[0109] P(t)=(2t 3 -3t 2 +1)P0+(t 3 -2t 2 +t)M0+(-2t 3 +3t 2 )P1+(t 3 -t 2 )M1, formula (17).

[0110] Here, a, b, c, and d are the four parameters that need to be determined; these four parameters are determined by solving a system of linear equations using the properties of cubic polynomials. P0 and P1 are the initial and final points, respectively, and M0 and M1 are the initial and final velocities, respectively.

[0111] Step 142: Input the trajectory point sequence into the lateral controller to obtain the steering angle sequence output by the lateral controller, which includes multiple target steering angles;

[0112] Step 143: Input the target velocity sequence into the longitudinal controller to obtain the acceleration sequence output by the longitudinal controller, which includes multiple target accelerations;

[0113] Step 144: Control the vehicle based on the steering angle sequence and acceleration sequence.

[0114] Specifically, the trajectory point sequence and the target speed sequence represent the driving trajectory and state of the vehicle, which is the state performance of the vehicle after being controlled; that is, it is necessary to determine the control parameters for the vehicle based on the trajectory point sequence and the target speed sequence, so that the vehicle can be controlled based on the control parameters and manifest as driving based on the trajectory point sequence and the target speed sequence.

[0115] In practice, vehicle control typically includes lateral control and longitudinal control. Lateral control primarily adjusts the vehicle's lateral position on the road by adjusting the steering angle, while longitudinal control controls the vehicle's speed and longitudinal position by controlling acceleration. The trajectory point sequence determined in step 141 is input to the lateral controller for the vehicle, resulting in a steering angle sequence output by the lateral controller. The target speed sequence is then input to the longitudinal controller, resulting in an acceleration sequence output by the longitudinal controller for the vehicle. The steering angle sequence includes multiple steering angles distributed along future time, and the acceleration sequence includes multiple accelerations distributed along future time. The vehicle is controlled based on the acceleration sequence and steering angle sequence, allowing it to travel along the trajectory point sequence at the target speed sequence.

[0116] Specifically, both the longitudinal controller and the lateral controller are PID (Proportional-Integral-Derivative Controller) controllers.

[0117] The above content explains how to determine the complexity of the current driving scenario for the autonomous vehicle, and how to control the autonomous vehicle to plan its route based on the scenario complexity and a pre-selected connection decision model. Furthermore, this application embodiment further elaborates on the training process of the decision model.

[0118] In one embodiment, reference Figure 6 The specific steps for training the decision model may include steps 101-104, wherein:

[0119] Step 101: Obtain the attribute information of each simulation element from the simulation space, and determine the scene complexity of the sample scene based on the attribute information of each simulation element. The simulation elements include the target vehicle, each simulation scene element, and the simulation implicit risk element; the attribute information includes the element category, position, direction of movement, and speed.

[0120] Specifically, an autonomous driving numerical simulation environment is built using CARLA simulation software. The specific steps are as follows: A typical long-tail scenario of a driving environment is built using CARLA simulation software. Based on the scenario task, the initial state and random dynamic attributes of each simulation element in the scenario are set. Each simulation element is created in the simulation space, initial attribute information is configured for each simulation element, and random variables are set for each simulation element. The random variables are used to randomly assign values ​​based on the initial attribute information. Based on the initial attribute information and random variables, the attribute information of each simulation element is obtained from the simulation space.

[0121] The simulation elements (traffic participants) include the target vehicle, various simulation scene elements, and implicit risk elements. Simulation scene elements are the vehicles on the roads, and implicit risk elements can be children or other elements invisible to the target vehicle. Further, initial attribute information for each simulation element is set, along with random dynamic attributes for each scene element. The initial attribute information represents a fixed position, fixed direction of movement, and fixed speed for each simulation element within the simulation space. Random dynamic attributes include increments (positive or negative values) based on the initial attribute information. For simulation elements other than the target vehicle, random dynamic attributes also include whether the element exists in the simulation space.

[0122] In one possible implementation, the specific parameters required to construct the simulation scenario are shown in Table 1.

[0123] Table 1

[0124] Simulation element name Dynamic random attributes describe lane width 3.5m Lane width From the vehicle's initial position (605,-17.08) Starting point of self-driving task The destination of the vehicle (495,-17.08) End of self-driving task Initial speed of the vehicle 0km / h The speed of the car at the starting point Target efficiency speed 50km / h <![CDATA[v efficiency Encourage efficient driving. Target safe speed 10km / h <![CDATA[v safe Ensure safe driving. Truck initial position (505,-20.58) Truck starting point Truck speed 0km / h Truck stationary in scene Child's initial position (500,-26.05) Children's starting position Children's Random Velocity Set [3,5] lateral random velocity

[0125] Step 102: Based on the attribute information of each simulation element, construct the simulation state space. The simulation state space represents the relationship between the state information of any simulation element and time. The state information includes velocity, position, direction of motion, and longitudinal angle.

[0126] Specifically, after the simulation space is constructed, attribute information of each simulation element is obtained from the simulation space based on the initial attribute information and random variables. For any simulation scene element in the simulation space, the state information of that simulation scene element is the initial attribute information of that simulation scene element plus the random dynamic attribute at the observation time. Furthermore, the update frequency of the random dynamic attribute can be set to once per second, although other frequencies are also possible. This embodiment does not impose a specific limitation on the update frequency of the random dynamic attribute.

[0127] Based on the constructed simulation space, multiple sampling (observations) are performed, and the state information of each simulation scene element in the simulation space obtained from each sampling is used as a driving scenario of the target vehicle. In this way, multiple simulation driving scenarios that the target vehicle may face can be obtained. Since each simulation scene element has initial attribute information and random dynamic attributes, the simulation driving scenario obtained from each observation is different, thus providing multiple usable driving scenarios for the training of the decision model.

[0128] For each simulated driving scenario obtained from each sampling, a topological relationship graph of the simulated driving scenario is constructed based on the steps disclosed in step 110. Based on the topological relationship graph, the scenario similarity and spatial risk index between each simulated driving scenario and the long-tail knowledge graph are determined, thereby determining the scenario complexity of each simulated driving scenario, thus providing usable data for the subsequent training process of the decision model.

[0129] Furthermore, a state space S is constructed, which includes the vehicle's position (x). e ,y e ), lateral and longitudinal velocities (v) xe ,v ye ), the location of traffic participants within a certain range (x i ,y i ), lateral and longitudinal velocities (v) xi ,v yi The driving direction and the longitudinal angle θ, as well as the scene complexity IR(t), are given. The state space S can be represented by formulas (8)-(10).

[0130] S = [S e ,S i ,IR(t)], formula (18); S e =[x e ,y e ,v xe ,v ye ], formula (19);

[0131] S i =[θ,x i ,y i ,vxi ,v yi ], formula (20).

[0132] Where: in formulas (18)-(20), the table e represents the status information of the target vehicle (the vehicle in the simulation environment), and i represents the number of different traffic participants (simulation elements).

[0133] Furthermore, an action space A is constructed, which includes the lateral and longitudinal control of the target vehicle. The action space A is represented by formula (21).

[0134] A = [Lateral_Control, Longitudinal_Control], Formula (21); where, (the set of center points of the lane lines (lateral); where, Lateral_Control is the lateral control, including the trajectory point sequence; Longitudinal_Control is the longitudinal control, including the target speed sequence.

[0135] Step 103: Take the attribute information of each simulation element in the simulation space at each acquisition time as a training sample, input it into the initial decision model, obtain the decision action output by the initial decision model for the training sample, and control the target vehicle in the simulation state space to execute the decision action. The decision action includes the lateral position sequence and the longitudinal velocity sequence. During the process of the target vehicle executing the decision action in the simulation state space, the simulation state space updates the state information of each simulation element.

[0136] Specifically, during the training phase of the decision-making model, the control of the target vehicle, and during the practical application phase of the trained decision-making model, the control of the self-vehicle, are both achieved using pre-built path planning and control modules. Steps 142-143 have already explained how the path planning and control modules control the self-vehicle during the practical application phase of the trained decision-making model. The process of controlling the target vehicle using the path planning and control modules during the decision-making model training phase can be referenced in steps 142-143 above; the following further explains how to construct the path planning and control modules.

[0137] Furthermore, a path planning module and a control module are designed. For the path planning module, a path planning algorithm is designed, using cubic polynomial interpolation for trajectory planning, where the planned path points (trajectory points) can be represented by formula (22).

[0138] P(t) = at 3 +bt 2 +ct+d, formula (22); where t is time, P is position; a, b, c and d are preset coefficients.

[0139] For the control module, two PID (Proportional-Integral-Derivative) controllers are used for longitudinal speed control and lateral position control of the target vehicle, respectively. In longitudinal speed control, the input is the target speed v output by the trajectory planning module. target and current speed v current The longitudinal controller uses a PID algorithm to calculate the acceleration α, which can be expressed by formula (23).

[0140] Where e is the velocity error v target -v current , Let ∫e^(-t) be the derivative of the velocity error, i.e., the rate of change of the velocity error, and let ∫e^(-t) be the integral of the velocity error, representing the accumulation of the velocity error over time. P K D K I These are the proportional, derivative, and integral parameters of the PID controller, and Δt is the sampling (observation) time. In lateral position control, the input data are the target path point output by the trajectory planning module and the current position of the target vehicle. The lateral controller uses the PID algorithm to calculate the steering angle δ, which can be expressed by formula (24).

[0141] Where e is the angular error between the current orientation of the target vehicle and the orientation of the target path point; The derivative of the included angle error, i.e., the rate of change of the included angle error; ∫e dt is the integral of the included angle error, representing the accumulation of the included angle error over time; K P K D K I These are the proportional, derivative, and integral parameters of the PID controller, and Δt is the sampling time. In one possible implementation, these parameters are taken as K. p =0.75, K D =0.02, K I =0.4, Δt=0.05s.

[0142] Step 104: Determine the reward value corresponding to the decision action based on the reward function, and determine the value of the loss function of the initial decision model; train the initial decision model using multiple training samples, and determine the initial decision model as the training model after training is completed if the reward value and the loss function of the initial decision model meet the preset rules. The reward function is constructed based on the scene complexity of the sample scene and the state information of each simulation element.

[0143] Specifically, the process of constructing the reward function may include: constructing a traffic efficiency reward term based on the actual speed and ideal efficiency speed of the target vehicle, where the ideal efficiency speed is the maximum speed achievable in the sample scenario; establishing a risk reward term based on the actual speed of the target vehicle, the safe speed, and the scenario complexity of the sample scenario, where the safe speed is the highest speed at which the target vehicle will not collide with the simulated implicit risk elements; establishing an arrival reward term based on the actual lateral position of the target vehicle and the lateral position sequence in the decision-making action; determining a collision reward term based on the state information of each simulation element; and constructing the reward function based on the traffic efficiency reward term, risk reward term, arrival reward term, and collision reward term.

[0144] Specifically, the constructed reward function R can be represented by formula (25). The reward function R can specifically include the traffic efficiency reward term r. efficiency Incentives for reaching the goal include arrival rewards. target Risk reward item r for driving safety considering spatial risk indicators safe_speed And the collision reward item r for whether a collision occurs. sa f e .

[0145] Specifically, r efficiency =k1(v efficiency -|v e -v efficiency |), Traffic efficiency reward items.

[0146] r target =k2χ target_position Arrive at the reward item.

[0147] r safe_speed =k3(v safe -|v e -v safe |)χ Impliciti_risk Risk-based reward items.

[0148] r safe =-k4χ collide Collision bonus items.

[0149] R = k1r efficiency +k2r target +k3r safe_speed +k4r safe , formula (25).

[0150] Where k1, k2, k3, and k4 are the weight coefficients of each reward item. χ target_position and X collide Indicates whether the predetermined destination was reached and whether a collision occurred, with a value of 0 or 1; χ Impliciti_riskThis indicates whether space risk exists; specifically, if the space risk index is greater than a preset threshold, it indicates that space risk exists; otherwise, it indicates that space risk does not exist. This result is provided by the space risk identification module. e v is the current actual speed of the target vehicle. efficiency For the preset ideal efficiency speed, v safe The preset safe speed.

[0151] In one possible implementation, the values ​​of k1, k2, k3, and k4 are respectively 1.4, 0.1, 0.3, and 400.

[0152] Furthermore, the initial decision model adopts the DQN (Deep Q-Network) model, which includes an online value network and a target network with identical structures, referencing... Figure 7 The training process for the initial decision model may specifically include steps 1041-1044, wherein:

[0153] Step 1041: For each training sample, input the training sample into the online value network to obtain the decision action output by the online value network, and control the target vehicle in the simulation state space to execute the decision action. Based on the state information of each simulation element updated in the simulation state space, determine the reward value corresponding to the decision action, as well as the latest state information of the target vehicle after executing the decision action.

[0154] Specifically, the online value network and the target network have the same structure, both consisting of three layers: an input layer, a hidden layer, and an output layer. The input layer represents the current vehicle state S, the hidden layers each have 128 neurons, and the output layer represents the action corresponding to the maximum Q-value selectable in the current state.

[0155] First, initialize the parameters θ of the online Q-network and the parameters of the target Q-network. For each training round corresponding to a training sample, the initial state S0 of the target vehicle corresponding to the training sample is input, and action A is selected from the action sequence output by the online value Q network using the ε-greedy strategy. t And execute, and determine in action A t The corresponding reward R and the state information S of the target vehicle in the next time step t+1 .

[0156] Step 1042: When the reward value is greater than the set threshold, construct an experience sample by combining the training sample, reward value, latest state information and decision action and put it into the experience pool.

[0157] Specifically, for each training round corresponding to a training sample, if the determined reward R is greater than a set threshold, then an effective experience sample (S) is constructed based on the target vehicle's state, the actions performed by the target vehicle, the reward value, and the target vehicle's state at the next time step. t A t ,R t ,S t+1 The experience samples are stored in the experience pool D. When the amount of experience samples exceeds the capacity of the experience pool D, the first-in, first-out principle is followed, and the latest generated experience sample replaces the experience sample stored earlier in the experience pool.

[0158] Step 1043: Input each empirical sample into the online value network and the target network respectively, and determine the target difference. The target difference is the difference between the online value determined by the online value network and the target value determined by the target network.

[0159] Step 1044: Update the gradient of the loss function of the online value network based on the difference of values, and after updating the loss function of the online value network by C gradients, copy the parameters of the online value network to the target network.

[0160] Specifically, empirical samples are randomly drawn from the experience pool to calculate the Q-value of the target network. The Q-value is the value of the loss function. Then, the difference between the Q-values ​​of the empirical samples in the online network and the target network is further calculated. The loss function L(θ) can be expressed by formula (26), where:

[0161] L(θ)=E s,a [(y i -Q(s,a;θ)) 2 ], Formula (26); further, y j Let be a step function, which can be represented by formula (27), where:

[0162]

[0163] Specifically, L(θ) represents the loss function, E s,a For target network The mean square error of the Q-output value (Q-value) of the online value network; y i Represents the target network The output Q value, r j Let γ be the reward value at step j, γ be the discount rate of the training hyperparameters, and a be the reward value at step j. ′ Indicates the action that may be taken in the next moment, θ ′ The neural network parameters represent the target network. The target value network selection state S represents the network state. j+1 The maximum state action value.

[0164] Furthermore, regarding (y)j -Q(S j ,a j ;θ)) 2 Perform gradient descent and update the parameters of the online value network Q. Every C steps, copy the parameters of the online value network Q to the target network. This training cycle continues; where C > 1, but the specific value of C is not specifically limited in this embodiment. When the reward R... t An ideal safety decision model is obtained when the curve of the loss function L(θ) tends to stabilize; that is, when the curve of the reward value satisfies the first curve and the curve of the loss function satisfies the second curve, the online value network Q or the target network is obtained. The decision model is determined to be successfully trained. The first and second curves are set according to actual training requirements; however, no limitations are imposed on the first and second curves in this embodiment.

[0165] In one possible implementation, the hyperparameters used in the training process of the decision model described above are shown in Table 2.

[0166] Table 2 Hyperparameter Design for Decision Model Training

[0167]

[0168]

[0169] Based on the above, and referring to Figure 8 In this embodiment, the training of the decision model includes three stages: S1, S2, and S3, wherein:

[0170] Phase S1 involves building a long-tail scenario (simulation space) for autonomous driving based on CARLA. In a typical long-tail scenario built using CLRLA simulation software, static information (initial attribute information) of scenario elements (simulation elements) is set, and dynamic information (random dynamic attribute information) is set for each scenario element. Then, simulation environment testing is performed.

[0171] Phase S2 is the phase for scene complexity identification: a hash table and GloVe long-tail knowledge graph are pre-built, and then a generation algorithm (topological relationship graph) is designed to generate scene graph structure based on the attribute information of each road element in the driving scene; a multi-dimensional similarity fusion and inference model based on graph structure (topological relationship graph) is designed.

[0172] Multiple samplings are performed on the simulation space to obtain attribute information of each scene element (simulation element) in multiple simulated driving scenarios. A topological relationship graph is constructed for each simulated driving scenario. Based on the matching of the topological relationship graph and the long-tail knowledge graph, the implicit risk type (scenario complexity) and risk value (spatial risk index) of each simulated driving scenario are determined. Then, based on the inference model, the topological relationship of each simulated driving scenario is determined, along with the similarity (node ​​similarity), graph structure similarity (edge ​​similarity), and real-time spatial risk (spatial risk index) between the graph and the long-tail knowledge graph, thereby determining the scenario complexity.

[0173] Phase S3 is the training phase of the decision model. A DQN-IR (Initial Decision Model) based on deep reinforcement learning is designed. Then, a path planning and control module is designed for the initial decision model. By designing the state space, action space, and reward function, the network of the DQN-IR (Initial Decision Model) decision model is trained to obtain a deep reinforcement learning safety decision model that considers implicit risks.

[0174] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0175] Based on the same inventive concept, this application also provides a safety decision-making device for implementing the deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving as described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more safety decision-making device embodiments provided below can be found in the limitations of the deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving described above, and will not be repeated here.

[0176] In one exemplary embodiment, such as Figure 9 As shown, a safety decision-making device is provided. The device includes an implicit risk element determination module 901, a spatial risk index determination module 902, a scene complexity determination module 903, and a decision module 904, wherein:

[0177] The implicit risk element determination module 901 is used to match the topological relationship graph in the long-tail knowledge graph to obtain scene similarity, and determine whether there are implicit risk elements in the driving scene of the vehicle based on the scene similarity. The topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene. The attribute information includes element category, position, direction of movement and speed. The long-tail knowledge graph is constructed based on multiple sample scene information. The sample scene information includes sample elements, sample categories and prior knowledge. The prior knowledge is the knowledge in the driving scene including sample elements and sample categories.

[0178] The spatial risk index determination module 902 is used to determine the attribute information of implicit risk elements when it is determined that there are implicit risk elements in the driving scenario of the vehicle, and to determine the spatial risk index based on the attribute information of the vehicle and the implicit risk elements respectively.

[0179] The scenario complexity determination module 903 is used to determine the scenario complexity of the vehicle's driving scenario based on scenario similarity and spatial risk indicators.

[0180] The decision module 904 is used to input the attribute information of the vehicle, the attribute information of each road element, and the scene complexity into the decision model to obtain the target action output by the decision model, and control the vehicle based on the target action. The target action includes the target lateral position sequence and the target longitudinal speed sequence. The decision model is trained based on multiple scene samples. Each scene sample includes the attribute information of the target vehicle and each road element, as well as the scene complexity of the target vehicle's driving scenario.

[0181] In one embodiment, the security decision-making device further includes a construction module, which is specifically used for:

[0182] The sample elements, sample categories, and words in prior knowledge are transformed into feature word vectors, and a co-occurrence matrix is ​​constructed based on each feature word vector. The co-occurrence matrix represents the frequency of co-occurrence among each word.

[0183] Each feature word vector is trained as a training sample, and the initial word vector embedding model is trained based on the training samples and the loss function. The co-occurrence matrix is ​​then updated based on the training results to obtain the updated co-occurrence matrix.

[0184] Based on the updated co-occurrence matrix and the word vectors of each feature, a long-tail knowledge graph is constructed.

[0185] In one embodiment, the decision module 904 is specifically used for:

[0186] Input the lateral position sequence into the path planning component to obtain the trajectory point sequence including multiple trajectory points and the target velocity sequence including multiple target velocities output by the path planning component;

[0187] Input the trajectory point sequence into the lateral controller to obtain the steering angle sequence output by the lateral controller, which includes multiple target steering angles;

[0188] The target velocity sequence is input into the longitudinal controller to obtain an acceleration sequence output by the longitudinal controller, which includes multiple target accelerations;

[0189] The vehicle is controlled based on acceleration and steering angle sequences.

[0190] In one embodiment, the security decision-making device further includes a training module, wherein the training module is specifically used for:

[0191] The attribute information of each simulation element is obtained from the simulation space, and the scene complexity of the sample scene is determined based on the attribute information of each simulation element. The simulation elements include the target vehicle, each simulation scene element, and the simulation implicit risk element; the attribute information includes element category, position, direction of movement, and speed.

[0192] Based on the attribute information of each simulation element, a simulation state space is constructed. The simulation state space represents the relationship between the state information of any simulation element and time. The state information includes velocity, position, direction of motion, and longitudinal angle.

[0193] The attribute information of each simulation element in the simulation space at each acquisition time is used as a training sample and input into the initial decision model to obtain the decision action output by the initial decision model in response to the training sample. The decision action is then used to control the target vehicle in the simulation state space to execute the decision action, which includes a lateral position sequence and a longitudinal velocity sequence. During the process of the target vehicle executing the decision action in the simulation state space, the simulation state space updates the state information of each simulation element.

[0194] The reward value corresponding to the decision action is determined based on the reward function, and the value of the loss function of the initial decision model is also determined. The initial decision model is trained using multiple training samples, and the initial decision model is determined as the completed decision model under the condition that the reward value and the loss function of the initial decision model meet the preset rules. The reward function is constructed based on the scene complexity of the sample scene and the state information of each simulation element.

[0195] In one embodiment, the training module is further configured to:

[0196] A traffic efficiency reward item is constructed based on the actual speed and the ideal efficiency speed of the target vehicle. The ideal efficiency speed is the maximum speed that can be achieved in this sample scenario.

[0197] A risk reward item is established based on the target vehicle's actual speed, safe speed, and the complexity of the sample scenario. The safe speed is the highest speed at which the target vehicle will not collide with the simulated implicit risk element.

[0198] Based on the actual lateral position of the target vehicle and the lateral position sequence in the decision-making actions, an arrival reward item is established;

[0199] Based on the state information of each simulation element, determine the collision reward item;

[0200] A reward function is constructed based on traffic efficiency reward, risk reward, arrival reward, and collision reward.

[0201] In one embodiment, the training module is further configured to:

[0202] Create each simulation element in the simulation space, configure initial attribute information for each simulation element, and set random variables for each simulation element. The random variables are used to assign random values ​​based on the initial attribute information.

[0203] Based on the initial attribute information and random variables, the attribute information of each simulation element is obtained from the simulation space.

[0204] In one embodiment, the training module is further configured to:

[0205] For each training sample, the training sample is input into the online value network to obtain the decision action output by the online value network, and the target vehicle in the simulation state space is controlled to execute the decision action. Based on the state information of each simulation element updated in the simulation state space, the reward value corresponding to the decision action and the latest state information of the target vehicle after executing the decision action are determined.

[0206] If the reward value is greater than the set threshold, the training sample, reward value, latest state information and decision action are combined to form an experience sample and put it into the experience pool.

[0207] Each empirical sample is input into the online value network and the target network respectively to determine the target difference, which is the difference between the online value determined by the online value network and the target value determined by the target network.

[0208] The loss function of the online value network is updated by gradient based on the difference in values. After C gradient updates are performed on the loss function of the online value network, the parameters of the online value network are copied to the target network.

[0209] In one embodiment, the preset rules include: the curve of the reward value satisfies a first curve; and the curve of the loss function value satisfies a second curve.

[0210] The modules in the aforementioned deep reinforcement learning safety decision-making device for long-tail scenarios of autonomous driving can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device configured in the vehicle, or stored in software in the memory of the computer device, so that the processor can call and execute the corresponding operations of each module.

[0211] In one exemplary embodiment, a vehicle is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the vehicle includes a processor, memory, input / output interfaces, a communication interface, a display unit, and input devices. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input devices are also connected to the system bus via the input / output interfaces. The vehicle's processor provides computing and control capabilities. The vehicle's memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The vehicle's input / output interfaces are used for exchanging information between the processor and external devices. The vehicle's communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a deep reinforcement learning-based safety decision-making method for long-tail scenarios in autonomous driving. The vehicle's display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the vehicle can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the vehicle body, or external keyboards, touchpads or mice, etc.

[0212] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the vehicle to which the present application is applied. A specific vehicle may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0213] In one exemplary embodiment, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement any of the steps in the above embodiments of the deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving.

[0214] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the steps described in the above embodiments of the deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving.

[0215] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the steps described in the above embodiments of the deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0217] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0218] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0219] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A deep reinforcement learning-based safety decision-making method for long-tail scenarios in autonomous driving, characterized in that, The method includes: The topological relationship graph is matched with the long-tail knowledge graph to obtain scene similarity, and the existence of implicit risk elements in the driving scene of the vehicle is determined based on the scene similarity. The topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene. The attribute information includes element category, position, direction of movement, and speed. The long-tail knowledge graph is constructed based on multiple sample scene information. The sample scene information includes sample elements, sample categories, and prior knowledge. The prior knowledge is the knowledge in the driving scene that includes the sample elements and sample categories. If it is determined that there are implicit risk elements in the driving scenario of the vehicle, the attribute information of the implicit risk elements is determined, and a spatial risk index is determined based on the attribute information of the vehicle and the implicit risk elements respectively. Based on the scene similarity and the spatial risk index, the scene complexity of the driving scene of the autonomous vehicle is determined; The vehicle's attribute information, the attribute information of each road element, and the scene complexity are input into the decision model to obtain the target action output by the decision model, and the vehicle is controlled based on the target action. The target action includes a target lateral position sequence and a target longitudinal speed sequence. The decision model is trained based on multiple scenario samples. Each scenario sample includes the attribute information of the target vehicle and each road element, as well as the scenario complexity of the target vehicle's driving scenario. The control of the vehicle based on the target action includes: The lateral position sequence is input into the path planning component to obtain a trajectory point sequence including multiple trajectory points and a target velocity sequence including multiple target velocities output by the path planning component. The trajectory point sequence is input into the lateral controller to obtain the steering angle sequence output by the lateral controller, which includes multiple target steering angles; The target velocity sequence is input into the longitudinal controller to obtain an acceleration sequence output by the longitudinal controller, which includes multiple target accelerations; The vehicle is controlled based on the acceleration sequence and the steering angle sequence; The training process of the decision model includes: The attribute information of each simulation element is obtained from the simulation space, and the scene complexity of the sample scene is determined based on the attribute information of each simulation element. The simulation elements include the target vehicle, each simulation scene element, and the simulation implicit risk element; the attribute information includes element category, position, direction of movement, and speed. Based on the attribute information of each simulation element, a simulation state space is constructed. The simulation state space represents the relationship between the state information of any simulation element and time. The state information includes velocity, position, direction of motion, and longitudinal angle. The attribute information of each simulation element in the simulation space at each acquisition time is used as a training sample and input into the initial decision model to obtain the decision action output by the initial decision model for the training sample. The target vehicle in the simulation state space is then controlled to execute the decision action, which includes a lateral position sequence and a longitudinal velocity sequence. During the execution of the decision action by the target vehicle in the simulation state space, the simulation state space updates the state information of each simulation element. The reward value corresponding to the decision action is determined based on the reward function, and the value of the loss function of the initial decision model is also determined. The initial decision model is trained using multiple training samples, and the initial decision model is determined as the trained decision model when the reward value and the loss function of the initial decision model meet the preset rules. The reward function is constructed based on the scene complexity of the sample scene and the state information of each simulation element. The process of constructing the reward function includes: A traffic efficiency reward item is constructed based on the actual speed and the ideal efficiency speed of the target vehicle, where the ideal efficiency speed is the maximum speed that can be achieved in this sample scenario. A risk reward item is established based on the actual speed of the target vehicle, the safe speed, and the scenario complexity of the sample scenario. The safe speed is the highest speed at which the target vehicle will not collide with the simulated implicit risk element. Based on the actual lateral position of the target vehicle and the lateral position sequence in the decision-making action, an arrival reward item is established; Based on the state information of each simulation element, the collision reward item is determined; A reward function is constructed based on the traffic efficiency reward, the risk reward, the arrival reward, and the collision reward.

2. The method according to claim 1, characterized in that, The construction process of the long-tail knowledge graph includes: The sample elements, sample categories, and words in the prior knowledge are converted into feature word vectors, and a co-occurrence matrix is ​​constructed based on each feature word vector. The co-occurrence matrix represents the frequency of co-occurrence among the words. The feature word vectors are trained as training samples, and the initial word vector embedding model is trained based on the training samples and the loss function. The co-occurrence matrix is ​​then updated according to the training results to obtain the updated co-occurrence matrix. Based on the updated co-occurrence matrix and the feature word vectors, a long-tail knowledge graph is constructed.

3. The method according to claim 1, characterized in that, The acquisition of attribute information of each simulation element from the simulation space includes: Each simulation element is created in the simulation space, initial attribute information is configured for each simulation element, and random variables are set for each simulation element, wherein the random variables are used to randomly assign values ​​based on the initial attribute information. Based on the initial attribute information and the random variables, the attribute information of each simulation element is obtained from the simulation space.

4. The method according to claim 1, characterized in that, The initial decision model includes an online value network and a target network with identical structures. Training the initial decision model using multiple training samples includes: For each training sample, the training sample is input into the online value network to obtain the decision action output by the online value network, and the target vehicle in the simulation state space is controlled to execute the decision action. Based on the state information of each simulation element updated in the simulation state space, the reward value corresponding to the decision action and the latest state information of the target vehicle after executing the decision action are determined. If the reward value is greater than a set threshold, the training sample, the reward value, the latest state information, and the decision action are combined to form an experience sample and placed into the experience pool. Each of the aforementioned empirical samples is input into the online value network and the target network respectively to determine the target difference, which is the difference between the online value determined by the online value network and the target value determined by the target network; The loss function of the online value network is updated with gradients based on the difference in the values, and after the loss function of the online value network is updated with gradients C times, the parameters of the online value network are copied to the target network.

5. The method according to claim 1, characterized in that, The preset rules include: The curve of the reward value satisfies the first curve; The curve of the value of the loss function satisfies the second curve.

6. A safety decision-making device, characterized in that, The device includes an implicit risk element determination module, a spatial risk index determination module, a scene complexity determination module, a decision-making module, and a training module, wherein: An implicit risk element determination module is used to match a topological relationship graph in a long-tail knowledge graph to obtain scene similarity, and determine whether implicit risk elements exist in the driving scene of the vehicle based on the scene similarity. The topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene. The attribute information includes element category, position, direction of movement, and speed. The long-tail knowledge graph is constructed based on multiple sample scene information. The sample scene information includes sample elements, sample categories, and prior knowledge. The prior knowledge is knowledge in the driving scene that includes the sample elements and sample categories. The spatial risk index determination module is used to determine the attribute information of the implicit risk element when it is determined that there is an implicit risk element in the driving scenario of the vehicle, and to determine the spatial risk index based on the attribute information of the vehicle and the implicit risk element respectively. The scenario complexity determination module is used to determine the scenario complexity of the driving scenario of the vehicle based on the scenario similarity and the spatial risk index. The decision module is used to input the attribute information of the vehicle, the attribute information of each road element, and the scene complexity into the decision model to obtain the target action output by the decision model, and control the vehicle based on the target action. The target action includes a target lateral position sequence and a target longitudinal speed sequence. The decision model is trained based on multiple scene samples, and each scene sample includes the attribute information of the target vehicle and each road element, as well as the scene complexity of the target vehicle's driving scenario. The decision module is specifically used to input the lateral position sequence into the path planning component to obtain the trajectory point sequence including multiple trajectory points and the target speed sequence including multiple target speeds output by the path planning component. The trajectory point sequence is input into the lateral controller to obtain the steering angle sequence output by the lateral controller, which includes multiple target steering angles; The target velocity sequence is input into the longitudinal controller to obtain an acceleration sequence output by the longitudinal controller, which includes multiple target accelerations; The vehicle is controlled based on the acceleration sequence and the steering angle sequence; The training module is specifically used to obtain attribute information of each simulation element from the simulation space, and determine the scene complexity of the sample scene based on the attribute information of each simulation element. The simulation elements include the target vehicle, each simulation scene element, and the simulation implicit risk element; the attribute information includes element category, position, direction of movement, and speed. Based on the attribute information of each simulation element, a simulation state space is constructed. The simulation state space represents the relationship between the state information of any simulation element and time. The state information includes velocity, position, direction of motion, and longitudinal angle. The attribute information of each simulation element in the simulation space at each acquisition time is used as a training sample and input into the initial decision model to obtain the decision action output by the initial decision model for the training sample. The target vehicle in the simulation state space is then controlled to execute the decision action, which includes a lateral position sequence and a longitudinal velocity sequence. During the execution of the decision action by the target vehicle in the simulation state space, the simulation state space updates the state information of each simulation element. The reward value corresponding to the decision action is determined based on the reward function, and the value of the loss function of the initial decision model is also determined. The initial decision model is trained using multiple training samples, and the initial decision model is determined as the trained decision model when the reward value and the loss function of the initial decision model meet the preset rules. The reward function is constructed based on the scene complexity of the sample scene and the state information of each simulation element. The training module is specifically used to construct a traffic efficiency reward item based on the actual speed and ideal efficiency speed of the target vehicle, wherein the ideal efficiency speed is the maximum speed that can be achieved in the sample scenario. A risk reward item is established based on the actual speed of the target vehicle, the safe speed, and the scenario complexity of the sample scenario. The safe speed is the highest speed at which the target vehicle will not collide with the simulated implicit risk element. Based on the actual lateral position of the target vehicle and the lateral position sequence in the decision-making action, an arrival reward item is established; Based on the state information of each simulation element, the collision reward item is determined; A reward function is constructed based on the traffic efficiency reward, the risk reward, the arrival reward, and the collision reward.

7. The apparatus according to claim 6, characterized in that, The device further includes a construction module, which is specifically used for: The sample elements, sample categories, and words in the prior knowledge are converted into feature word vectors, and a co-occurrence matrix is ​​constructed based on each feature word vector. The co-occurrence matrix represents the frequency of co-occurrence among the words. The feature word vectors are trained as training samples, and the initial word vector embedding model is trained based on the training samples and the loss function. The co-occurrence matrix is ​​then updated according to the training results to obtain the updated co-occurrence matrix. Based on the updated co-occurrence matrix and the feature word vectors, a long-tail knowledge graph is constructed.

8. The apparatus according to claim 6, characterized in that, The training module is also used for: Each simulation element is created in the simulation space, initial attribute information is configured for each simulation element, and random variables are set for each simulation element, wherein the random variables are used to randomly assign values ​​based on the initial attribute information. Based on the initial attribute information and the random variables, the attribute information of each simulation element is obtained from the simulation space.

9. The apparatus according to claim 6, characterized in that, The preset rules include: The curve of the reward value satisfies the first curve; The curve of the value of the loss function satisfies the second curve.

10. A vehicle comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

Images