Deep reinforcement learning safety decision-making method for automatic driving long-tail scene

Abstract

The invention relates to the technical field of intelligent driving, in particular to a deep reinforcement learning safety decision-making method for an automatic driving long-tail scene. The method comprises the following steps: matching a topological relation graph in a long-tail knowledge graph to obtain scene similarity, and determining whether an implicit risk element exists in a driving scene of a vehicle based on the scene similarity; determining attribute information of the implicit risk element under the condition of determining that the implicit risk element exists in the driving scene of the vehicle, and determining a space risk index based on the attribute information of the vehicle and the implicit risk element; determining the scene complexity of the driving scene of the vehicle based on the scene similarity and the space risk index; and inputting the attribute information of the vehicle, the attribute information of each road element and the scene complexity into a decision model to obtain a target action output by the decision model, and controlling the vehicle based on the target action. By the adoption of the method, the vehicle driving safety of the vehicle in a complex scene can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent driving, and in particular to a deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving. Background Art

[0002] Autonomous driving vehicles are becoming a disruptive innovation and are gradually moving towards commercial applications. Real-life traffic scenarios are highly diverse, complex, and dynamic, which poses new challenges to the safety of autonomous driving vehicles.

[0003] In related technologies, autonomous driving technology can handle most common scenarios and easily avoid the explicit risks they bring. Real traffic scenarios have a typical long-tail distribution, which is characterized by the high frequency of common scenarios and the low probability of complex scenarios caused by extreme situations, unforeseen accidents and unpredictable human behaviors, thus bringing unknown implicit risks to vehicles. In related technologies, most of them are based on the scene information of actual road conditions to train decision models related to intelligent driving, but the decision-making ability in complex scenarios is still insufficient.

[0004] Therefore, how to improve the safety of vehicle driving in complex scenarios is an urgent problem that needs to be solved. Summary of the invention

[0005] Based on this, it is necessary to provide a deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving, which can improve the vehicle's driving safety in complex scenarios, in response to the above technical problems.

[0006] In the first aspect, the present application provides a deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving, including:

[0007] Matching a topological relationship graph in a long-tail knowledge graph to obtain scene similarity, and determining whether there is an implicit risk element in the driving scene of the vehicle based on the scene similarity; the topological relationship graph is constructed based on attribute information of the vehicle and other road elements in the driving scene, and the attribute information includes element category, position, movement direction and speed; the long-tail knowledge graph is constructed based on multiple sample scene information, and the sample scene information includes sample elements, sample categories and prior knowledge, and the prior knowledge is knowledge in the driving scene including the sample elements and the sample categories;

[0008] In the case of determining that there is an implicit risk element in the driving scene of the ego vehicle, determining attribute information of the implicit risk element, and determining a spatial risk index based on the attribute information of each of the ego vehicle and the implicit risk element;

[0009] Determining the scene complexity of the driving scene of the ego-vehicle based on the scene similarity and the spatial risk index;

[0010] Inputting the attribute information of the ego vehicle, the attribute information of each road element, and the scene complexity into a decision model, obtaining a target action output by the decision model, and controlling the ego vehicle based on the target action, wherein the target action includes a target lateral position sequence and a target longitudinal speed sequence;

[0011] The decision model is obtained by training based on a plurality of scene samples, each of which includes attribute information of a target vehicle and each road element, and scene complexity of a driving scene of the target vehicle.

[0012] In one embodiment, the process of constructing the long-tail knowledge graph includes:

[0013] Convert the sample elements, the sample categories and the words in the prior knowledge into feature word vectors, and construct a co-occurrence matrix based on each of the feature word vectors, wherein the co-occurrence matrix represents the frequency of co-occurrence between each of the words;

[0014] Each of the feature word vector trainings is used as a training sample, and an initial word vector embedding model is trained based on the training samples and the loss function, and the co-occurrence matrix is ​​updated according to the training results to obtain an updated co-occurrence matrix;

[0015] Based on the updated co-occurrence matrix and each of the feature word vectors, a long-tail knowledge graph is constructed.

[0016] In one embodiment, controlling the vehicle based on the target action includes:

[0017] Inputting the lateral position sequence into a path planning component to obtain a trajectory point sequence including a plurality of trajectory points and a target speed sequence including a plurality of target speeds output by the path planning component;

[0018] Inputting the trajectory point sequence into a lateral controller to obtain a steering angle sequence including a plurality of target steering angles output by the lateral controller;

[0019] Inputting the target velocity sequence into a longitudinal controller to obtain an acceleration sequence including a plurality of target accelerations output by the longitudinal controller;

[0020] The ego vehicle is controlled based on the acceleration sequence and the steering angle sequence.

[0021] In one embodiment, the training process of the decision model includes:

[0022] Acquire attribute information of each simulation element from the simulation space, and determine the scene complexity of the sample scene based on the attribute information of each simulation element, wherein the simulation elements include a target vehicle, each simulation scene element, and a simulation implicit risk element; the attribute information includes element category, position, movement direction, and speed;

[0023] Based on the attribute information of each simulation element, construct a simulation state space, wherein the simulation state space represents the relationship between the state information of any simulation element and time, wherein the state information includes speed, position, movement direction and longitudinal angle;

[0024] The attribute information of each simulation element corresponding to the simulation space at each acquisition moment is used as a training sample, input into an initial decision model, obtain a decision action output by the initial decision model for the training sample, and control the target vehicle in the simulation state space to execute the decision action, wherein the decision action includes a lateral position sequence and a longitudinal speed sequence; wherein, during the process of the target vehicle executing the decision action in the simulation state space, the simulation state space updates the state information of each simulation element;

[0025] Based on the reward function, a reward value corresponding to the decision action is determined, and the value of the loss function of the initial decision model is determined; the initial decision model is trained using a plurality of the training samples, and under the condition that the reward value and the loss function of the initial decision model meet preset rules, the initial decision model is determined as a trained decision model, and the reward function is constructed based on the scene complexity of the sample scene and the state information of each of the simulation elements.

[0026] In one embodiment, the process of constructing the reward function includes:

[0027] Constructing a traffic efficiency reward item based on the actual speed of the target vehicle and the ideal efficiency speed, wherein the ideal efficiency speed is the maximum speed that can be achieved in the sample scenario;

[0028] Establishing a risk reward item based on the actual speed of the target vehicle, the safe speed and the scene complexity of the sample scene, wherein the safe speed is the maximum speed at which the target vehicle does not collide with the simulated implicit risk element;

[0029] establishing an arrival reward item based on the actual lateral position of the target vehicle and the lateral position sequence in the decision action;

[0030] Determining a collision reward item based on the status information of each simulation element;

[0031] A reward function is constructed based on the traffic efficiency reward item, the risk reward item, the arrival reward item, and the collision reward item.

[0032] In one embodiment, the acquiring the attribute information of each simulation element from the simulation space includes:

[0033] Creating each simulation element in the simulation space, configuring initial attribute information for each simulation element, and setting a random variable for each simulation element, wherein the random variable is used to randomly assign a value based on the initial attribute information;

[0034] Based on the initial attribute information and the random variable, the attribute information of each simulation element is acquired from the simulation space.

[0035] In one embodiment, the initial decision model includes an online value network and a target network with the same structure, and the training of the initial decision model using a plurality of the training samples includes:

[0036] For each training sample, the training sample is input into the online value network to obtain a decision action output by the online value network, and the target vehicle in the simulation state space is controlled to execute the decision action, and a reward value corresponding to the decision action and the latest state information of the target vehicle after executing the decision action are determined according to the state information of each simulation element updated in the simulation state space;

[0037] When the reward value is greater than a set threshold, construct an experience sample by using the training sample, the reward value, the latest state information and the decision action and put it into the experience pool;

[0038] Input each of the experience samples into the online value network and the target network respectively to determine a target difference, where the target difference is the difference between the online value determined by the online value network and the target value determined by the target network;

[0039] The loss function of the online value is gradient updated based on the difference in the value, and after the loss function of the online value network is gradient updated C times, the parameters of the online value network are copied to the target network.

[0040] In one embodiment, the preset rules include:

[0041] The curve of the reward value satisfies the first curve;

[0042] The curve of the value of the loss function satisfies the second curve.

[0043] In a second aspect, the present application also provides a safety decision-making device, the device comprising an implicit risk element determination module, a spatial risk index determination module, a scene complexity determination module and a decision module, wherein:

[0044] An implicit risk element determination module is used to match a topological relationship graph in a long-tail knowledge graph to obtain scene similarity, and determine whether there are implicit risk elements in the driving scene of the vehicle based on the scene similarity; the topological relationship graph is constructed based on attribute information of the vehicle and other road elements in the driving scene, and the attribute information includes element category, position, movement direction and speed; the long-tail knowledge graph is constructed based on multiple sample scene information, and the sample scene information includes sample elements, sample categories and prior knowledge, and the prior knowledge is knowledge in the driving scene including the sample elements and the sample categories;

[0045] A spatial risk index determination module, configured to determine attribute information of an implicit risk element when it is determined that an implicit risk element exists in the driving scene of the ego vehicle, and determine a spatial risk index based on the attribute information of each of the ego vehicle and the implicit risk element;

[0046] A scene complexity determination module, used to determine the scene complexity of the driving scene of the ego vehicle based on the scene similarity and the spatial risk index;

[0047] A decision module is used to input the attribute information of the ego vehicle, the attribute information of each road element and the scene complexity into a decision model, obtain a target action output by the decision model, and control the ego vehicle based on the target action, wherein the target action includes a target lateral position sequence and a target longitudinal speed sequence; wherein the decision model is obtained by training based on multiple scene samples, each of which includes the attribute information of the target vehicle and each road element, and the scene complexity of the driving scene of the target vehicle.

[0048] In a third aspect, the present application further provides a vehicle, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any one of the first aspects above when executing the computer program.

[0049] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects above are implemented.

[0050] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0051] The above-mentioned deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving constructs a topological relationship graph based on the attribute information of the vehicle and other road elements in the driving scene, and then matches the topological relationship graph from the long-tail knowledge graph to obtain the similarity between the driving scene of the vehicle and the scene with implicit risk elements in the long-tail knowledge graph, and then determines whether there are implicit risk elements in the current driving scene of the vehicle. When determining that there are implicit risk elements in the current driving scene of the own vehicle, the spatial risk index is determined by combining the attribute information of the own vehicle and other road elements in the driving scene, as well as the attribute information of the determined implicit risk elements, and then the scene complexity of the current driving scene of the own vehicle is determined according to the risk index; the attribute information of the own vehicle and other road elements in the driving scene and the scene complexity are further input into the trained decision model to obtain the target action output by the decision model; the decision model is pre-trained and is trained for multiple scene samples with implicit risk elements, each scene sample includes the attribute information of each road element, the controlled target vehicle and the scene complexity of the driving scene of the target vehicle; that is, the decision model has the ability to make decisions on the target vehicle in the driving scene with risk elements; therefore, the own vehicle is controlled according to the target action output by the decision model, so that the implicit risk elements can be avoided and the danger brought by the implicit risk elements in the driving scene of the own vehicle can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flowchart of a security decision-making method in one embodiment;

[0054] Figure 2 is a schematic diagram of a typical long-tail driving scenario in one embodiment;

[0055] Figure 3 A schematic diagram of a process for constructing a long-tail knowledge graph in one embodiment;

[0056] Figure 4 A schematic diagram of a process for determining a real-time topology relationship diagram in an embodiment;

[0057] Figure 5 A schematic diagram of a process of controlling a vehicle in one embodiment;

[0058] Figure 6A flowchart of the steps of training a decision model in an embodiment;

[0059] Figure 7 Schematic diagram of the training process of the initial decision model in one embodiment;

[0060] Figure 8 A schematic diagram of the stages of training and obtaining a decision model in an embodiment;

[0061] Fig. 9 is a structural block diagram of a security decision-making device in one embodiment;

[0062] Fig.10 FIG. 1 is a diagram showing the internal structure of a vehicle in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] In an exemplary embodiment, Figure 1 As shown, a deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving is provided, and the method is applied to a vehicle as an example for illustration, but this does not constitute a limitation on the application scenario of the method of the present application. The method provided by the present application may specifically include the following steps 110 to 140, wherein:

[0065] Step 110: Match the topological relationship graph in the long-tail knowledge graph to obtain scene similarity, and determine whether there are implicit risk elements in the driving scene of the vehicle based on the scene similarity; the topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene, and the attribute information includes element category, position, movement direction and speed; the long-tail knowledge graph is constructed based on multiple sample scene information, and the sample scene information includes sample elements, sample categories and prior knowledge, and the prior knowledge is the knowledge in the driving scene including sample elements and sample categories.

[0066] For the embodiments of the present application, the vehicle is equipped with various types of sensors, such as distance sensors, visual sensors, ultrasonic / lidar detectors, etc.; in the current driving scene of the vehicle, the attribute information of the vehicle can be directly obtained by the vehicle, and the attribute information of each road element within the perception field of the vehicle in the driving scene can also be directly obtained, or obtained by obtaining relevant information and processing. Among them, road elements include but are not limited to vehicles traveling on the road, and also include pedestrians, animals and other targets that appear on the road ahead of the vehicle in the direction of travel. The element category is a specific classification of road elements. For example, the road element is a target of the vehicle, and its element category can be one of a truck, a motorcycle and a bus. The topological relationship diagram represents the relationship between the various road elements in the current driving scene of the vehicle. In the subsequent content, the steps of how to construct the topological relationship diagram will be explained in detail.

[0067] Sample elements include a variety of targets that may appear on the road, and sample labels may also include classifications that do not often appear on the road; for example, sample labels may include footballs, and may also include dolls, etc. In other words, the sample elements and sample labels in multiple sample scenes cover both targets and classifications that appear frequently on the road, as well as targets and classifications that appear less frequently on the road. Prior knowledge is knowledge that is the result of human rationality or intuition. In an embodiment of the present application, prior knowledge specifically refers to knowledge in road driving scenarios related to sample elements and sample labels. For example, for a target whose sample element is a toy and whose sample label is a football, a corresponding prior knowledge may be "a football does not usually appear alone on the road."

[0068] A topological relationship graph is constructed based on the attributes of the current vehicle, and the topological relationship graph is matched in the pre-constructed long-tail knowledge graph to determine the scene similarity between the current driving scene of the vehicle and the relevant scene in the long-tail knowledge graph. Thus, it can be inferred whether there are implicit risk elements in the current driving scene of the vehicle.

[0069] Step 120: When it is determined that there is an implicit risk element in the driving scene of the own vehicle, attribute information of the implicit risk element is determined, and based on the attribute information of each of the own vehicle and the implicit risk element, a spatial risk index is determined.

[0070] For the embodiment of the present application, the spatial risk index represents the magnitude of the probability of a collision between the vehicle and the implicit risk element. The interval distance between the vehicle and the implicit risk element is determined by the attribute information of each of the vehicle and the implicit risk element, and the spatial risk index is negatively correlated with the interval distance; that is, the smaller the interval distance between the vehicle and the implicit risk element, the greater the probability of a collision, and the larger the spatial risk index; conversely, the larger the interval distance between the vehicle and the implicit risk element, the smaller the probability of a collision, and the smaller the spatial risk index.

[0071] Step 130: Determine the scene complexity of the driving scene of the vehicle based on the scene similarity and the spatial risk index.

[0072] For the embodiment of the present application, the scene complexity represents the complexity of the driving scene of the vehicle, that is, the probability of collision risk. The scene complexity is related to the scene similarity determined by the attribute information of other road elements in the scene, and is directly related to the spatial risk index. In the subsequent content, the calculation logic of the scene complexity will be explained in detail.

[0073] Step 140: input the attribute information of the vehicle, the attribute information of each road element, and the scene complexity into the decision model, obtain the target action output by the decision model, and control the vehicle based on the target action, the target action includes a target lateral position sequence and a target longitudinal speed sequence; wherein the decision model is obtained by training based on multiple scene samples, each scene sample includes the attribute information of the target vehicle and each road element, and the scene complexity of the driving scene of the target vehicle.

[0074] For the embodiment of the present application, the decision model is pre-trained and is obtained by training for multiple scene samples with implicit risk elements. Each scene sample includes each road element, attribute information of the controlled target vehicle and the scene complexity of the target vehicle's driving scene. In other words, the decision model has the ability to make decisions on the target vehicle in the driving scene with risk elements. In the target action, the target speed sequence is used to indicate the adjustment of the longitudinal acceleration and longitudinal position of the vehicle; the lateral position sequence is used to indicate the adjustment of the lateral position of the vehicle. Therefore, after determining the scene complexity of the driving scene of the vehicle, the acquired attribute information of the vehicle, the attribute information of each road element and the scene complexity are input into the decision model, and the vehicle is controlled according to the target action output by the decision model, thereby achieving the avoidance of implicit risk elements.

[0075] In the above-mentioned deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving, a topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene, and then the topological relationship graph is matched from the long-tail knowledge graph to obtain the similarity between the driving scene of the vehicle and the scene with implicit risk elements in the long-tail knowledge graph, and then determine whether there are implicit risk elements in the current driving scene of the vehicle. When determining that there are implicit risk elements in the current driving scene of the own vehicle, the spatial risk index is determined by combining the attribute information of the own vehicle and other road elements in the driving scene, as well as the attribute information of the determined implicit risk elements, and then the scene complexity of the current driving scene of the own vehicle is determined according to the risk index; the attribute information of the own vehicle and other road elements in the driving scene and the scene complexity are further input into the trained decision model to obtain the target action output by the decision model; the decision model is pre-trained and is trained for multiple scene samples with implicit risk elements, each scene sample includes the attribute information of each road element, the controlled target vehicle and the scene complexity of the driving scene of the target vehicle, that is, the decision model has the ability to make decisions on the target vehicle in the driving scene with risk elements; therefore, the own vehicle is controlled according to the target action output by the decision model, so that the implicit risk elements can be avoided and the danger brought by the implicit risk elements in the driving scene of the own vehicle can be reduced.

[0076] In a typical long-tail driving scenario such as Figure 2 As shown in the figure, the ego vehicle is traveling in the direction of the arrow. There is a child in front of the truck as a blind spot of the ego vehicle, and the football is a road element that suddenly appears and is visible to the ego vehicle. The ego vehicle obtains the state information St of each road element (excluding children) on the road, and sends the state information St of each road element to the long-tail scene implicit risk reasoning identification module (model), so as to determine the implicit risk (scenario complexity and spatial risk index) of the ego vehicle's current driving scene, and then inputs the implicit risk and the state information St of each road element into the DQN-IR (safety decision model), and the safety decision model outputs the action At that the ego vehicle needs to perform. The trajectory planning module determines the trajectory point sequence required by the ego vehicle based on the action At, and the PID control module generates the steering angle sequence and acceleration sequence based on the trajectory point sequence, and then controls the driving of the ego vehicle based on the steering angle sequence and the acceleration sequence, so as to avoid the implicit risk (children).

[0077] In one embodiment, referring to Figure 3 The construction process of the long-tail knowledge graph may specifically include steps 001 to 003, wherein:

[0078] Step 001: convert sample elements, sample categories and words in prior knowledge into feature word vectors, and construct a co-occurrence matrix based on each feature word vector, wherein the co-occurrence matrix represents the frequency of co-occurrence between words;

[0079] Step 002: training each feature word vector as a training sample, and training the initial word vector embedding model based on the training sample and the loss function, and updating the co-occurrence matrix according to the training results to obtain an updated co-occurrence matrix;

[0080] Step 003: Construct a long-tail knowledge graph based on the updated co-occurrence matrix and each feature word vector.

[0081] Specifically, a long-tail knowledge graph based on hash tables and GloVe models is constructed. The hash function can support fast search and matching of long-tail knowledge graphs. The unsupervised word vector embedding method based on GloVe is used to segment semantic words such as scene elements, label categories, and prior knowledge to obtain multiple words, and then each word is converted into a feature vector expression. Then, an initial co-occurrence matrix is ​​constructed based on each feature vector. The co-occurrence matrix records the number of times the feature vectors corresponding to any two words co-appear in the context, and GloVe captures the correlation between words through the co-occurrence matrix.

[0082] Furthermore, each feature vector is used as a training sample to perform unsupervised training on the word vector embedding model; by training the word vector embedding model based on GloVe and optimizing the loss function to learn the relationship between word vectors, a trained word vector embedding model is finally obtained. In the process of training the word vector model, as the training samples train the model, the initial co-occurrence matrix is ​​continuously updated, so that after the model training is completed, an updated contribution matrix is ​​obtained. The updated co-occurrence matrix can more accurately represent the correlation between sample elements, sample categories, and vocabulary in prior knowledge.

[0083] Furthermore, the structure of the co-occurrence matrix can be represented by formula (1):

[0084] The vocabulary size (number of words) is V, the vocabulary usage index is represented by i and j, and the vocabulary ranges from 1 to V. ij represents the number of times word i and word j appear together in the context window, w i is the word vector of the target vocabulary, is the word vector of the context vocabulary, b i and They are w i and The bias term, f(X ij ) is a weight function.

[0085] The weight function can be expressed as formula (2), where:

[0086]

[0087] Further, according to the loss function J, the word vector w i , and the bias term b i , Perform gradient update. For the specific update function, please refer to formula (3)-formula (6), where:

[0088]

[0089] Among them, l r is the learning rate, which controls the step size of each update. and They represent the loss function J with respect to w i , b i , The partial derivative of .

[0090] Furthermore, in step 110, the structure of the topological relationship graph of the ego vehicle can be expressed by formula (7).

[0091] G T ={N,E,F}, formula (7); where G T It is a topological structure generated in real time, N and E are defined as nodes and edges respectively, and F is the time frame. Specifically, N is composed of a digital id (representing the identity of the road element), an element class, and position x, y and speed v x , v y Among them, x is the horizontal position, y is the vertical position, and v is x is the lateral velocity, v y is the longitudinal speed.

[0092] The nodes in the topological relationship graph can be expressed by formula (8), and the edge structure can be expressed by formula (9). x ,v y}, formula (8); E = {dist,direction,d,δ}, formula (9); where, δ = arctan(dx, dy); d is the Euclidean distance, and the spatial relationship between two nodes is determined by the angle δ. dist is defined as "near", "normal", "far" depending on d. Direction is defined as "front", "back", "left", "right", "left front", "right front".

[0093] Furthermore, the scene similarity includes the similarity of the edge structure and the node similarity between the topological relationship graph and the long-tail knowledge graph; where the scene similarity is S G , expressed by formula (10); the similarity of the nodes is S N , expressed by formula (11); the similarity of the edge structure is S E , expressed by formula (12).

[0094] in: Formula (10);

[0095] S G =W N *S N +W E *S E , formula (12).

[0096] Specifically, w i and are the feature vectors corresponding to the nodes in the topological relationship graph and the long-tail knowledge graph, respectively. ij and are the edges in the topological relationship graph and the long-tail knowledge graph, respectively, N and W E They are the weights of knowledge similarity (node ​​similarity) and edge structure similarity respectively.

[0097] Furthermore, in step 130, the scene complexity is IR, and the scene complexity can be expressed by formula (13).

[0098] IR=W G *S G +W space *Space risk , formula (13). Where, W G and W space are the weights of scene similarity and space risk index respectively; Space risk It is a spatial risk index. The closer the vehicle is to the risk element, the greater the spatial risk index value.

[0099] The design of hash functions can support fast search and matching of long-tail knowledge graphs. The process of searching for a specific key in a hash table can be expressed by formulas (14) and (15), where:

[0100] G i ,value i =H[h(key)], formula (14);

[0101] H={(G1,key1,value1),(G2,key2,value2),…}, formula (15).

[0102] Among them, H is the hash table, h is the hash function, key is the key to be queried, G represents the knowledge graph associated with the key, and value is the attention value associated with the key.

[0103] like Figure 4 As shown in FIG. 1 , a process of determining a real-time topological relationship diagram based on a driving scenario is shown. Figure 2 An example of a driving scene in the example is given. Based on the attribute information of each road element in the current driving scene of the vehicle, where children are invisible to the vehicle, an initial topological relationship graph of each road element is generated first, and then the initial topological relationship graph is matched with the long-tail knowledge graph to infer that there are implicit risk elements (children) in the current driving scene, and then a real-time topological relationship graph structure (topological relationship graph) is constructed.

[0104] Further, refer to Figure 5 In step 140, the vehicle is controlled based on the target action, which may specifically include steps 141 and 144, wherein:

[0105] Step 141: Input the target action into the path planning component, and obtain a trajectory point sequence including a plurality of trajectory points and a target speed sequence including a plurality of target speeds output by the path planning component.

[0106] Specifically, the target action output by the decision model includes a target lateral position sequence and a target longitudinal speed sequence; however, in practice, if the vehicle executes according to the target action, there may be a large change of direction and swing, or even sudden braking or acceleration, which may easily cause the vehicle to lose control and become unstable. In an embodiment of the present application, the target action determined by the decision model is input into the path planning component, and the path planning component uses cubic interpolation to smooth the target lateral position sequence and the target longitudinal speed sequence, thereby obtaining a trajectory point sequence and a target speed sequence.

[0107] The well-known cubic polynomial interpolation can be expressed as formula (16) and formula (17), where:

[0108] P(t)=at 3 +bt 2 +ct+d, formula (16);

[0109] P(t)=(2t 3 -3t 2 +1)P0+(t 3 -2t 2 +t)M0+(-2t 3 +3t2 )P1+(t 3 -t 2 )M1, formula (17).

[0110] Among them, a, b, c and d are the four parameters that need to be determined; using the properties of cubic polynomials, these four parameters are determined by solving a set of linear equations. Among them, P0 and P1 are the initial point and the end point respectively, and M0 and M1 are the initial speed and the end speed respectively.

[0111] Step 142: input the trajectory point sequence into the lateral controller to obtain a steering angle sequence including a plurality of target steering angles output by the lateral controller;

[0112] Step 143, inputting the target speed sequence into the longitudinal controller to obtain an acceleration sequence including a plurality of target accelerations output by the longitudinal controller;

[0113] Step 144: Control the vehicle based on the steering angle sequence and the acceleration sequence.

[0114] Specifically, the trajectory point sequence and the target speed sequence represent the driving trajectory and state of the ego vehicle, and are the state performance of the ego vehicle after being controlled; that is, it is necessary to determine the control parameters for the ego vehicle based on the trajectory point sequence and the target speed sequence, so that the ego vehicle can be controlled based on the control parameters, which is manifested as driving based on the trajectory point sequence and the target speed sequence.

[0115] In practice, the control of a vehicle usually includes lateral control and longitudinal control; lateral control mainly adjusts the lateral position of the vehicle on the road by steering angle, and longitudinal control mainly controls the speed and longitudinal position of the vehicle by controlling acceleration. The trajectory point sequence determined by step 141 is input into the lateral controller for the vehicle, and a steering angle sequence for the vehicle output by the lateral controller is obtained; and the target speed sequence is input into the longitudinal controller, and an acceleration sequence for the vehicle output by the longitudinal controller is obtained. The steering angle sequence includes multiple steering angles distributed along the future time, and the acceleration sequence includes multiple accelerations distributed along the future time. The vehicle is controlled based on the acceleration sequence and the steering angle sequence, so that the vehicle can travel along the trajectory point sequence at the target speed sequence.

[0116] Specifically, both the longitudinal controller and the lateral controller are PID (Proportional-Integral-Derivative Controller) controllers.

[0117] The above content describes how to determine the complexity of the scene in which the vehicle is currently traveling, and how to control the vehicle to perform route planning based on the scene complexity and the pre-selected decision model. Furthermore, the embodiment of the present application further describes the training process of the decision model.

[0118] In one embodiment, referring to Figure 6 The step of training the decision model may specifically include steps 101 to 104, wherein:

[0119] Step 101, obtain attribute information of each simulation element from the simulation space, and determine the scene complexity of the sample scene based on the attribute information of each simulation element, the simulation elements include the target vehicle, each simulation scene element and simulation implicit risk element; the attribute information includes element category, position, movement direction and speed.

[0120] Specifically, the CARLA simulation software is used to build an autonomous driving numerical simulation environment, and the specific steps are: based on the CARLA simulation software, a long-tail scenario of a typical driving environment is built, and according to the scenario task, the initial state and random dynamic properties of each simulation element in the scenario are set; each simulation element is created in the simulation space, initial attribute information is configured for each simulation element, and random variables are set for each simulation element, where the random variables are used to randomly assign values ​​based on the initial attribute information; based on the initial attribute information and the random variables, the attribute information of each simulation element is obtained from the simulation space.

[0121] Among them, the simulation elements (traffic participants) include the target vehicle, each simulation scene element and the simulation implicit risk element. The simulation scene element is each road vehicle, and the implicit risk element can be a child or other elements that are invisible to the target vehicle. The initial attribute information of each simulation element is further set, and the random dynamic attribute of each scene element is set. Among them, the initial attribute information represents a fixed position, a fixed movement direction and a fixed speed of each simulation element in the simulation space; the random dynamic attribute includes an increment (positive or negative value) for each initial attribute information; for the remaining simulation elements except the target vehicle, the random dynamic attribute also includes whether the element exists in the simulation space.

[0122] In a possible implementation, specific parameters required to construct a simulation scenario are shown in Table 1.

[0123] Table 1

[0124] Simulation element name Dynamic random attributes describe Lane Width 3.5m Lane width Initial position of the vehicle (605,-17.08) Starting point of the self-driving task The target end position of the vehicle (495,-17.08) End of the self-driving mission Initial speed of the vehicle 0km / h The speed of the vehicle at the starting point Target efficiency speed 50km / h <![CDATA[v efficiency Encourage efficient driving]]> Target safe speed 10km / h <![CDATA[v safe Ensure safe driving]]> Truck initial position (505,-20.58) Truck starting location Truck speed 0km / h Truck stationary and in the scene Children's initial position (500,-26.05) Children's starting position Children's Random Speed ​​Set [3,5] Lateral random velocity

[0125] Step 102: construct a simulation state space based on the attribute information of each simulation element. The simulation state space represents the relationship between the state information of any simulation element and time. The state information includes speed, position, movement direction and longitudinal angle.

[0126] Specifically, after the simulation space is constructed, the attribute information of each simulation element is obtained from the simulation space based on the initial attribute information and random variables. Among them, for any simulation scene element in the simulation space, the state information of the simulation scene element is the initial attribute information of the simulation scene element + the random dynamic attribute at the observation time; further, the update frequency of the random dynamic attribute can be set to once per second, of course, it can also be set to other frequencies, and the update frequency of the random dynamic attribute is not specifically limited in the embodiment of the present application.

[0127] Based on the constructed simulation space, multiple sampling (observation) is performed, and the state information of each simulation scene element in the simulation space obtained from each sampling is used as a driving scene of the target vehicle, so that a variety of simulated driving scenes that the target vehicle may face can be obtained. Since each simulation scene element has initial attribute information and random dynamic attributes, the simulated driving scene obtained by each observation is different, thereby providing a variety of available driving scenes for the training of the decision model.

[0128] For each simulated driving scene obtained by each sampling, a topological relationship graph of the simulated driving scene is constructed based on the steps disclosed in step 110, and the scene similarity and spatial risk index of each simulated driving scene with the long-tail knowledge graph are determined based on the topological relationship graph, so as to determine the scene complexity of each simulated driving scene, thereby providing available data for the subsequent training process of the decision model.

[0129] Furthermore, a state space S is constructed, which includes the vehicle position (x e ,y e ), transverse and longitudinal speed (v xe ,v ye ), the position of traffic participants within a certain range (x i ,y i ), transverse and longitudinal speed (v xi ,v yi ), the driving direction and the longitudinal angle θ, and the scene complexity IR(t). The state space S can be expressed by formula (8)-formula (10).

[0130] S=[S e ,S i ,IR(t)], formula (18); S e =[x e ,y e ,v xe,v ye ], formula (19);

[0131] S i =[θ,x i ,y i ,v xi ,v yi ], formula (20).

[0132] Wherein: the following table e in formula (18)-formula (20) represents the state information of the target vehicle (the vehicle in the simulation environment), and i represents the number of different traffic participants (simulation elements).

[0133] Furthermore, an action space A is constructed, which includes the lateral control and longitudinal control of the target vehicle. The action space A is expressed by formula (21).

[0134] A = [Lateral_Control, Longitudinal_Control], formula (21); where (the set of lane center points (lateral); where Lateral_Control is lateral control, including the trajectory point sequence; Longitudinal_Control is longitudinal control, including the target speed sequence.

[0135] Step 103: Use the attribute information of each simulation element corresponding to the simulation space at each acquisition moment as a training sample, input it into the initial decision model, obtain the decision action output by the initial decision model for the training sample, and control the target vehicle in the simulation state space to perform the decision action, the decision action includes a lateral position sequence and a longitudinal speed sequence; wherein, during the process of the target vehicle performing the decision action in the simulation state space, the simulation state space updates the state information of each simulation element.

[0136] Specifically, in the training stage of the decision model, the control of the target vehicle, and in the stage of the decision model after the actual application training, the control of the self-vehicle are all realized by using the pre-built path planning module and control module. In steps 142-143, it has been explained how to realize the control of the self-vehicle through the path planning module and the control module in the stage of the decision model after the actual application training. In the stage of training the decision model, the process of realizing the control of the target vehicle through the path planning module and the control module can refer to the above steps 142-143; the following content further explains how to construct the path planning module and the control module.

[0137] Furthermore, a path planning module and a control module are designed. For the path planning module, a path planning algorithm is designed, and a cubic polynomial interpolation is used for trajectory planning, wherein the planned path points (trajectory points) can be expressed by formula (22).

[0138] P(t)=at 3 +bt 2 +ct+d, formula (22); where t is time, P is position; a, b, c and d are all preset coefficients.

[0139] For the control module, two PID (Proportional-Integral-Derivative Controller) controllers are used for the longitudinal speed control and lateral position control of the target vehicle respectively. In the longitudinal speed control, the input is the target speed v output by the trajectory planning module. target and current speed v current , the longitudinal controller uses the PID algorithm to calculate the acceleration α, which can be expressed by formula (23).

[0140] Where e is the velocity error v target -v current , is the derivative of the speed error, that is, the rate of change of the speed error, and ∫edt is the integral of the speed error, which represents the accumulation of the speed error over time. P , K D , K I are the proportional, differential and integral parameters of the PID controller, and Δt is the sampling (observation) time. In the lateral position control, the input data are the target path point output by the trajectory planning module and the current position of the target vehicle. The lateral controller uses the PID algorithm to calculate the steering angle δ, which can be expressed by formula (24).

[0141] Where e is the angle error between the current orientation of the target vehicle and the orientation of the target path point; The derivative of the angle error, that is, the rate of change of the angle error; ∫e dt is the integral of the angle error, which represents the accumulation of the angle error over time; K P , K D , K I are the proportional, differential and integral parameters of the PID controller, and Δt is the sampling time. In one possible implementation, they are K p =0.75, K D =0.02, K I =0.4, Δt=0.05s.

[0142] Step 104: determine the reward value corresponding to the decision action based on the reward function, and determine the value of the loss function of the initial decision model; use multiple training samples to train the initial decision model, and under the condition that the reward value and the loss function of the initial decision model meet the preset rules, determine the initial decision model as a trained training model, and the reward function is constructed based on the scene complexity of the sample scene and the state information of each simulation element.

[0143] Specifically, the reward function construction process may include: constructing a traffic efficiency reward item based on the actual speed of the target vehicle and the ideal efficiency speed, the ideal efficiency speed being the maximum speed that can be achieved in the sample scenario; establishing a risk reward item based on the actual speed of the target vehicle, the safe speed and the scene complexity of the sample scenario, the safe speed being the maximum speed at which the target vehicle does not collide with the simulated implicit risk element; establishing an arrival reward item based on the actual lateral position of the target vehicle and the lateral position sequence in the decision-making action; determining a collision reward item based on the status information of each simulation element; and constructing a reward function based on the traffic efficiency reward item, the risk reward item, the arrival reward item and the collision reward item.

[0144] Specifically, the constructed reward function R can be expressed by formula (25). The reward function R can specifically include the traffic efficiency reward term r efficiency , arrival reward items to encourage reaching the target target , the risk reward item r for driving safety considering spatial risk index safe_speed and the collision reward item r if a collision occurs sa f e .

[0145] Specifically, r efficiency = k1(v efficiency -|v e -v efficiency |), traffic efficiency reward item.

[0146] r target =k2χ target_position , arrive at the reward item.

[0147] r safe_speed =k3(v safe -|v e -v safe |) Impliciti_risk , risk reward item.

[0148] r safe =-k4χ collide , collision bonus item.

[0149] R=k1r efficiency +k2r target +k3rsafe_speed +k4r safe , formula (25).

[0150] Among them, k1, k2, k3, k4 are the weight coefficients of each reward item. target_position and X collide Indicates whether the predetermined end point is reached and whether a collision occurs, and takes the value 0 or 1; Impliciti_risk Indicates whether there is a spatial risk; when the spatial risk index is greater than the preset threshold, it indicates that there is a spatial risk, otherwise it does not exist. The result is provided by the spatial risk identification module. e is the actual speed of the target vehicle, v efficiency is the preset ideal efficiency speed, v safe The preset safety speed.

[0151] In one possible implementation, k1, k2, k3, and k4 are 1.4, 0.1, 0.3, and 400, respectively.

[0152] Furthermore, the initial decision model adopts the DQN (Deep Q-Network) model, which includes an online value network and a target network with the same structure. Figure 7 The training process of the initial decision model may specifically include steps 1041 to 1044, wherein:

[0153] Step 1041: For each training sample, the training sample is input into the online value network to obtain the decision action output by the online value network, and the target vehicle in the simulation state space is controlled to perform the decision action. According to the state information of each simulation element updated in the simulation state space, the reward value corresponding to the decision action and the latest state information of the target vehicle after performing the decision action are determined.

[0154] Specifically, the structures of the online value network and the target network are exactly the same, both of which include a three-layer structure, namely an input layer, a hidden layer, and an output layer. The input layer is the current vehicle state S, the hidden layer has 128 neural units, and the output layer is the action corresponding to the maximum Q value that can be selected under the current state.

[0155] First, initialize the parameters θ of the online value Q network and the parameters of the target Q network For each training round corresponding to a training sample, the initial state S0 of the target vehicle corresponding to the training sample is input, and the action A is selected from the action sequence output by the online value Q network using the ε-greedy strategy. t And execute, and determine in action A t The corresponding reward R and the state information S of the target vehicle in the next time step t+1 .

[0156] Step 1042: When the reward value is greater than the set threshold, the training sample, the reward value, the latest state information and the decision action are used to construct an experience sample and put it into the experience pool;

[0157] Specifically, for each training round corresponding to a training sample, if the determined reward R is greater than the set threshold, a valid experience sample (S ) is constructed based on the state of the target vehicle, the action performed by the target vehicle, the reward value, and the state of the target vehicle in the next time step. t ,A t ,R t ,S t+1 ), and store the experience samples in the experience pool D. When the amount of experience samples exceeds the capacity of the experience pool D, the first-in-first-out principle is followed, and the experience samples first stored in the experience pool are replaced with the latest generated experience samples.

[0158] Step 1043: input each experience sample into the online value network and the target network respectively to determine a target difference, where the target difference is the difference between the online value determined by the online value network and the target value determined by the target network;

[0159] Step 1044: perform a gradient update on the loss function of the online value based on the value difference, and after performing C gradient updates on the loss function of the online value network, copy the parameters of the online value network to the target network.

[0160] Specifically, an experience sample is randomly selected from the experience pool to calculate the Q value of the target network. The Q value is the value of the loss function, and then the difference between the Q values ​​of the experience sample in the online value network and the target network is further calculated. The loss function L(θ) can be expressed by formula (26), where:

[0161] L(θ)=E s,a [(y i -Q(s,a;θ) 2 ], formula (26); further, y j is a step function, which can be expressed by formula (27), where:

[0162]

[0163] Specifically, L(θ) represents the loss function, E s,a For the target network The mean square error with the online value network Q output value (Q value); i Indicates the target network Output Q value, r j is the reward value of the jth step, γ is the training hyperparameter discount rate, a ′ represents the possible action to be taken at the next moment, θ ′represents the neural network parameters of the target network, Indicates the target value network selection state S j+1 The maximum state action value.

[0164] Furthermore, for (y j -Q(S j ,a j ;θ)) 2 Perform gradient descent and update the parameters of the online value network Q, copying the parameters of the online value network Q to the target network every C steps The training is repeated in this way; wherein C>1, but the specific value of C is not specifically limited in the embodiment of the present application. t When the change curve of the loss function L(θ) tends to be stable, an ideal safety decision model is obtained; that is, when the reward value curve satisfies the first curve and the loss function value curve satisfies the second curve, the online value network Q or the target network The decision model is determined to be trained. The first curve and the second curve are both set according to actual training requirements, and the first curve and the second curve are not limited in any way in the embodiment of the present application.

[0165] In one possible implementation, in the above-mentioned training process of the decision model, the hyperparameters applied are shown in Table 2.

[0166] Table 2 Decision model training hyperparameter design

[0167]

[0168]

[0169] Combined with the above, refer to Figure 8 In the embodiment of the present application, the training of the decision model includes three stages: S1, S2 and S3, wherein:

[0170] The S1 stage is the construction of an autonomous driving long-tail scenario (simulation space) based on CARLA; in a typical long-tail scenario built based on the CLRLA simulation software, the static information (initial attribute information) of the scene elements (simulation elements) is set, and the dynamic information (random dynamic attribute information) is set for each scene element, and then a simulation environment test is performed.

[0171] The S2 stage is the stage of scene complexity identification: pre-build a long-tail knowledge graph based on hash tables and GloVe, and then design a generation algorithm (topological relationship graph) to generate the scene graph structure based on the attribute information of each road element in the driving scene; design a multi-dimensional similarity fusion and inference model based on the graph structure (topological relationship graph).

[0172] Multiple samplings are performed on the simulation space to obtain the attribute information of each scene element (simulation element) in multiple simulated driving scenes, and a topological relationship graph of each simulated driving scene is constructed. Based on the matching of the topological relationship graph and the long-tail knowledge graph, the implicit risk type (scenario complexity) and risk value (spatial risk index) of each simulated driving scene are determined. Then, based on the inference model, the topological relationship of each simulated driving scene is determined, and the similarity (node ​​similarity), graph structure similarity (edge ​​similarity) and spatial real-time risk (spatial risk index) with the long-tail knowledge graph are determined, and then the scene complexity is determined.

[0173] The S3 stage is the training stage of the decision model. DQN-IR (initial decision model) based on deep reinforcement learning is designed, and then the path planning and control module is designed for the initial decision model. By designing the state space, action space and reward function, the network of the DQN-IR (initial decision model) decision model is trained to obtain a safe decision model of deep reinforcement learning that considers implicit risks.

[0174] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0175] Based on the same inventive concept, the embodiment of the present application also provides a safety decision device for implementing the above-mentioned deep reinforcement learning safety decision method for long-tail scenarios of autonomous driving. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more safety decision device embodiments provided below can refer to the above limitations on the deep reinforcement learning safety decision method for long-tail scenarios of autonomous driving, and will not be repeated here.

[0176] In an exemplary embodiment, Fig. 9 As shown, a safety decision-making device is provided, the device includes an implicit risk element determination module 901, a spatial risk index determination module 902, a scene complexity determination module 903 and a decision module 904, wherein:

[0177] The implicit risk element determination module 901 is used to match the topological relationship graph in the long-tail knowledge graph, obtain the scene similarity, and determine whether there is an implicit risk element in the driving scene of the vehicle based on the scene similarity; the topological relationship graph is constructed based on the attribute information of the vehicle and other road elements in the driving scene, and the attribute information includes element category, position, movement direction and speed; the long-tail knowledge graph is constructed based on multiple sample scene information, and the sample scene information includes sample elements, sample categories and prior knowledge, and the prior knowledge is the knowledge in the driving scene including sample elements and sample categories;

[0178] A spatial risk index determination module 902 is used to determine the attribute information of the implicit risk element when it is determined that there is an implicit risk element in the driving scene of the ego vehicle, and determine the spatial risk index based on the attribute information of the ego vehicle and the implicit risk element;

[0179] A scene complexity determination module 903, used to determine the scene complexity of the driving scene of the ego vehicle based on the scene similarity and the spatial risk index;

[0180] The decision module 904 is used to input the attribute information of the vehicle, the attribute information of each road element and the scene complexity into the decision model, obtain the target action output by the decision model, and control the vehicle based on the target action, the target action includes a target lateral position sequence and a target longitudinal speed sequence; wherein the decision model is obtained by training based on multiple scene samples, each scene sample includes the attribute information of the target vehicle and each road element, and the scene complexity of the driving scene of the target vehicle.

[0181] In one embodiment, the security decision-making device further includes a construction module, and the construction module is specifically used to:

[0182] The sample elements, sample categories and words in the prior knowledge are converted into feature word vectors, and a co-occurrence matrix is ​​constructed based on each feature word vector. The co-occurrence matrix represents the frequency of co-occurrence between words.

[0183] Each feature word vector training is used as a training sample, and the initial word vector embedding model is trained based on the training sample and the loss function, and the co-occurrence matrix is ​​updated according to the training results to obtain an updated co-occurrence matrix;

[0184] Based on the updated co-occurrence matrix and each feature word vector, a long-tail knowledge graph is constructed.

[0185] In one embodiment, the decision module 904 is specifically configured to:

[0186] Inputting the lateral position sequence into the path planning component, obtaining a trajectory point sequence including a plurality of trajectory points and a target speed sequence including a plurality of target speeds output by the path planning component;

[0187] Inputting the trajectory point sequence into the lateral controller, obtaining a steering angle sequence including a plurality of target steering angles output by the lateral controller;

[0188] Inputting the target velocity sequence into the longitudinal controller, obtaining an acceleration sequence including a plurality of target accelerations output by the longitudinal controller;

[0189] Control the ego vehicle based on the acceleration sequence and the steering angle sequence.

[0190] In one embodiment, the security decision-making device further includes a training module, wherein the training module is specifically used to:

[0191] Acquire attribute information of each simulation element from the simulation space, and determine the scene complexity of the sample scene based on the attribute information of each simulation element, the simulation elements include the target vehicle, each simulation scene element and the simulation implicit risk element; the attribute information includes element category, position, movement direction and speed;

[0192] Based on the attribute information of each simulation element, a simulation state space is constructed. The simulation state space represents the relationship between the state information of any simulation element and time. The state information includes speed, position, movement direction and longitudinal angle.

[0193] The attribute information of each simulation element corresponding to each acquisition moment in the simulation space is used as a training sample, input into the initial decision model, obtain the decision action output by the initial decision model for the training sample, and control the target vehicle in the simulation state space to execute the decision action, the decision action includes a lateral position sequence and a longitudinal speed sequence; wherein, during the process of the target vehicle executing the decision action in the simulation state space, the simulation state space updates the state information of each simulation element;

[0194] Based on the reward function, a reward value corresponding to the decision action is determined, and the value of the loss function of the initial decision model is determined; multiple training samples are used to train the initial decision model, and under the condition that the reward value and the loss function of the initial decision model meet the preset rules, the initial decision model is determined as the trained decision model, and the reward function is constructed based on the scene complexity of the sample scene and the state information of each simulation element.

[0195] In one embodiment, the training module is further configured to:

[0196] Construct a traffic efficiency reward item based on the actual speed of the target vehicle and the ideal efficiency speed. The ideal efficiency speed is the maximum speed that can be achieved in this sample scenario.

[0197] The risk reward item is established based on the actual speed of the target vehicle, the safe speed and the scenario complexity of the sample scenario. The safe speed is the maximum speed at which the target vehicle does not collide with the implicit risk element of the simulation.

[0198] Establishing arrival reward items based on the actual lateral position of the target vehicle and the lateral position sequence in the decision action;

[0199] Determine collision reward items based on status information of each simulation element;

[0200] A reward function is constructed based on the traffic efficiency reward item, the risk reward item, the arrival reward item, and the collision reward item.

[0201] In one embodiment, the training module is further configured to:

[0202] Creating each simulation element in the simulation space, configuring initial attribute information for each simulation element, and setting a random variable for each simulation element, wherein the random variable is used to randomly assign a value based on the initial attribute information;

[0203] Based on the initial attribute information and random variables, the attribute information of each simulation element is obtained from the simulation space.

[0204] In one embodiment, the training module is further configured to:

[0205] For each training sample, the training sample is input into the online value network to obtain the decision action output by the online value network, and the target vehicle in the simulation state space is controlled to execute the decision action, and the reward value corresponding to the decision action and the latest state information of the target vehicle after executing the decision action are determined according to the state information of each simulation element updated in the simulation state space;

[0206] When the reward value is greater than the set threshold, the training sample, reward value, latest state information and decision action are used to construct an experience sample and put it into the experience pool;

[0207] Input each experience sample into the online value network and the target network respectively to determine the target difference, where the target difference is the difference between the online value determined by the online value network and the target value determined by the target network;

[0208] The loss function of the online value is gradient updated based on the value difference, and after the loss function of the online value network is gradient updated C times, the parameters of the online value network are copied to the target network.

[0209] In one embodiment, the preset rule includes: a curve of the reward value satisfies a first curve; and a curve of the value of the loss function satisfies a second curve.

[0210] Each module in the above-mentioned deep reinforcement learning safety decision-making device for long-tail scenarios of autonomous driving can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device configured in the vehicle in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0211] In an exemplary embodiment, a vehicle is provided, which may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The vehicle includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the vehicle is used to provide computing and control capabilities. The memory of the vehicle includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the vehicle is used to exchange information between the processor and an external device. The communication interface of the vehicle is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving is implemented. The display unit of the vehicle is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the vehicle can be a touch layer covering the display screen, or a button, trackball or touchpad set on the vehicle shell, or an external keyboard, touchpad or mouse.

[0212] Those skilled in the art will understand that Fig.10 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the vehicle to which the scheme of the present application is applied. A specific vehicle may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0213] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any step in the above-mentioned embodiment of the deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving is implemented.

[0214] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above-mentioned deep reinforcement learning safety decision-making method embodiment for long-tail scenarios of autonomous driving is implemented.

[0215] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements any step in the above-mentioned deep reinforcement learning safety decision-making method embodiment for long-tail scenarios of autonomous driving.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0217] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0218] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0219] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A deep reinforcement learning safety decision-making method for long-tail scenarios of autonomous driving, characterized in that: The method comprises: Matching a topological relationship graph in a long-tail knowledge graph to obtain scene similarity, and determining whether there is an implicit risk element in the driving scene of the vehicle based on the scene similarity; the topological relationship graph is constructed based on attribute information of the vehicle and other road elements in the driving scene, and the attribute information includes element category, position, movement direction and speed; the long-tail knowledge graph is constructed based on multiple sample scene information, and the sample scene information includes sample elements, sample categories and prior knowledge, and the prior knowledge is knowledge in the driving scene including the sample elements and the sample categories; In the case of determining that an implicit risk element exists in the driving scene of the ego vehicle, determining attribute information of the implicit risk element, and determining a spatial risk index based on the attribute information of each of the ego vehicle and the implicit risk element; Determining the scene complexity of the driving scene of the ego-vehicle based on the scene similarity and the spatial risk index; Inputting the attribute information of the ego vehicle, the attribute information of each road element, and the scene complexity into a decision model, obtaining a target action output by the decision model, and controlling the ego vehicle based on the target action, wherein the target action includes a target lateral position sequence and a target longitudinal speed sequence; The decision model is obtained by training based on a plurality of scene samples, each of which includes attribute information of a target vehicle and each road element, and scene complexity of a driving scene of the target vehicle.

2. The method according to claim 1, characterized in that: The construction process of the long-tail knowledge graph includes: Convert the sample elements, the sample categories and the words in the prior knowledge into feature word vectors, and construct a co-occurrence matrix based on each of the feature word vectors, wherein the co-occurrence matrix represents the frequency of co-occurrence between the words; Each of the feature word vector trainings is used as a training sample, and an initial word vector embedding model is trained based on the training samples and the loss function, and the co-occurrence matrix is ​​updated according to the training results to obtain an updated co-occurrence matrix; Based on the updated co-occurrence matrix and each of the feature word vectors, a long-tail knowledge graph is constructed.

3. The method according to claim 2, characterized in that The controlling the vehicle based on the target action includes: Inputting the lateral position sequence into a path planning component to obtain a trajectory point sequence including a plurality of trajectory points and a target speed sequence including a plurality of target speeds output by the path planning component; Inputting the trajectory point sequence into a lateral controller to obtain a steering angle sequence including a plurality of target steering angles output by the lateral controller; Inputting the target velocity sequence into a longitudinal controller to obtain an acceleration sequence including a plurality of target accelerations output by the longitudinal controller; The ego vehicle is controlled based on the acceleration sequence and the steering angle sequence.

4. The method according to claim 1, characterized in that: The training process of the decision model includes: Acquire attribute information of each simulation element from the simulation space, and determine the scene complexity of the sample scene based on the attribute information of each simulation element, wherein the simulation elements include a target vehicle, each simulation scene element, and a simulation implicit risk element; the attribute information includes element category, position, movement direction, and speed; Based on the attribute information of each simulation element, construct a simulation state space, wherein the simulation state space represents the relationship between the state information of any simulation element and time, wherein the state information includes speed, position, movement direction and longitudinal angle; The attribute information of each simulation element corresponding to the simulation space at each acquisition moment is used as a training sample, input into an initial decision model, obtain a decision action output by the initial decision model for the training sample, and control the target vehicle in the simulation state space to execute the decision action, wherein the decision action includes a lateral position sequence and a longitudinal speed sequence; wherein, during the process of the target vehicle executing the decision action in the simulation state space, the simulation state space updates the state information of each simulation element; Based on the reward function, a reward value corresponding to the decision action is determined, and the value of the loss function of the initial decision model is determined; the initial decision model is trained using a plurality of the training samples, and under the condition that the reward value and the loss function of the initial decision model meet preset rules, the initial decision model is determined as a trained decision model, and the reward function is constructed based on the scene complexity of the sample scene and the state information of each of the simulation elements.

5. The method according to claim 4, characterized in that The process of constructing the reward function includes: Constructing a traffic efficiency reward item based on the actual speed of the target vehicle and the ideal efficiency speed, wherein the ideal efficiency speed is the maximum speed that can be achieved in the sample scenario; Establishing a risk reward item based on the actual speed of the target vehicle, the safe speed and the scene complexity of the sample scene, wherein the safe speed is the maximum speed at which the target vehicle does not collide with the simulated implicit risk element; establishing an arrival reward item based on the actual lateral position of the target vehicle and the lateral position sequence in the decision action; Determining a collision reward item based on the status information of each simulation element; A reward function is constructed based on the traffic efficiency reward item, the risk reward item, the arrival reward item, and the collision reward item.

6. The method according to claim 4, characterized in that The acquiring the attribute information of each simulation element from the simulation space includes: Creating each simulation element in the simulation space, configuring initial attribute information for each simulation element, and setting a random variable for each simulation element, wherein the random variable is used to randomly assign a value based on the initial attribute information; Based on the initial attribute information and the random variable, the attribute information of each simulation element is acquired from the simulation space.

7. The method according to claim 4, characterized in that The initial decision model includes an online value network and a target network with the same structure, and the using of a plurality of the training samples to train the initial decision model includes: For each training sample, the training sample is input into the online value network to obtain a decision action output by the online value network, and the target vehicle in the simulation state space is controlled to execute the decision action, and a reward value corresponding to the decision action and the latest state information of the target vehicle after executing the decision action are determined according to the state information of each simulation element updated in the simulation state space; When the reward value is greater than a set threshold, construct an experience sample by using the training sample, the reward value, the latest state information and the decision action and put it into the experience pool; Input each of the experience samples into the online value network and the target network respectively to determine a target difference, where the target difference is the difference between the online value determined by the online value network and the target value determined by the target network; The loss function of the online value is gradient updated based on the difference in the value, and after the loss function of the online value network is gradient updated C times, the parameters of the online value network are copied to the target network.

8. The method according to claim 4, characterized in that The preset rules include: The curve of the reward value satisfies the first curve; The curve of the value of the loss function satisfies the second curve.

9. A security decision-making device, characterized in that: The device includes an implicit risk element determination module, a spatial risk index determination module, a scene complexity determination module and a decision module, wherein: An implicit risk element determination module is used to match a topological relationship graph in a long-tail knowledge graph to obtain scene similarity, and determine whether there are implicit risk elements in the driving scene of the vehicle based on the scene similarity; the topological relationship graph is constructed based on attribute information of the vehicle and other road elements in the driving scene, and the attribute information includes element category, position, movement direction and speed; the long-tail knowledge graph is constructed based on multiple sample scene information, and the sample scene information includes sample elements, sample categories and prior knowledge, and the prior knowledge is knowledge in the driving scene including the sample elements and the sample categories; A spatial risk index determination module, configured to determine attribute information of an implicit risk element when it is determined that an implicit risk element exists in the driving scene of the ego vehicle, and determine a spatial risk index based on the attribute information of each of the ego vehicle and the implicit risk element; A scene complexity determination module, used to determine the scene complexity of the driving scene of the ego vehicle based on the scene similarity and the spatial risk index; A decision module is used to input the attribute information of the ego vehicle, the attribute information of each road element and the scene complexity into a decision model, obtain a target action output by the decision model, and control the ego vehicle based on the target action, wherein the target action includes a target lateral position sequence and a target longitudinal speed sequence; wherein the decision model is obtained by training based on multiple scene samples, each of which includes the attribute information of the target vehicle and each road element, and the scene complexity of the driving scene of the target vehicle.

10. A vehicle, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images