Automatic driving scene controllable generation method based on knowledge enhancement

Abstract

The invention relates to an automatic driving scene controllable generation method based on knowledge enhancement, and the method comprises the following steps: building an interactive representation model of traffic participants, carrying out the joint motion prediction of all traffic participants in an automatic driving scene through the interactive representation model, and obtaining the predicted joint motion information; establishing guidance characterization of an automatic driving scene through traffic prior knowledge; and establishing a diffusion model for controllable generation of the automatic driving scene, embedding the guide representation into the diffusion model, and disturbing the joint action information under the constraint of the guide representation to generate a diversified automatic driving scene meeting actual requirements. Compared with the prior art, the method has the advantages of high authenticity, high controllability, high safety, high real-time performance and the like.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for controllable generation of autonomous driving scenarios based on knowledge enhancement. Background Art

[0002] High-fidelity autonomous driving scenarios play a vital role in the verification and development of autonomous driving systems. Considering the high cost of actual road testing and the difficulty in covering edge scenarios with long-tail distributions, how to use scenario generation methods to expand the existing scenario library is an urgent problem to be solved. However, existing rule-based or data-driven scenario generation methods are difficult to personalize, and their coverage of certain long-tail scenarios is obviously insufficient, which also makes it an important challenge to generate high-fidelity and controllable safety-critical scenarios. Therefore, this patent proposes a general controllable scenario generation method for autonomous driving enhanced by driving knowledge, covering typical driving scenarios and adversarial driving scenarios, while ensuring the quality of the generated scenes and improving the controllability of the scenes to meet the diverse needs of actual autonomous driving.

[0003] Although existing scene generation methods can achieve unlimited scene expansion, not all scenes will appear in reality and conform to actual driving constraints. Therefore, how to improve the authenticity and controllability of generated scenes based on driving prior knowledge of driving environment and traffic rules is a major problem in the current scene generation field. Existing typical generation methods such as data-driven generation or adversarial optimization generation ignore the impact of driving prior knowledge on the authenticity of generated scenes, and are prone to generate driving scenes that do not conform to reality, which has a negative impact on subsequent algorithm development. How to efficiently use driving prior knowledge to design a general scene guidance representation to guide the high-fidelity generation of scenes is also an urgent problem to be solved in current research. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a controllable generation method of autonomous driving scenes based on knowledge enhancement.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for controlling the generation of autonomous driving scenarios based on knowledge enhancement includes the following steps:

[0007] S1: Establish an interactive representation model of traffic participants, and use the interactive representation model to predict the joint motion of each traffic participant in the autonomous driving scenario to obtain the predicted joint action information;

[0008] S2: Building guided representations of autonomous driving scenarios through traffic prior knowledge;

[0009] S3: Establish a diffusion model for controllable generation of autonomous driving scenarios, embed the guided representation into the diffusion model, and perturb the joint action information under the constraint of the guided representation to obtain the autonomous driving scenario.

[0010] Furthermore, in step S1, the loss function for joint action prediction of the training interaction representation model is:

[0011]

[0012] In the formula, is the loss function value, c is the scene context information, k is the noise level, is the noise action, and u is the agent action.

[0013] Furthermore, step S1 includes the following steps:

[0014] S11: Establish an interactive representation model of traffic participants, which includes the relative distance, relative heading and relative speed between traffic participants;

[0015] S12: Using the factorization attention mechanism, the agent's historical position information, map encoding, and relative position encoding are sequentially embedded into the interaction representation model to obtain the scene context encoding;

[0016] S13: De-noising the scene context coding, and using the denoised scene context coding to perform joint motion prediction to obtain predicted joint motion information.

[0017] Furthermore, the process of denoising the scene context coding is performed by a motion distribution estimation module. For a K-step diffusion model, the motion distribution estimation module skips the denoising step before the τ-th step of the diffusion model by directly predicting the prior distribution of the τ-th step.

[0018] Furthermore, the calculation expression of the motion distribution estimation module is:

[0019] μ mpe,θ =f μ (x,c)

[0020] σ mpe,θ =f σ (x,c)

[0021]

[0022] In the formula, f μ , f σ and are all trainable modules composed of multiple layers of MLP, μ mpe,θ With σ mpe,θ They correspond to the predicted mean and standard deviation of the τth denoising step, respectively, is the normalized value of the predicted sample, is the motion distribution before the τth denoising step;

[0023] During the training process of the motion distribution estimation module, the loss function is:

[0024]

[0025] In the formula, ω is the weight parameter, is the loss function value of the interaction representation model, is the result of normalizing the variance estimate by the uncertainty loss, σ mpe,θ is the variance estimate.

[0026] Furthermore, the guided representation includes constraint conditions and adversarial representation. The constraint conditions include behavior constraints, collision constraints, and scene constraints. The calculation expression of the behavior constraint is:

[0027]

[0028] In the formula, To constrain behavior, represents the target state or action of the agent, It indicates the current state or action. represents the smoothed L1 loss;

[0029] The calculation expression of the collision constraint is:

[0030]

[0031] In the formula, is the collision constraint, d ij is the relative distance between agents i and j at time t, ε d is the potential collision distance threshold, ε d is the collision threshold;

[0032] The calculation expression of the scene constraint is:

[0033]

[0034] In the formula, is the scene constraint, d offroad represents the distance between agent i and the road boundary at time t, ε r Indicates the distance threshold to the road boundary;

[0035] The computational expression of adversarial representation is:

[0036]

[0037] In the formula, For adversarial representation, is the relative distance between the adversarial agent and the self-agent, is the distance between the agent and the road boundary.

[0038] Further, step S3 includes the following steps:

[0039] S31: establishing a diffusion model for controllable generation of autonomous driving scenarios, wherein the input of the diffusion model is the predicted joint action information;

[0040] S32: Using the guided representation as a constraint, perturbing the joint action information, and generating an autonomous driving scenario that satisfies the constraint.

[0041] Furthermore, the specific process of generating an autonomous driving scenario that meets the constraints is as follows:

[0042] Using map information as well as agent action and state information, the scene encoder is used to realize agent interaction representation, and combined with the actions sampled from the initial noise action distribution, the denoising model is used to realize joint motion prediction. In the scene reasoning stage, according to the constraints of the guided representation, the mean of the distribution of the joint motion prediction is gradient perturbed, and the perturbed agent motion distribution is controllably sampled to generate a driving scenario that meets the constraints of the guided representation.

[0043] Furthermore, the diffusion model includes a self-agent and an adversarial agent, and the guided representation includes a constraint condition and an adversarial representation. When the guided representation of the diffusion model is a constraint condition, the diffusion model uses a traversal gradient descent method to generate autonomous driving scene samples; when the guided representation of the diffusion model is an adversarial representation, the self-agent uses a traversal gradient descent method to generate autonomous driving scene samples, and the adversarial agent uses a traversal gradient ascent method to generate autonomous driving scene samples.

[0044] Furthermore, the generation results of the autonomous driving scenario are evaluated by the controllability index and the authenticity index. The calculation expression of the controllability index is:

[0045] r no-collision :=Σ i≠j 1[||(x i,t ,y i,t )-(x j,t ,y j,t )||≤ε d ]

[0046] In the formula, r no-collision is the controllability index value, x is the horizontal coordinate of the intelligent agent, y is the vertical coordinate of the intelligent agent, t is the time, i and j are the intelligent agent numbers, ε d is the collision threshold;

[0047] The calculation expression of the authenticity index is:

[0048]

[0049] Where Real is the authenticity index value, Wass() is the Wasserstein distance between data distributions, n is the total number of agents, and ρ is the agent data distribution.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1) The present invention realizes the joint motion prediction of multiple traffic participants by integrating the spatiotemporal coupling interactive features based on the factor decomposition attention mechanism. It combines the guided representation based on prior knowledge to constrain the generation process of driving scenes, guide the disturbance process of the diffusion model, generate driving scenes that meet actual needs, and ensure the authenticity, safety and controllability of the generated results.

[0052] 2) The present invention predicts the prior distribution of the denoising step through the motion distribution estimation module and skips some denoising steps, thereby accelerating the denoising process, reducing the inference time of the diffusion model and improving real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is the overall framework diagram of the present invention.

[0054] Figure 2 Schematic diagram of the factor decomposition attention mechanism in the present invention.

[0055] Figure 3 Flowchart of the knowledge-guided controllable diffusion sampling algorithm.

[0056] Figure 4 Visualize the results for a controllable driving scenario.

[0057] Figure 5 Visualizing results for adversarial driving scenarios. DETAILED DESCRIPTION

[0058] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0059] Example 1

[0060] The present invention is a method for controlling the generation of autonomous driving scenarios based on knowledge enhancement, comprising the following steps:

[0061] S1: Establish an interactive representation model of traffic participants, and use the interactive representation model to predict the joint motion of each traffic participant in the autonomous driving scenario to obtain the predicted joint action information;

[0062] The scene-level joint motion prediction of multiple traffic participants is mainly implemented based on the encoder-decoder architecture, in which the scene context encoder obtains the potential embedding representation of the agent's historical state and map representation through a factorized attention mechanism, and uses a conditional denoiser to perform joint motion decoding based on the scene context embedding from the scene context encoder to obtain the motion prediction trajectory. Figure 1 shown.

[0063] S11: Establish an interactive representation model of traffic participants, which includes the relative distance, relative heading and relative speed between traffic participants;

[0064] The input of the scene context encoder includes the state s of the agent t =[x t ,y t ,v t ,θ t ], action information And the scene context feature c. Among them, the state information includes the 2-D position coordinates x of the agent t ,y t , current driving speed v t and the heading angle θ t The action information includes the current acceleration of the agent and angular velocity Context features include the historical status information of all traffic participants in the scene Road map information and traffic participant types. H represents the number of historical time steps, and A represents the number of traffic participants in the scene. A VectorNet-based map encoder is used to process the polyline tensor and extract structured road map topology information.

[0065] m emb =PolylineEncoder(v i ) (1)

[0066] Among them, v i Represents map vector information.

[0067] Next, the relative embedding representation is used to model the interaction relationship between traffic participants, including relative distance rel_d, relative heading rel_θ, and relative speed rel_v. The state tensor of agent i at time t can be expressed as represents the position vector of the agent. Therefore, the relative representation of the interaction information between agents can be expressed as:

[0068]

[0069]

[0070] Based on the above relative information representation, RPE can reconstruct the absolute position of another agent from the information of one agent and provide the spatiotemporal information representation of scene elements. The interaction edge embedding representation is obtained through a multi-layer perceptron (MLP). ij :

[0071]

[0072] in, They represent the horizontal and vertical position deviations between agent j at time t and agent i at the current time, Represents the rotation matrix of agent i at the current moment.

[0073] S12: Using the factorization attention mechanism, the agent's historical position information, map encoding, and relative position encoding are sequentially embedded into the interaction representation model to obtain the scene context encoding;

[0074] The factorized attention mechanism is used to extract and fuse the spatiotemporal features of the historical status of traffic participants and map information. The temporal attention, agent-map attention and agent-agent attention are stacked in the encoder module and the above module L is repeated. enc Next, we implement fusion scene context encoding, such as Figure 2 shown.

[0075] First, the temporal attention mechanism is used to extract the cross-temporal dynamic information of the historical state, and the multi-head self-attention mechanism (MHSA) layer is used to obtain the temporal embedding, and the agent history processed by the position encoding module PE() is embedded into a i As query q, key k, value v, the entire temporal attention calculation process can be expressed as:

[0076] F temp [i] = MHSA (q = k = v = PE (a i )) (6)

[0077] Among them, a time mask is used to shield the potential impact of future time steps on the prediction.

[0078] Next, Multi-Head Cross Attention (MHCA) is used to extract the interaction representation between the agent and the map, and the updated map encoding will be used for subsequent trajectory decoding. Here, the output of the previous layer is used as the query, and the map embedding representation m output by the map encoder is used as the query. emb As a key and value vector, the specific calculation process can be expressed as:

[0079] F M2A =MHCA(q=F temp ,k=v=m emb ) (7)

[0080] Finally, we use the MHCA layer to perform agent interaction attention, where the keys and values ​​come from the relative position embedding encoding. ij , an interaction mask is applied here to consider only the neighbor sets within the interaction radius The specific calculation process can be expressed as:

[0081]

[0082] S13: denoising the scene context code, and using the denoised scene context code to perform joint motion prediction to obtain predicted joint motion information;

[0083] The scene context encoding C obtained in step S12 takes the noise motion trajectory and the noise level as input and stacks L dec The decoding module predicts the future actions of traffic participants. The noise level k is encoded by sinusoidal position to obtain the corresponding noise embedding, and the dynamic model is used to convert the noise action into the noise state: Where s0 represents the initial state, represents the noise action, and uses MLP to get the noise state embedding. The overall input can be expressed as c is the scene context feature. In the decoding process, the self-attention mechanism is first used to extract temporal features, and the temporal mask matrix M is used tmp Ensure that future time information is not used at the current time, specifically expressed as:

[0084] X tmp =MHSA(q=k=v=x,M tmp ) (9)

[0085] Next, a cross-attention layer is used to decode the interaction representation and encode the relative embedding e ij Integrate into the cross attention layer as the key and value, and the output of the previous layer as the query. The specific calculation process can be expressed as:

[0086]

[0087] Finally, the state embedding obtained above is used as the input of the multi-layer MLP to decode the predicted joint action information. The overall scene generation framework is divided into two stages. The first stage of training is based on the idea of ​​diffusion model to realize the joint motion prediction of multiple traffic participants. Specifically, in the training stage, the scene context information c and the real initial action u0 are collected from the real data set, and the noise is gradually increased by the forward noise addition process to obtain the noise action. The conditional noise decoder performs the denoising process of the diffusion model to obtain the predicted action Both actions and states are supervised by the loss function during training to improve the quality of generated samples. The loss function of the overall motion prediction training process is It can be expressed as:

[0088]

[0089] S2: Building guided representations of autonomous driving scenarios through traffic prior knowledge;

[0090] By taking driving knowledge or traffic environment constraints as priors to design a general scenario guidance representation, it is specifically divided into controllable driving scenarios and adversarial driving scenario guidance. The controllable driving scenario guidance aims to guide the agent to generate corresponding target actions or driving behaviors based on the scenario constraints predefined by driving knowledge, such as driving at a specified speed or driving to a specified location. According to the idea of ​​signal timing logic, it can be expressed as a differentiable cost function, specifically expressed as:

[0091]

[0092] in, represents the target state or action of the agent, It indicates the current state or action. represents the smoothed L1 loss.

[0093] Controllability scenario guidance can also be described based on constraints, such as avoiding collisions or avoiding leaving the drivable area. The collision avoidance driving guidance can be expressed as:

[0094]

[0095] Among them, d ij The relative distance between agents i and j at time t is calculated, ε d is the potential collision distance threshold. If the distance between agents is less than the threshold ε d It is considered that a collision has occurred. The scenario guidance to avoid leaving the drivable area can be expressed as:

[0096]

[0097] Among them, d offroad represents the distance between agent i and the road boundary at time t, ε r Indicates the distance threshold to the road boundary.

[0098] In addition to controllable driving scenario guidance, it can also be extended to safety-critical scenarios that are difficult to cover in the original data set under the constraints of driving knowledge. It can be regarded as an extension of controllable driving scenario guidance. Consider a pair of traffic participants in the scenario, divided into ego agents and adversarial agents. The goal of the former is to avoid collisions and leave the drivable area, while the adversarial agent encourages some relatively aggressive driving behaviors and potential conflicts. According to the idea of ​​signal timing logic, based on the relative states and actions of the agents, a general representation of adversarial safety-critical scenarios is designed, which can be expressed as:

[0099]

[0100] in, The relative distance between the adversarial agent and the self-agent is calculated. The distance between the agent and the road boundary is calculated to encourage risky behaviors such as collision or leaving the drivable area.

[0101] The general scenario representation based on driving knowledge enhancement designed in this step will be embedded in the scenario generation framework in the form of a differentiable cost function, guiding the generation of corresponding driving scenarios that meet the constraints in the scenario generation stage.

[0102] S3: Establish a diffusion model for controllable generation of autonomous driving scenarios, embed the guided representation into the diffusion model, and perturb the joint action information under the constraint of the guided representation to obtain the autonomous driving scenario;

[0103] S31: establishing a diffusion model for controllable generation of autonomous driving scenarios, wherein the input of the diffusion model is the predicted joint action information;

[0104] The generation of autonomous driving scenarios can be formulated as an optimization problem as follows:

[0105]

[0106] Where f represents the vehicle's dynamic model, s, u and c represent the agent state, action and context information respectively. Scenario generation can be viewed as solving the optimal action that satisfies the constraints of real driving behavior based on the objective function. Given the context information c and the agent action u, the probability density of the generated scenario can be expressed as p(S) = p(u|c)p(c). Where p(u|c) can be approximated as:

[0107]

[0108] In the formula, Z θ is the partition function considered as a constant. The above problem can be trained by maximizing the conditional log-likelihood of the dataset:

[0109]

[0110] Score matching can directly learn to optimize the object J θ The gradient of as a score function:

[0111]

[0112] However, due to the non-optimal scenarios in the action space, the gradient estimation in the non-optimal area will lead to significant deviations. Therefore, the diffusion model uses a random process to transform the data distribution p to the known noise distribution p by adding k steps of noise. k , transform the above problem into training a score function The gradient of the specific approximate logarithmic noise distribution can be expressed as:

[0113]

[0114] Diffusion-based scene generation can be solved by learning the gradient of trajectory optimization and sampling the optimal control sequence through back diffusion, that is, stochastic gradient descent along the diffusion step to a fixed high probability area. Controllable generation uses guided g to guide the generation of scenes that meet driving constraints without retraining the model, similar to classifier guided diffusion. The optimization problem of controlled diffusion generation can be corrected as:

[0115]

[0116] Therefore, the reverse diffusion process of the diffusion model can be expressed as:

[0117]

[0118] in, represents the probability of the target taking a given action at a noise step length k. It is not equivalent to p(u|c,g), and a practical approximation using the gradient likelihood method with respect to noise is used to achieve a controllable guided combination.

[0119] S32: Using the guided representation as a constraint, perturbing the joint action information, and generating an autonomous driving scenario that satisfies the constraint.

[0120] Using map information as well as agent action and state information, the scene encoder is used to realize agent interaction representation, and combined with the actions sampled from the initial noise action distribution, the denoising model is used to realize joint motion prediction. In the scene reasoning stage, according to the constraints of the guided representation, the mean of the distribution of the joint motion prediction is gradient perturbed, and the perturbed agent motion distribution is controllably sampled to generate a driving scenario that meets the constraints of the guided representation.

[0121] Given the k-th diffusion step action trajectory u k , the forward noise addition process of the Denoising Diffusion Probabilistic Model (DDPM) can be expressed as:

[0122]

[0123] Among them, β k ∈(0,1) represents a predefined noise schedule and controls the noise scale injected in each diffusion step. u is the initial noise-free action, u k The k-th step of the noise addition process, after K steps of forward noise addition process, the data distribution gradually tends to the isotropic Gaussian distribution The calculation process of relevant parameters is expressed as:

[0124] α k =1-β k (25)

[0125]

[0126] In the formula, It can be defined as the cosine noise schedule, β k It can be expressed as:

[0127]

[0128] According to the original action trajectory u, the noise action can be calculated by forward diffusion as follows:

[0129]

[0130] Using the given context information c to learn the conditional denoising model, the clean action information generated by iterative prediction of the sampled noise can be expressed as:

[0131]

[0132] in, is a Gaussian noise distribution, μ θ is the posterior mean of the predicted k-1th step action, σ kis the standard deviation based on fixed noise scheduling, D θ The prediction result of the denoiser represented by , the mean can be calculated by predicting the denoising action and the noise action, and the calculation expression of the mean is:

[0133]

[0134] Next, in order to generate controllable driving scenarios that meet actual personalized needs, the cost function of the knowledge-enhanced general scenario guidance designed in step S2 is embedded in the controllable diffusion framework. The controllable scenario generation problem can be regarded as the optimal trajectory distribution sampling based on knowledge guidance g, and the controllable samples are realized by perturbing the model prediction mean in each reverse denoising step, which is specifically expressed as:

[0135]

[0136] Aiming at the controllable generation problem of driving scenes, a knowledge-guided general scene controllable diffusion sampling method is proposed. The gradient descent method is used to guide sample generation. The detailed calculation process is as follows: Figure 3 As shown, the specific calculation process includes the following:

[0137] The input of the algorithm includes: denoiser model D θ , scene guided cost function J g , diffusion steps K, gradient steps N g , scale parameter α, standard deviation σ k , ego agent number e, adversarial agent number a, gradient descent steps N d , gradient ascent steps N a .

[0138] The output result is the guided denoising action u.

[0139] First, sample from the initial noisy motion trajectory:

[0140]

[0141] Traverse the reverse denoising process from K→1 and calculate the denoising prediction action:

[0142]

[0143] The mean can be calculated by predicting the denoising action and the noise action, i.e., formula (31).

[0144] Next, different gradient calculation methods are selected according to the type of guidance scene. If it is a controllable scene guidance, the gradient descent method is used. The specific calculation is to traverse the gradient steps and calculate the predicted mean after disturbance according to formula (32).

[0145] When the scene guidance is adversarial guidance, the gradient ascent under the original sampling framework is modified to gradient ascent-descent. The modified gradient calculation can be expressed as:

[0146] For the self-agent, traverse the gradient descent steps 1→N d :

[0147]

[0148] For adversarial agents, the gradient ascent steps 1→N are repeated a , encouraging the adversarial agent to make driving actions that violate the rules:

[0149]

[0150] After obtaining the perturbed predicted mean, the predicted distribution is sampled to obtain the denoising action:

[0151]

[0152] Based on the above gradient calculation method, the denoising action after disturbance can be obtained, that is, the action that satisfies the constraints of the corresponding scene cost function, so as to achieve the purpose of realizing the controllability of the driving scene.

[0153] Preferably, in order to solve the problem that the standard diffusion model has a slow reasoning speed that affects real-time performance, the present invention adds a learnable motion distribution estimation module based on the pre-trained denoising model. For a K-step standard diffusion model, the prior distribution of the denoising step τ is directly predicted, and the usual denoising process is performed on the denoising steps after the τth step to obtain clean motion samples. This method can skip the denoising steps before the τth step to accelerate the process.

[0154] By decomposing the motion prior distribution into mean, variance and sample prediction, each part corresponds to a trainable module, with historical state and scene context encoding as input, the calculation process of the entire module can be expressed as:

[0155] μ mpe,θ =f μ (x,c) (38)

[0156] σ mpe,θ =f σ (x,c) (39)

[0157]

[0158] Among them, f μ , f σ and It is composed of three trainable modules consisting of multiple layers of MLP. mpe,θ With σ mpe,θcorresponds to the mean and standard deviation of the τth denoised distribution, which is shared among samples. is the normalized value of the predicted sample. The prior distribution obtained It can be viewed as the motion distribution before the denoising process at step τ. Denoising only the steps after step τ can significantly reduce the running time.

[0159] During this stage of training, only the motion prior distribution estimation module is trained. The denoising prediction is accelerated by freezing the scene context encoding and other modules. The loss function is modified as follows:

[0160]

[0161] Where ω is the weight parameter, the first loss term is used to evaluate the difference between the generated sample and the true value. The second term is the variance estimate σ through the uncertainty loss. mpe,θ Normalization is performed to balance the diversity and accuracy of predictions.

[0162] Scenario performance evaluation:

[0163] The performance of the generated scenario is evaluated from three aspects: controllability, authenticity, and stability. The controllability index is used to measure the degree of violation of the rules. For the controllability index, a specific example is given here. The evaluation index of the controllable driving scenario for collision avoidance is given below:

[0164] r no-collision :=Σ i≠j 1[||(x i,t ,y i,t )-(x j,t ,y j,t )||≤ε d ] (43)

[0165] The authenticity index is used to evaluate the statistical similarity between the generated samples and the true values, which is defined by the average value of the longitudinal acceleration, lateral acceleration and bump value distribution distance, and is specifically expressed as:

[0166]

[0167] Among them, Real is the authenticity indicator, Wass() represents the Wasserstein distance between data distributions, which is used to measure the statistical similarity between data distributions.

[0168] The stability metric is calculated by the average proportion of agents that produce critical failures in a driving scenario. It is worth noting that lower evaluation metrics indicate better performance.

[0169] Simulation experiment results:

[0170] This example is implemented in Ubuntu 20.04 and Pytorch 1.13.1, using RTX-4090 GPU (24GB) for model training and subsequent scene reasoning. The nuScene dataset is used to train the scene diffusion model, and the collision-free guidance and adversarial guidance of the controllable driving scene are selected as examples to verify the effect of scene generation. Figure 4 and Figure 5 Shown are the visualization results of the controllability-guided collision-free driving scenario and the adversarial driving scenario. Figure 4 It can be seen that the generated scene can meet the scene constraint requirements without collision, and Figure 5 In the adversarial traffic, some aggressive behaviors will be performed by the traffic participants, resulting in some adversarial driving scenarios. Table 1 gives the qualitative analysis results of the scenario generation performance. It can be seen that compared with the comparison method, the controllability, authenticity and stability indicators of the generated scenarios are improved by 8.1%, 15.3% and 9.2%, respectively. Table 2 verifies the effectiveness of the acceleration module. It can be seen that when the acceleration module is added, the performance indicators of the generated scenarios can still be guaranteed, but the scenario reasoning speed is increased by 41.5%, which verifies the feasibility of the proposed scenario generation method.

[0171] Table 1 Scene generation performance

[0172]

[0173] Table 2 Acceleration module

[0174]

[0175] The present invention is mainly aimed at a controllable generation method of high-fidelity driving scenes, and its specific advantages mainly include the following aspects. First, an accurate scene-level joint motion prediction model is designed, and the spatiotemporal coupling interactive feature fusion based on the factorized attention mechanism is used to achieve accurate joint motion prediction of multiple traffic participants, providing a reliable foundation for subsequent scene generation. Secondly, a general driving scene guidance representation is designed in combination with driving prior knowledge and traffic constraints to guide the generation of driving scenes that meet actual needs and ensure the authenticity and controllability of the generated scenes. Finally, a controllable diffusion sampling method is designed. Based on a controllable generation framework, the differentiable cost function obtained by the general representation of the scene is used to perform gradient perturbations on the predicted behavior information in the scene generation stage to obtain driving scenes that meet actual needs and meet the personalized needs of real driving.

[0176] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A controllable generation method for autonomous driving scenarios based on knowledge enhancement, characterized in that: The following steps are involved: S1: Establish an interactive representation model of traffic participants, and use the interactive representation model to predict the joint motion of each traffic participant in the autonomous driving scenario to obtain the predicted joint action information; S2: Building guided representations of autonomous driving scenarios through traffic prior knowledge; S3: Establish a diffusion model for controllable generation of autonomous driving scenarios, embed the guided representation into the diffusion model, and perturb the joint action information under the constraint of the guided representation to obtain the autonomous driving scenario.

2. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 1, characterized in that: In step S1, the loss function for joint motion prediction of the training interactive representation model is: In the formula, is the loss function value, c is the scene context information, k is the noise level, is the noise action, and u is the agent action.

3. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 1, characterized in that: Step S1 includes the following steps: S11: establishing an interaction representation model of traffic participants, wherein the interaction representation model includes relative distance, relative heading and relative speed between traffic participants; S12: Using the factorization attention mechanism, the agent's historical position information, map encoding, and relative position encoding are sequentially embedded into the interaction representation model to obtain the scene context encoding; S13: De-noising the scene context coding, and using the denoised scene context coding to perform joint motion prediction to obtain predicted joint motion information.

4. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 3 is characterized in that: The process of denoising the scene context coding is performed by a motion distribution estimation module. For a K-step diffusion model, the motion distribution estimation module skips the denoising step before the τ-th step of the diffusion model by directly predicting the prior distribution of the τ-th step.

5. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 4, characterized in that: The calculation expression of the motion distribution estimation module is: μ mpe,θ =f μ (x,c) s mpe,θ =f σ (x,c) In the formula, f μ , f σ and are all trainable modules composed of multiple layers of MLP, μ mpe,θ With σ mpe,θ They correspond to the predicted mean and standard deviation of the τth denoising step, respectively, is the normalized value of the predicted sample, is the motion distribution before the τth denoising step; During the training process of the motion distribution estimation module, the loss function is: In the formula, ω is the weight parameter, is the loss function value of the interaction representation model, is the result of normalizing the variance estimate by the uncertainty loss, σ mpe,θ is the variance estimate.

6. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 1, characterized in that: The guided representation includes constraint conditions and adversarial representations. The constraint conditions include behavior constraints, collision constraints, and scene constraints. The calculation expression of the behavior constraints is: In the formula, To constrain behavior, represents the target state or action of the agent, It indicates the current state or action. represents the smoothed L1 loss; The calculation expression of the collision constraint is: In the formula, is the collision constraint, d ij is the relative distance between agents i and j at time t, ε d is the potential collision distance threshold, ε d is the collision threshold; The calculation expression of the scene constraint is: In the formula, is the scene constraint, d offroad represents the distance between agent i and the road boundary at time t, ε r Indicates the distance threshold to the road boundary; The computational expression of the adversarial representation is: In the formula, For adversarial representation, is the relative distance between the adversarial agent and the self-agent, is the distance between the agent and the road boundary.

7. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 1, characterized in that: Step S3 includes the following steps: S31: establishing a diffusion model for controllable generation of autonomous driving scenarios, wherein the input of the diffusion model is predicted joint action information; S32: Using the guided representation as a constraint, perturbing the joint action information, and generating an autonomous driving scenario that satisfies the constraint.

8. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 7, characterized in that: The specific process of generating an autonomous driving scenario that meets the constraints is as follows: Using map information as well as agent action and state information, the scene encoder is used to realize agent interaction representation, and combined with the actions sampled from the initial noise action distribution, the denoising model is used to realize joint motion prediction. In the scene reasoning stage, according to the constraints of the guided representation, the mean of the distribution of the joint motion prediction is gradient perturbed, and the perturbed agent motion distribution is controllably sampled to generate a driving scenario that meets the constraints of the guided representation.

9. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 8 is characterized in that: The diffusion model includes a self-agent and an adversarial agent, and the guided representation includes a constraint condition and an adversarial representation. When the guided representation of the diffusion model is a constraint condition, the diffusion model uses a traversal gradient descent method to generate autonomous driving scene samples; when the guided representation of the diffusion model is an adversarial representation, the self-agent uses a traversal gradient descent method to generate autonomous driving scene samples, and the adversarial agent uses a traversal gradient ascent method to generate autonomous driving scene samples.

10. The method for controlling the generation of autonomous driving scenarios based on knowledge enhancement according to claim 1, characterized in that: The generation result of the autonomous driving scenario is evaluated by the controllability index and the authenticity index. The calculation expression of the controllability index is: r no-collision :=Σ i≠j 1[||(x i,t ,and i,t )-(x j,t ,and j,t )||≤ε d ] In the formula, r no-collision is the controllability index value, x is the horizontal coordinate of the intelligent agent, y is the vertical coordinate of the intelligent agent, t is the time, i and j are the intelligent agent numbers, ε d is the collision threshold; The calculation expression of the authenticity index is: Where Real is the authenticity index value, Wass() is the Wasserstein distance between data distributions, n is the total number of agents, and ρ is the agent data distribution.

Images