A knowledge enhancement-based controllable generation method for an autonomous driving scene

Abstract

The application relates to a knowledge enhancement-based controllable generation method for an automatic driving scene, comprising the following steps: establishing an interactive representation model of a traffic participant, performing joint motion prediction on each traffic participant in the automatic driving scene through the interactive representation model to obtain predicted joint action information; establishing a guide representation of the automatic driving scene through traffic prior knowledge; establishing a diffusion model for controllable generation of the automatic driving scene, embedding the guide representation into the diffusion model, and generating diversified automatic driving scenes meeting actual requirements by perturbing the joint action information under the constraint of the guide representation. Compared with the prior art, the application has the advantages of strong authenticity, strong controllability, high safety, strong real-time performance and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a controllable automatic driving scene generation method based on knowledge enhancement. BACKGROUND

[0002] High-fidelity automatic driving scenes are crucial for the verification and development of automatic driving systems. Considering the high cost and difficulty of covering long-tail distribution edge scenes in actual road testing, how to expand the existing scene library using scene generation methods is a problem to be solved. However, existing rule-based or data-driven scene generation methods are difficult to achieve personalized settings, and their coverage of some long-tail scenes is obviously insufficient, which also makes it an important challenge to generate high-fidelity controllable safety-critical scenes. Therefore, the present application proposes a driving knowledge enhanced automatic driving general controllable scene generation method, which covers typical driving scenes and adversarial driving scenes, while ensuring the quality of the generated scenes and improving the controllability of the scenes to meet the diversified needs of actual automatic driving.

[0003] Although existing scene generation methods can achieve infinite scene expansion, not all scenes will occur in reality, and they conform to actual driving constraints. Therefore, how to improve the authenticity and controllability of generated scenes based on driving environment and traffic rule driving prior knowledge is a major problem in the current scene generation field. Existing typical generation methods such as data-driven generation or adversarial optimization generation ignore the influence of driving prior knowledge on the authenticity of generated scenes, which can easily generate driving scenes that do not conform to reality, thereby negatively affecting subsequent algorithm development. How to efficiently utilize driving prior knowledge to design general scene guide representations to guide high-fidelity scene generation is also a problem to be solved in current research. SUMMARY

[0004] The purpose of the present application is to overcome the defects of the prior art and provide a controllable automatic driving scene generation method based on knowledge enhancement.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] A controllable automatic driving scene generation method based on knowledge enhancement, comprising the following steps:

[0007] S1: Establish an interactive representation model of traffic participants, and perform joint motion prediction on each traffic participant in the automatic driving scene through the interactive representation model to obtain predicted joint action information;

[0008] S2: Establish a guide representation of the automatic driving scene through traffic prior knowledge;

[0009] S3: Establish a diffusion model for controllable generation of autonomous driving scenes, embed the guidance representation into the diffusion model, perturb the joint action information under the constraint of the guidance representation, and obtain the autonomous driving scene.

[0010] Further, in step S1, the loss function for training the joint action prediction of the interaction representation model is:

[0011]

[0012] In the formula, is the loss function value, c is the scene context information, k is the noise level, is the noise action, and u is the agent action.

[0013] Further, step S1 includes the following steps:

[0014] S11: Establish an interaction representation model of traffic participants, which includes the relative distance, relative heading and relative speed between traffic participants;

[0015] S12: Embed the agent historical position information, map encoding and relative position encoding into the interaction representation model in turn using the factorization attention mechanism to obtain the scene context encoding;

[0016] S13: Denoise the scene context encoding, and use the denoised scene context encoding for joint motion prediction to obtain the predicted joint action information.

[0017] Further, the process of denoising the scene context encoding is performed by a motion distribution estimation module. For a K-step diffusion model, the motion distribution estimation module directly predicts the prior distribution of the τth step, skipping the denoising step before the τth step of the diffusion model.

[0018] Further, the calculation expression of the motion distribution estimation module is:

[0019] μ mpe,θ =f μ (x,c)

[0020] σ mpe,θ =f σ (x,c)

[0021]

[0022] In the formula, f μ , f σ and are trainable modules composed of multiple MLPs, μ mpe,θ and σ mpe,θ correspond to the predicted mean and standard deviation of the τth denoising step, respectively, is a normalized value of the prediction sample, is the motion distribution before the de-noising step of the τ-th step;

[0023] During the training process, the loss function of the motion distribution estimation module is:

[0024]

[0025] where ω is a weight parameter, is the loss function value of the interaction representation model, is the result of normalizing the variance estimation by the uncertainty loss, σ mpe,θ is the variance estimation.

[0026] Further, the guiding representation includes constraint conditions and adversarial representation, the constraint conditions include behavior constraints, collision constraints and scene constraints, and the calculation expression of the behavior constraints is:

[0027]

[0028] wherein, is the behavior constraint, denotes the target state or action of the agent, denotes the state or action at the current time, denotes a smooth L1 loss;

[0029] The calculation expression of the collision constraints is:

[0030]

[0031] wherein, is the collision constraint, d ij is the relative distance between the agent i and j at time t, ε d is a potential collision distance threshold, ε d is a collision threshold;

[0032] The calculation expression of the scene constraints is:

[0033]

[0034] wherein, is the scene constraint, d offroad denotes the distance between the agent i and the road boundary at time t, ε r denotes a distance threshold from the road boundary;

[0035] The calculation expression of the adversarial representation is:

[0036]

[0037] wherein, For adversarial representation, is the relative distance between the adversarial agent and the ego agent, is the distance between the agent and the road boundary.

[0038] Furthermore, step S3 includes the following steps:

[0039] S31: Establish a diffusion model for controllable generation of autonomous driving scenarios, where the input of the diffusion model is the predicted joint action information;

[0040] S32: Using the guided representation as a constraint, the joint action information is perturbed to generate an autonomous driving scenario that satisfies the constraint.

[0041] Furthermore, the specific process of generating an autonomous driving scenario that meets the constraints is as follows:

[0042] By utilizing map information as well as agent action and state information, a scene encoder is used to realize agent interaction representation. Combined with the actions sampled from the initial noisy action distribution, a denoising model is used to realize joint motion prediction. In the scene reasoning stage, according to the constraints of the guided representation, the mean of the distribution of the joint motion prediction is gradient perturbed, and the perturbed agent motion distribution is controllably sampled to generate a driving scenario that meets the constraints of the guided representation.

[0043] Furthermore, the diffusion model includes a self-agent and an adversarial agent, and the guided representation includes a constraint condition and an adversarial representation. When the guided representation of the diffusion model is a constraint condition, the diffusion model uses the ergodic gradient descent method to generate autonomous driving scene samples; when the guided representation of the diffusion model is an adversarial representation, the self-agent uses the ergodic gradient descent method to generate autonomous driving scene samples, and the adversarial agent uses the ergodic gradient ascent method to generate autonomous driving scene samples.

[0044] Furthermore, the generated results of the autonomous driving scenario are evaluated by the controllability index and the authenticity index. The calculation expression of the controllability index is:

[0045] r no-collision :=Σ i≠j 1[||(x i,t ,y i,t )-(x j,t ,y j,t )||≤ε d ]

[0046] Where r no-collision is the controllability index value, x is the horizontal coordinate of the agent, y is the vertical coordinate of the agent, t is the time, i and j are the agent numbers, ε d is the collision threshold;

[0047] The calculation expression of the authenticity index is:

[0048]

[0049] In the formula, Real is the authenticity index value, Wass() is the Wasserstein distance between data distributions, n is the total number of agents, and p is the agent data distribution.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] 1) The present application realizes multi-traffic participant joint motion prediction based on spatio-temporal coupling interaction feature fusion of factor decomposition attention mechanism, combines with prior knowledge based guidance representation to constrain the generation process of the driving scene, guides the perturbation process of the diffusion model, generates a driving scene that meets the actual demand, and guarantees the authenticity, safety and controllability of the generated result.

[0052] 2) The present application predicts the prior distribution of the denoising step through the motion distribution estimation module, skips part of the denoising step, thereby accelerating the denoising process, reducing the inference time of the diffusion model, and improving the real-time performance. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is the overall framework diagram of the present application.

[0054] Figure 2 is a schematic diagram of the factor decomposition attention mechanism in the present application.

[0055] Figure 3 is a flowchart of the knowledge guided controllable diffusion sampling algorithm.

[0056] Figure 4 is a controllable driving scene visualization result.

[0057] Figure 5 is an adversarial driving scene visualization result. DETAILED DESCRIPTION

[0058] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present application, and gives a detailed implementation manner and specific operation process, but the protection scope of the present application is not limited to the following examples.

[0059] Example 1

[0060] The present application is a knowledge enhanced automatic driving scene controllable generation method, comprising the following steps:

[0061] S1: Establish an interactive representation model of traffic participants, and use the interactive representation model to predict the joint motion of each traffic participant in the autonomous driving scenario to obtain the predicted joint action information;

[0062] The scene-level joint motion prediction of multiple traffic participants is mainly implemented based on the encoder-decoder architecture, in which the scene context encoder obtains the potential embedding representation of the agent's historical state and map representation through the factorization attention mechanism, and uses the conditional denoiser to perform joint motion decoding based on the scene context embedding from the scene context encoder to obtain the motion prediction trajectory. Figure 1 shown.

[0063] S11: Establish an interaction representation model of traffic participants, which includes the relative distance, relative heading, and relative speed between traffic participants;

[0064] The input of the scene context encoder includes the state s of the agent t =[x t ,y t ,v t ,θ t ], action information And the scene context feature c. Among them, the state information includes the 2-D position coordinates x of the intelligent body t ,y t , current driving speed v t and the heading angle θ t The action information includes the current acceleration of the agent and angular velocity Contextual features include historical status information of all traffic participants in the scene Road map information and traffic participant types. H represents the number of historical time steps, and A represents the number of traffic participants in the scene. A VectorNet-based map encoder is used to process the polyline tensor and extract structured road map topology information.

[0065] m emb =PolylineEncoder(v i ) (1)

[0066] Among them, v i Represents map vector information.

[0067] Next, the relative embedding representation is used to model the interaction relationship between traffic participants, including relative distance rel_d, relative heading rel_θ, and relative speed rel_v. The state tensor of agent i at time t can be expressed as Position vector representing the position of an agent. Thus, the relative representation of the interaction information between agents can be represented as:

[0068]

[0069]

[0070] Based on the above relative information representation RPE, the absolute position of another agent can be reconstructed from the information of one agent and the spatiotemporal information representation of the scene elements is provided. The interaction edge embedding representation e ij :

[0071]

[0072] where, denote the lateral and longitudinal position deviation between agent j at time t and agent i at the current time, respectively, denotes the rotation matrix of agent i at the current time.

[0073] S12: Adopting the factorization attention mechanism, sequentially embed the historical position information of the agent, the map encoding and the relative position encoding into the interaction representation model to obtain the scene context encoding;

[0074] The factorization attention mechanism is used to extract and fuse the spatiotemporal features of the historical state of the traffic participant and the map information. By stacking time attention, agent-map attention and agent-agent attention in the encoder module, and repeating the above module L enc times, the fused scene context encoding is realized, as shown in Figure 2

[0075] Firstly, the cross-time dynamic information of the historical state is extracted using the time attention mechanism. A multi-head self-attention mechanism (MHSA) layer is used to obtain the time embedding. The agent historical embedding a i is processed by the position encoding module PE(), and is taken as the query q, the key k and the value v. The whole time attention calculation process can be represented as:

[0076] F temp [i]=MHSA(q=k=v=PE(a i )) (6)

[0077] Wherein, a time mask is used to mask the potential influence of future time steps on prediction.

[0078] ​Next, the multi-head cross attention (MHCA) is used to extract the interaction representation between the agent and the map, and the updated map encoding will be used for subsequent trajectory decoding. Here, the output of the previous layer is taken as the query, and the map embedding table m emb As a key and value vector, the specific calculation process can be represented as:

[0079] F M2A = MHCA(q=F temp , k=v=m emb ) (7)

[0080] Finally, the agent interaction attention based on the MHCA layer, where the key and value come from the relative position embedding encoding e ij Here, an interaction mask is applied to only consider the neighbor set of the agent within the interaction radius The specific calculation process can be represented as:

[0081]

[0082] S13: Denoising the scene context encoding, using the denoised scene context encoding for joint motion prediction to obtain predicted joint action information;

[0083] The scene context encoding C obtained by step S12 is taken as input, and the noise action trajectory and noise level are predicted through L dec noise embedding is obtained by sinusoidal position encoding, and the noise action is converted into noise state by using the dynamics model: where s0 represents the initial state, represents the noise action, and the noise state embedding is obtained by using the MLP. The overall input can be represented as c is the scene context feature. In the decoding process, the self-attention mechanism is first used for time feature extraction, and the time mask matrix M tmp ensures that future time information is not used at the current time, and the specific representation is:

[0084] X tmp = MHSA(q=k=v=x, M tmp ) (9)

[0085] Next, the interaction representation is decoded by a cross-attention layer, and the relative embedding encoding e ij is integrated into the cross-attention layer as the key and value, and the output of the previous layer is taken as the query. The specific calculation process can be represented as:

[0086]

[0087] Finally, the state obtained above is embedded as the input of a multi-layer MLP to decode the predicted joint action information. The overall scene generation framework is divided into two stages. The first stage is to realize joint multi-traffic participant motion prediction based on the idea of diffusion model. Specifically, in the training stage, the scene context information c and the real initial action u0 are collected from the real data set. The forward noise adding process is used to gradually increase the noise to obtain the noisy action and the diffusion model denoising process is performed through the conditional noise decoder to obtain the predicted action The action and state are supervised by the loss function during training to improve the quality of the generated samples. The loss function of the overall motion prediction stage training process can be expressed as:

[0088]

[0089] S2: Establish the guidance representation of the autonomous driving scene through traffic prior knowledge;

[0090] By using driving knowledge or traffic environment constraints as priori to design a general scene guidance representation, it is specifically divided into controllable driving scene guidance and adversarial driving scene guidance. The controllable driving scene guidance aims to guide the agent to generate the corresponding target action or driving behavior according to the scene constraints predefined by the driving knowledge, such as driving at a specified speed or driving to a specified location. According to the idea of signal timing logic, it can be represented as a differentiable cost function, which is specifically expressed as:

[0091]

[0092] wherein, represents the target state or action of the agent, represents the state or action at the current time, represents a smooth L1 loss.

[0093] The controllable scene guidance can also be based on constraint conditions, such as avoiding collision or avoiding driving out of the drivable area. The driving guidance to avoid collision can be expressed as:

[0094]

[0095] wherein, d ij is the relative distance between agent i and j at time t, and ε d is the potential collision distance threshold. If the distance between the agents is less than the threshold ε d , it is considered as a collision. The scene guidance to avoid driving out of the drivable area can be expressed as:

[0096]

[0097] where d offroad denotes the distance between the agent i and the road boundary at time t, ε r denotes the distance threshold to the road boundary.

[0098] In addition to the controllable driving scene guidance, it can also be extended to the safety-critical scene that is difficult to cover in the original dataset under the constraint of driving knowledge, which can be regarded as an extension of controllable driving scene guidance. Considering a pair of traffic participants in the scene, it is divided into ego agent and adversarial agent. The former aims to avoid collision and drive off the drivable area, while the adversarial agent encourages some relatively aggressive driving behavior and potential conflict. According to the idea of signal timing logic, based on the relative state and action of the agent, a general representation of adversarial safety-critical scene is designed, which can be expressed as:

[0099]

[0100] where, The relative distance between the adversarial agent and the ego agent is calculated, The distance between the agent and the road boundary is calculated to encourage dangerous behaviors such as collision or driving off the drivable area.

[0101] The general scene representation based on driving knowledge enhancement designed in this step will be embedded in the form of a differentiable cost function into the scene generation framework, guiding the generation of driving scenes that meet the constraints during the scene generation stage.

[0102] S3: Establish a diffusion model for controllable generation of autonomous driving scenes, embed the guide representation into the diffusion model, and perturb the joint action information under the constraint of the guide representation to obtain an autonomous driving scene.

[0103] S31: Establish a diffusion model for controllable generation of autonomous driving scenes, and the input of the diffusion model is the predicted joint action information.

[0104] The generation of autonomous driving scenes can be expressed as an optimization problem as follows:

[0105]

[0106] where f represents the vehicle dynamics model, s, u, and c represent the agent state, action, and context information, respectively. Scene generation can be regarded as solving the optimal action that meets the constraint of real driving behavior according to the objective function. Given the context information c and the agent action u, the probability density of generating a scene can be represented as p(S) = p(u|c)p(c). Where p(u|c) can be approximated as:

[0107]

[0108] where Z θ is the partition function considered as a constant. The above problem can be trained by maximizing the conditional log-likelihood of the dataset:

[0109]

[0110] Score matching can directly learn the gradient of the optimization objective J θ as a score function:

[0111]

[0112] However, due to the non-optimal scenario of the action space, the gradient estimation in the non-optimal region will cause significant deviation. Therefore, the diffusion model adopts a stochastic process to transform the data distribution p by k-step noise addition to the known noise distribution p k , which converts the above problem into training a score function The specific approximation of the gradient of the log-noise distribution can be expressed as:

[0113]

[0114] The diffusion-based scene generation can be solved by learning the gradient of the trajectory optimization and sampling the optimal control sequence by reverse diffusion, that is, the stochastic gradient descent along the fixed high-probability region of the diffusion step. While the controllable generation utilizes the guide g to guide the generation of scenes that meet the driving constraint requirements without the need to retrain the model, similar to the classifier guiding diffusion, the optimization problem of controllable diffusion generation can be modified as:

[0115]

[0116] Therefore, the reverse diffusion process of the diffusion model can be expressed as:

[0117]

[0118] where, represents the probability of the target given the action at noise step k. Since and p(u|c,g) are not equivalent, the actual approximation using the gradient likelihood method of the noise is used to realize the combination of controllable guidance.

[0119] S32: Take the guide representation as a constraint, perturb the joint action information, and generate an autonomous driving scene that meets the constraint limit.

[0120] The map information, the agent action and state information are used, the agent interaction representation is realized by using a scene encoder, the guided representation is combined with the action sampled from the initial noise action distribution, the joint motion prediction is realized by using a denoising model, in the scene reasoning stage, the mean value of the distribution of the joint motion prediction is gradient disturbed according to the constraint of the guided representation, and controllable sampling is performed on the agent motion distribution after the disturbance, to generate a driving scene satisfying the constraint of the guided representation.

[0121] Given the action trajectory u of the kth diffusion step k The denoising diffusion probabilistic model (DDPM) forward noise injection process can be represented as:

[0122]

[0123] Where β k ∈ (0, 1) represents a predefined noise schedule and controls the scale of the noise injected at each diffusion step. u is the initial noise-free action, u k The action of the kth step noise injection process, after K-step forward noise injection process, the data distribution gradually tends to be isotropic Gaussian distribution The calculation process of the related parameters is represented as:

[0124] α k = 1-β k (25)

[0125]

[0126] In the formula, The cosine noise schedule can be defined as β k The formula can be represented as:

[0127]

[0128] According to the original action trajectory u, the noise action can be calculated by forward diffusion as follows:

[0129]

[0130] The conditional denoising model is learned by using the given context information c, and the clean action information generated by iterative prediction of the sampled noise can be represented as:

[0131]

[0132] Where, Is a Gaussian noise distribution, μ θ is the posterior mean of the predicted k-1 step action, σ kis the standard deviation obtained based on the fixed noise scheduling, D θ The prediction result of the denoiser represented by is calculated by the predicted denoising action and the noise action, and the calculation expression of the mean value is:

[0133]

[0134] Next, in order to generate controllable driving scenes that meet the actual individualized needs, the cost function of the knowledge-enhanced general scene guidance designed in step S2 is embedded into the controllable diffusion framework. The controllable scene generation problem can be regarded as optimal trajectory distribution sampling based on knowledge guidance g, and controllable samples are realized by perturbing the model prediction mean value in each reverse denoising step, which is specifically represented as:

[0135]

[0136] For the controllable generation problem of driving scenes, a knowledge-guided controllable diffusion sampling method of general scenes is proposed, which uses the gradient descent method to guide sample generation. The detailed calculation process is as shown in Figure 3 The specific calculation process includes the following:

[0137] The inputs of the algorithm include: denoiser model D θ , scene guidance cost function J g , diffusion step number K, gradient step number N g , scale parameter a, standard deviation s k , self-agent number e, adversarial agent number a, gradient descent step N d , gradient ascent step N a .

[0138] The output result is the guided denoising action u.

[0139] First, sample from the initial noise action trajectory:

[0140]

[0141] From K→1, the reverse denoising process is calculated, and the denoising prediction action is calculated:

[0142]

[0143] The mean value can be calculated by the predicted denoising action and the noise action, that is, formula (31).

[0144] Next, according to the type of guided scene, different gradient calculation methods are selected. If it is a controllable scene guidance, the gradient descent method is used, and the specific calculation is to traverse the gradient step, and calculate the perturbed prediction mean value according to formula (32).

[0145] When the scene guidance is adversarial guidance, the gradient ascent under the original sampling framework is modified to gradient ascent-descent, and the modified gradient calculation can be represented as:

[0146] For the ego agent, the traversal gradient descent step 1→N d :

[0147]

[0148] For the adversarial agent, the traversal gradient ascent step 1→N a , which encourages the adversarial agent to make rule-violating driving actions:

[0149]

[0150] After obtaining the perturbed prediction mean, the prediction distribution is sampled to obtain the denoising action:

[0151]

[0152] Based on the above gradient calculation method, the perturbed denoising action, i.e. the action satisfying the corresponding scene cost function constraint, can be obtained to achieve the controllability of the driving scene.

[0153] Preferably, in view of the problem that the inference speed of the standard diffusion model is slow and affects the real-time performance, the present application adds a learnable motion distribution estimation module to the pre-trained denoising model. For a K-step standard diffusion model, the prior distribution of the denoising step τ is directly predicted, and the denoising step after the τ step is executed normally to obtain clean action samples. This method can skip the denoising step before the τ step to speed up.

[0154] By decomposing the motion prior distribution into mean, variance and sample prediction, each part corresponds to a trainable module, and the calculation process of the whole module can be represented as:

[0155] μ mpe,θ =f μ (x,c) (38)

[0156] σ mpe,θ =f σ (x,c) (39)

[0157]

[0158] Where f μ , f σ and are three trainable modules composed of multiple MLPs. μ mpe,θ and σ mpe,θThe mean and standard deviation of the de-noising distribution at the t-th step, which is shared across samples. is the normalized value of the predicted sample. The resulting prior distribution can be seen as the motion distribution before the de-noising process at the t-th step. De-noising only for steps after the t-th step can significantly reduce the running time.

[0159] During the training of this stage, only the motion prior distribution estimation module is trained, and the denoising prediction is accelerated by freezing the scene context encoding module, etc. The loss function is modified as:

[0160]

[0161] where ω is the weight parameter, the first item of the loss is used to evaluate the difference between the generated sample and the true value. The second item is the uncertainty loss on the variance estimate σ mpe,θ Normalization is performed to balance the diversity and accuracy of the prediction.

[0162] Scene performance evaluation:

[0163] The performance of the generated scene is evaluated from controllability, realism, and stability. The controllability index is used to measure the degree of rule violation. For the controllability index, a specific case is given, and the evaluation index of the controllable driving scene to avoid collision is given below:

[0164] r no-collision :=Σ i≠j 1[||(x i,t ,y i,t )-(x j,t ,y j,t )||≤ε d ] (43)

[0165] The realism index is used to evaluate the statistical similarity between the generated sample and the true value, which is defined by the average value of the longitudinal acceleration, lateral acceleration, and pitch value distribution distance, and is specifically represented as:

[0166]

[0167] where Real is the realism index, and Wass() represents the Wasserstein distance between data distributions, which is used to measure the statistical similarity between data distributions.

[0168] The stability measure is calculated by the average proportion of agents that produce serious failures in the driving scene. It is worth noting that a lower evaluation index indicates better performance.

[0169] Simulation experiment results:

[0170] The embodiment is implemented under Ubuntu 20.04 system, Pytorch 1.13.1 environment, and RTX-4090 GPU (24GB) is used for model training and subsequent scene reasoning. The nuScene dataset is used to train the scene diffusion model, and the collision-free guidance of controllable driving scenes and the adversarial guidance are selected as example verification scenes to generate the effect of the scene. As shown in Figure 4 As shown in Figure 5 , respectively, are the visualization results of the controllable collision-free driving scene and the adversarial driving scene. Figure 4 As can be seen in the scene, the generated scene can meet the collision-free scene constraint requirement, while in Figure 5 , the adversarial traffic participants will make some aggressive behaviors, and some adversarial driving scenes will be generated. Table 1 gives the qualitative analysis results of the scene generation performance, which can be seen that compared with the comparative method, the controllability, reality and stability indicators of the generated scene are improved by 8.1%, 15.3% and 9.2 respectively. Table 2 verifies the effectiveness of the acceleration module, which can be seen that in the case of increasing the acceleration module, the performance indicators of the generated scene can still be guaranteed, but the scene reasoning speed is improved by 41.5%, which verifies the feasibility of the proposed scene generation method.

[0171] Table 1 Scene generation performance

[0172]

[0173] Table 2 Acceleration module

[0174]

[0175] The present application mainly aims at the controllable generation method of high-fidelity driving scene, and the specific advantages mainly include the following aspects. First, an accurate scene-level joint motion prediction model is designed, based on the spatio-temporal coupling interaction feature fusion of factor decomposition attention mechanism, accurate multi-traffic participant joint motion prediction is realized, and a reliable basis is provided for subsequent scene generation. Secondly, the general driving scene guidance representation is designed combined with driving prior knowledge and traffic constraints, which is used to guide the generation of driving scenes that meet the actual needs, and to ensure the reality and controllability of the generated scenes. Finally, a controllable diffusion sampling method is designed, based on the controllable generation framework, and the gradient disturbance of the predicted behavior information is performed in the scene generation stage by using the differentiable cost function obtained by the general scene representation to obtain the driving scene that meets the actual needs, and the individualized needs of real driving are met.

[0176] The preferred embodiments of the present application have been described above in detail. It should be understood that modifications and variations to the present application can be affected by those skilled in the art without departing from the scope of the application. Accordingly, it is intended that all possible modifications and alterations be included within the scope of the present application as defined by the following claims.

Claims

1. A knowledge-enhanced automatic driving scene controllable generation method, characterized in that, The method comprises the following steps: S1: establishing an interactive representation model of traffic participants, performing joint motion prediction on each traffic participant in the autonomous driving scene through the interactive representation model, and obtaining predicted joint action information; S2: establishing a guide representation of the autonomous driving scene through traffic prior knowledge; S3: establishing a diffusion model for controllable generation of the autonomous driving scene, embedding the guide representation into the diffusion model, and perturbing the joint action information under the constraint of the guide representation to obtain the autonomous driving scene; The guide representation comprises constraint conditions and adversarial representations, the constraint conditions comprise behavior constraints, collision constraints, and scene constraints, and the calculation expression of the behavior constraints is: wherein, is a behavioral constraint, represents a goal state or action of the agent, represents a state or action at the current time, represents a smooth L1 loss; The calculation expression of the collision constraints is: In the formula, is a collision constraint, is an agent with at a relative distance at a time instant, is a collision threshold; The calculation expression of the scene constraints is: wherein is a scene constraint, represents an agent at time distance between the agent and the road boundary, represents a distance threshold from the road boundary; The calculation expression of the adversarial representations is: wherein is the adversarial representation, is the relative distance between the adversarial agent and the ego agent, is the distance between the agent and the road boundary.

2. The knowledge-enhanced automatic driving scene controllable generation method according to claim 1, characterized in that, In step S1, the loss function of the joint motion prediction of the interactive representation model is: In the formula, is a loss function value, is a scene context information, is a noise level, is a noise action, is an agent action, is a denoiser model.

3. The knowledge-enhanced automatic driving scene controllable generation method according to claim 1, characterized in that, Step S1 comprises the following steps: S11: establishing an interactive representation model of traffic participants, the interactive representation model comprising relative distances, relative headings, and relative speeds between the traffic participants; S12: embedding, into the interactive representation model in sequence, historical position information of intelligent agents, map encoding, and relative position encoding by using a factorization attention mechanism to obtain scene context encoding; S13: denoising the scene context encoding, and performing joint motion prediction using the denoised scene context encoding to obtain predicted joint action information.

4. The knowledge-enhanced automatic driving scene controllable generation method according to claim 3, characterized in that, The process of denoising the encoding of the scene context is performed by a motion distribution estimation module, which, for one step of the diffusion model, skips the denoising step before the first step of the diffusion model by directly predicting the prior distribution of the first step of the diffusion model.

5. The knowledge-enhanced automatic driving scene controllable generation method according to claim 4, characterized in that, The calculation expression of the motion distribution estimation module is: Where, , and Both are trainable modules composed of multiple layers of MLP. and Corresponding to the The predicted mean and standard deviation of the denoising steps, is the normalized value of the predicted sample, For the The motion distribution before the denoising step, For overall input; In the training process of the motion distribution estimation module, the loss function is: In the formula, is a weight parameter, is a loss function value of the interaction representation model, is a result of normalizing the variance estimation by the uncertainty loss.

6. The knowledge-enhanced automatic driving scene controllable generation method according to claim 1, characterized in that, Step S3 comprises the following steps: S31: establishing a diffusion model for controllable generation of the autonomous driving scene, the input of the diffusion model being the predicted joint action information; S32: perturbing the joint action information as a constraint to generate an autonomous driving scene satisfying the constraint limit.

7. The knowledge-enhanced automatic driving scene controllable generation method according to claim 6, characterized in that, The specific process of generating the autonomous driving scene satisfying the constraint limit is: By using map information and intelligent agent action and state information, realizing intelligent agent interactive representation by using a scene encoder, combining actions sampled from an initial noise action distribution, realizing joint motion prediction by using a denoising model, in a scene reasoning stage, performing gradient perturbation on the mean of the distribution of the joint motion prediction according to the constraint of the guide representation, and controllably sampling the perturbed intelligent agent motion distribution to generate a driving scene satisfying the constraint of the guide representation.

8. The knowledge-enhanced automatic driving scene controllable generation method according to claim 7, characterized in that, The diffusion model comprises a self-intelligent agent and an adversarial intelligent agent, the guide representation comprises constraint conditions and adversarial representations, when the guide representation of the diffusion model is the constraint conditions, the diffusion model generates autonomous driving scene samples by using a traversal gradient descent method; when the guide representation of the diffusion model is the adversarial representations, the self-intelligent agent generates autonomous driving scene samples by using a traversal gradient descent method, and the adversarial intelligent agent generates autonomous driving scene samples by using a traversal gradient ascent method.

9. The knowledge-enhanced automatic driving scene controllable generation method according to claim 1, characterized in that, The generation result of the autonomous driving scene is evaluated by controllability indexes and reality indexes, the calculation expression of the controllability indexes is: In the formula, is the controllability index value, is the agent abscissa, is the agent ordinate, is is the time, and are both agent numbers, is the collision threshold; The calculation expression of the reality indexes is: In the formula, is the authenticity index value, is the Wasserstein distance between data distributions, and n is the total number of agents, is the data distribution of the agent.

Images