Method and device for generating video data of automatic driving scene

Abstract

The invention relates to a video data generation method and device for an automatic driving scene, and the method comprises the steps: inputting the text information used for describing the driving scene of a vehicle into a large language model LLM, and outputting the target information used for constructing the automatic driving scene; respectively inputting the traffic road data into a static driving scene reconstruction dedicated neural rendering network and a dynamic traffic participant reconstruction dedicated neural rendering network to obtain output results of the static driving scene reconstruction dedicated neural rendering network and the dynamic traffic participant reconstruction dedicated neural rendering network; and generating video data of an automatic driving scene associated with the vehicle based on the target information and the output result. According to the method and the device, the video data of the automatic driving scene can be generated in batches, so that the technical problem that a driving scene generation mode in the prior art cannot meet the requirement of obtaining large-batch data on a large scale required by an automatic driving perception algorithm is solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and device for generating video data for an autonomous driving scene. Background Art

[0002] At present, there are two main strategies for generating image data in new driving scenarios: one is based on the digital twin scene building module, and the other is based on the trajectory editing module and implicit 3D reconstruction module. The strategy based on the digital twin scene building module is to model different physical entities through 3D reconstruction technology and build a twin model library, and then build a virtual twin world based on game engine software; finally, the physical entity data collected in the real world is mapped to the corresponding twin model, and the real data is used to drive the operation of the autonomous driving system in the digital twin scenario, so as to realize the real-time mapping and synchronization of the real space and the virtual space; specifically, it includes twin modeling, weather system modeling, road modeling, and scene modeling. The labor and time costs of this strategy are not enough to bear the large-scale reconstruction of urban roads.

[0003] The strategy based on the trajectory editing module and the implicit 3D reconstruction module is to reconstruct the autonomous driving scene using a multi-view implicit surface reconstruction method to obtain a scene reconstruction result; based on the offline high-precision map, the target detection and tracking results, and the road network structure, the vehicle trajectory is edited to generate a vehicle trajectory. Although this strategy expresses the autonomous driving scene through implicit neural rendering, the vehicle trajectory is still manually edited and still does not have the ability to generate large quantities of data required for the autonomous driving perception algorithm in batches. However, neither the strategy based on the digital twin scene building module nor the strategy based on the trajectory editing module and the implicit 3D reconstruction module can meet the needs of the autonomous driving perception algorithm for large-scale acquisition of large quantities of data. Summary of the invention

[0004] The present application provides a method and device for generating video data of an autonomous driving scene, so as to solve the problem that the way of generating driving scenes in the prior art cannot meet the demand for obtaining large amounts of data on a large scale required by the autonomous driving perception algorithm.

[0005] In a first aspect, the present application provides a method for generating video data of an autonomous driving scene, comprising: inputting text information used to describe a driving scene of an autonomous vehicle into a large language model (LLM), and outputting target information for constructing an autonomous driving scene; wherein the LLM is a vertical domain LLM for generating trajectories of traffic participants; the target information includes information of a static driving scene associated with the autonomous vehicle, and information of the autonomous vehicle and other physical entities associated with it in the driving scene; the target information is respectively input into a neural rendering network dedicated to static driving scene reconstruction and a neural rendering network dedicated to dynamic traffic participant reconstruction, to obtain output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction; wherein the neural rendering network dedicated to static driving scene reconstruction is trained based on data associated with static driving scenes in previously collected traffic road data, and the neural rendering network dedicated to dynamic traffic participant reconstruction is trained based on data of the autonomous vehicle and other physical entities associated with it in previously collected traffic road data; the output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction are fused to obtain video data of the autonomous driving scene associated with the target information and associated with the autonomous vehicle.

[0006] In the second aspect, the present application provides a video data generation device for an autonomous driving scene, including: a first processing module, used to input text information used to describe the driving scene of the self-driving vehicle into a large language model LLM, and output target information for constructing the autonomous driving scene; wherein the LLM is a vertical domain LLM for generating the trajectory of traffic participants; the target information includes information about the static driving scene associated with the self-vehicle, and information about other physical entities associated with the self-vehicle and the self-vehicle in the driving scene; a second processing module, used to input the target information into a neural rendering network dedicated to static driving scene reconstruction and a neural rendering network dedicated to dynamic traffic participant reconstruction, respectively, to obtain the static driving scene reconstruction The output results of the dedicated neural rendering network and the dedicated neural rendering network for dynamic traffic participant reconstruction; wherein the dedicated neural rendering network for static driving scene reconstruction is trained based on data associated with static driving scenes in the traffic road data collected in the early stage, and the dedicated neural rendering network for dynamic traffic participant reconstruction is trained based on data associated with the self-vehicle and other physical entities associated with it in the traffic road data collected in the early stage; a third processing module is used to fuse the output results of the dedicated neural rendering network for static driving scene reconstruction and the dedicated neural rendering network for dynamic traffic participant reconstruction to obtain video data of the automatic driving scene associated with the target information and associated with the self-vehicle.

[0007] In a third aspect, the present application provides an electronic device comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the video data generation method for an autonomous driving scenario as described in any one of the embodiments of the first aspect above.

[0008] In a fourth aspect, the present application also provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the video data generation method for the autonomous driving scene described in any one of the embodiments of the first aspect above.

[0009] Beneficial effects of this application:

[0010] In the present application, after the text information used to describe the driving scene of the own vehicle is input into the vertical domain LLM used to generate the trajectories of traffic participants, target information including information of the static driving scene associated with the own vehicle and information of the own vehicle and other physical entities associated with it in the driving scene can be generated. Then, combined with the neural network field representation of the static scene of autonomous driving constructed by the neural rendering network dedicated to static driving scene reconstruction, and the neural network field representation of the physical entities in the driving scene constructed by the neural rendering network dedicated to dynamic traffic participant reconstruction, image data under new driving scenes can be generated. Since the vertical domain LLM used to generate traffic participant trajectories, the neural rendering network dedicated to static driving scene reconstruction, and the neural rendering network models dedicated to dynamic traffic participant reconstruction are all obtained by fine-tuning or training with traffic road data of different scenarios. Therefore, based on the above-mentioned method in the embodiment of the present application, corresponding video data of autonomous driving scenes can be generated for text information in different application scenarios, that is, the vertical LLM for generating traffic participant trajectories, the neural rendering network dedicated to static driving scene reconstruction, and the neural rendering network dedicated to dynamic traffic participant reconstruction in the embodiment of the present application can batch generate video data of autonomous driving scenes, thereby solving the technical problem that the method of generating driving scenes in the prior art cannot meet the technical problem of large-scale acquisition of large amounts of data required by the autonomous driving perception algorithm, and improving the efficiency of generating autonomous driving scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of a method for generating video data for an autonomous driving scenario provided by the present application;

[0012] Figure 2 One of the optional flow charts of a method for generating video data for an autonomous driving scenario provided in an embodiment of the present application;

[0013] Figure 3A second optional flowchart of a method for generating video data for an autonomous driving scenario provided in an embodiment of the present application;

[0014] Figure 4 A third optional flowchart of a method for generating video data for an autonomous driving scenario provided in an embodiment of the present application;

[0015] Figure 5 A fourth optional flowchart of a method for generating video data for an autonomous driving scenario provided in an embodiment of the present application;

[0016] Figure 6 An optional flowchart of a method for generating video data for an autonomous driving scenario provided in an embodiment of the present application;

[0017] Figure 7 A schematic diagram of a video data generation algorithm framework for an autonomous driving scenario provided in an embodiment of the present application;

[0018] Figure 8 A schematic diagram of a flow chart of a video data generation algorithm for an autonomous driving scenario provided in an embodiment of the present application;

[0019] Fig. 9 A flowchart of a method for generating image data in a driving scenario based on LLM assistance provided in an embodiment of the present application;

[0020] Fig.10 A schematic diagram of the structure of a video data generating device for an autonomous driving scenario provided in an embodiment of the present application;

[0021] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0024] For ease of description, spatial relative terms may be used herein to describe the relative positional relationship or movement of an element or feature relative to another element or feature as shown in the figure, such as "inside", "outside", "inner side", "outer side", "below", "below", "above", "above", "front", "back", etc. Such spatial relative terms are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the figure. For example, if the device in the figure undergoes a position flip or a change in posture or a change in motion state, then these directional indications also change accordingly, for example: an element described as "below other elements or features" or "below other elements or features" will subsequently be oriented as "above other elements or features" or "above other elements or features". Therefore, the example term "below..." may include both upper and lower orientations. The device may be otherwise oriented (rotated 90 degrees or in other directions) and the spatial relative descriptors used herein are interpreted accordingly.

[0025] In order to solve the technical problem that the existing method of generating driving scenes cannot meet the large-scale data requirements required by the autonomous driving perception algorithm, this application provides a method for generating video data of autonomous driving scenes, such as Figure 1 As shown, the steps of the method include:

[0026] Step 101, inputting text information used to describe the driving scene of the ego vehicle into the large language model LLM, and outputting target information for constructing the autonomous driving scene; wherein the LLM is a vertical domain LLM used to generate the trajectory of traffic participants; the target information includes information of the static driving scene associated with the ego vehicle, and information of other physical entities associated with the ego vehicle and the driving scene;

[0027] It should be noted that LLM (Large Language Model) is a natural language processing model that can understand and generate human-like language by pre-training massive text data. Its core is a multi-layer Transformer architecture, which includes N Transformer blocks, each of which consists of a self-attention mechanism and a feedforward neural network. Fine-tuning is then used to fine-tune specific tasks, and the fine-tuning methods include: mask prediction and next sentence prediction. Among them, the mask method refers to masking the position and deflection angle of any frame in the time series, and calculating the loss between the prediction and the true value. The next sentence prediction refers to predicting the position and deflection angle of the next frame by the position and deflection angle of the previous frames. Fine-tuning can make the LLM large model adapt to trajectory generation to improve the performance of the model on this task (trajectory generation).

[0028] In addition, the text information in the above-mentioned embodiment of the present application may be "I am driving in the middle of the road, and there is a car in front of me. It drives forward to the next intersection and turns left", or "I overtake in the overtaking lane on the highway, and then get off the highway at the next intersection", etc. The text information can be set according to actual needs to generate an autonomous driving scene. Furthermore, the information of the static driving scene associated with the self-vehicle in the target information may refer to the semantic information in the traffic road data, such as the number of lanes on the current driving road, whether the sides of the current lane are solid lines or dashed lines, and other static description information related to the current driving road, and may also be the sensor parameters of the self-vehicle, and the information of the self-vehicle and other physical entities associated with it in the target information may refer to the trajectory of the self-vehicle in the current traffic road data, and the trajectories of other vehicles around the self-vehicle (traffic participant trajectories), etc.

[0029] Step 102, inputting the target information into a neural rendering network dedicated to static driving scene reconstruction and a neural rendering network dedicated to dynamic traffic participant reconstruction, respectively, to obtain output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction; wherein the neural rendering network dedicated to static driving scene reconstruction is trained based on data associated with static driving scenes in the traffic road data collected in the early stage, and the neural rendering network dedicated to dynamic traffic participant reconstruction is trained based on data associated with the vehicle and other physical entities associated with the vehicle in the traffic road data collected in the early stage;

[0030] It should be noted that the neural rendering network dedicated to static driving scene reconstruction in the embodiment of the present application is intended to synthesize images from real-world observations. The synthesized images are photo-realistic images and videos. The scene representation is reconstructed from the observation using a differentiable rendering loss, thereby realizing the process of reverse graphics or reverse rendering. The neural rendering network dedicated to dynamic traffic participant reconstruction in the embodiment of the present application is used to simulate and reconstruct dynamic traffic participants. Through the neural rendering network dedicated to dynamic traffic participant reconstruction, it is possible to process and simulate the behavior of dynamic traffic participants, including vehicles, pedestrians, etc., and the dynamic changes in complex traffic scenes.

[0031] In order to make the autonomous driving perception algorithm robust in different scenarios, the traffic road data collected in the early stage in the embodiment of the present application is in large quantities in order to ensure the accuracy of the neural network training, and the traffic road data includes data in different scene distributions, such as sensor angles, parameters, tunnels, elevated roads, highways, urban roads, sunny days, cloudy days, light rain, heavy rain, evening, and late night of different models. In the embodiment of the present application, these data can be obtained in batches and scaled to train a universal perception algorithm, which is suitable for all scenarios, so that the autonomous driving perception algorithm in the embodiment of the present application can be robust in different scenarios.

[0032] Step 103, fusing the output results of the neural rendering network dedicated to reconstructing static driving scenes and the neural rendering network dedicated to reconstructing dynamic traffic participants, to obtain video data of the autonomous driving scene associated with the target information and the self-vehicle.

[0033] Through steps 101 to 103 of the above-mentioned embodiment of the present application, after the text information used to describe the driving scene of the own vehicle is input into the vertical domain LLM used to generate the trajectories of traffic participants, target information including information of the static driving scene associated with the own vehicle and information of the own vehicle and other physical entities associated with it in the driving scene can be generated. Then, combined with the neural network field representation of the static scene of autonomous driving constructed by the neural rendering network dedicated to static driving scene reconstruction, and the neural network field representation of the physical entities in the driving scene constructed by the neural rendering network dedicated to dynamic traffic participant reconstruction, image data under a new driving scene can be generated. Since the vertical domain LLM used to generate traffic participant trajectories, the neural rendering network dedicated to static driving scene reconstruction, and the neural rendering network models dedicated to dynamic traffic participant reconstruction are all obtained by fine-tuning or training based on traffic road data of different scenarios. Therefore, based on the above-mentioned method in the embodiment of the present application, corresponding video data of autonomous driving scenes can be generated for text information in different application scenarios, that is, the vertical LLM for generating traffic participant trajectories, the neural rendering network dedicated to static driving scene reconstruction, and the neural rendering network dedicated to dynamic traffic participant reconstruction in the embodiment of the present application can batch generate video data of autonomous driving scenes, thereby solving the technical problem that the method of generating driving scenes in the prior art cannot meet the technical problem of large-scale acquisition of large amounts of data required by the autonomous driving perception algorithm, and improving the efficiency of generating autonomous driving scenes.

[0034] In the embodiment of the present application, before generating target information based on LLM, it is necessary to fine-tune LLM to obtain a method for generating vertical LLM of traffic participant trajectories, such as Figure 2 As shown, it may include:

[0035] Step 201, collecting traffic road data, wherein the traffic road data includes image data and point cloud data associated with the vehicle under different scenes, different lighting conditions, and different sensors;

[0036] Step 202, extracting first data from the traffic road data, wherein the first data includes a trajectory of the vehicle, a trajectory of a traffic participant, semantic information, and sensor parameters associated with the vehicle;

[0037] In a specific example, the vehicle trajectory may refer to the vehicle's driving trajectory from the current moment to a preset time period thereafter, for example, the vehicle will turn right in 5 seconds, that is, the vehicle will change from the current lane to the right-turn lane within the 5 seconds, and the traffic participant trajectory refers to the driving trajectory of other vehicles around the vehicle, such as vehicle A around the vehicle will go straight through the intersection ahead within the 5 seconds, semantic information refers to how many lanes there are currently, whether the lanes around the vehicle are solid lines or dashed lines, etc. Sensor parameters may refer to the number of sensors of the vehicle, and the angle range of data collected by each sensor, etc.

[0038] Step 203, fine-tuning the base large language model LLM based on the first data to obtain a vertical domain LLM for generating the trajectory of the traffic participant, wherein the vertical domain LLM for generating the trajectory of the traffic participant has the ability to generate a time-series posture;

[0039] In a specific example, the method of fine-tuning LLM in the embodiment of the present application includes mask prediction and next sentence prediction. The mask method refers to masking the position and deflection angle of any frame in the time series, and calculating the loss between the prediction and the true value. The next sentence prediction refers to predicting the position and deflection angle of the next frame by the position and deflection angle of the previous frames. Furthermore, through fine-tuning, the LLM large model can be adapted to the field of trajectory generation and improve the performance of the model on this task.

[0040] Based on this, in an embodiment of the present application, for the method of inputting text information into the vertical domain LLM for generating traffic participant trajectories and outputting target information for constructing an autonomous driving scenario involved in the above step 101, text information can be input into the vertical domain LLM for generating traffic participant trajectories and outputting target information for constructing an autonomous driving scenario, wherein the information of the static driving scene associated with the ego vehicle in the target information includes semantic information and sensor parameters associated with the ego vehicle, and the information of the ego vehicle and other physical entities associated with the ego vehicle in the target information includes the ego vehicle trajectory and the traffic participant trajectory.

[0041] In this regard, in a specific example, it can be: For example, the text information described by the user is: "I am driving in the middle of the road, and there is a Mercedes-Benz in front of me. It drives forward to the next intersection and turns left." According to the user's description, first generate a richer description (second text information): "I am driving in the middle of the road, and there is a black Mercedes-Benz 5 meters in front of me. It is moving forward at a constant speed of 40 kilometers and will turn left at the next intersection. The moving coordinate system satisfies the northeast sky coordinate system, that is, x points to the front, y points to the left, and z points to the sky. The rotating coordinate system rotates counterclockwise by theta angle and rotates around the z axis." Then, according to the detailed description, generate JS ON, contains key-value pairs of x, y, z and θ, {'ego':{'x':0,'y':0,'z':0.2,'θ':30,…},'car':{'id':00001,'attribute':'black',location:{'x':0,'y':0,'z':0.2,'θ':30,…},'walker':{'id':00001,'attribute':'black',location:{'x':0,'y':0,'z':0.2,'θ':30,…},…}. That is, in the specific example, the target information can be expressed in JSON.

[0042] In the embodiment of the present application, it is also necessary to train a neural rendering network dedicated to static driving scene reconstruction and a neural rendering network dedicated to dynamic traffic participant reconstruction, wherein the method steps for training a neural rendering network dedicated to static driving scene reconstruction are as follows: Figure 3 Shown include:

[0043] Step 301, constructing a first training data set based on the collected traffic road data, wherein the first training data set includes a plurality of first training samples, each of which includes image data of a current frame, point cloud data of the current frame, a camera intrinsic parameter matrix associated with the current frame, a rotation matrix of the current frame, and a translation vector of the current frame;

[0044] Step 302 : preprocess the first training sample, and train a neural rendering network dedicated to static driving scene reconstruction based on the preprocessed first training sample.

[0045] It can be seen from the content of the first training sample in the first training data set that the first training sample is a time series data that is continuous in time, that is, the data after removing the vehicle and traffic participants from multiple images and point clouds in the road traffic data. In the specific example, the training data set X, training sample x i ∈X,i∈[0,N-1], where N is the number of samples in the training data set. Sample x i = {I i ,P i ,K i ,R i ,T i}, where I i is the i-th frame image, P i is the point cloud of the i-th frame, K i is the camera’s intrinsic parameter matrix, R i is the rotation matrix of the i-th frame, T i is the translation vector of the i-th frame. Further, in the embodiment of the present application, data preprocessing is performed, x i ′=Preprocess(x i ), including image distortion, scaling, cropping, pixel sampling, and point cloud fusion, splicing, and downsampling. Finally, model training is performed: θ :(I,P,K,R,T)→b(R,G,B). Calculate loss, back propagate, and optimize parameters Get better reconstructed images. Get the neural network field of driving static scenes θ The neural rendering network dedicated to static driving scene reconstruction is trained in the above manner, and the trained neural network can be used for the neural network field representation of static driving scenes.

[0046] In another embodiment of the present application, a method step for training a neural rendering network dedicated to dynamic traffic participant reconstruction is as follows: Figure 4 The shown may include:

[0047] Step 401, obtaining the bounding boxes and posture transformation matrices of the vehicle and traffic participants in the collected traffic road data by automatic annotation;

[0048] Step 402, extracting image data and point cloud data of traffic participants through a bounding box;

[0049] Step 403, converting the image data and the point cloud data to the same coordinate system through a posture transformation matrix to obtain a second training set; wherein the second training set includes a plurality of second training samples, and the second training samples include the image data of the traffic participant, the point cloud data of the traffic participant, the camera intrinsic parameter matrix associated with the traffic participant, the rotation matrix of the traffic participant, and the translation vector of the traffic participant;

[0050] Step 404 : training a neural rendering network dedicated to dynamic traffic participant reconstruction based on second training samples in a second training set.

[0051] From the above steps 401 to 404, it can be seen that for the training of the neural rendering network dedicated to dynamic traffic participant reconstruction, the Bbox (bounding box) and pose transformation matrix of the vehicle and the traffic participant are first obtained through the automatic annotation system. Then, the image and point cloud of the traffic participant are extracted through the Bbox, and the image and point cloud are transformed to the same coordinate system through the rotation and translation matrix, and scaled to the same size, so as to obtain the physical entity traffic participant dataset C in the driving scene. Therefore, the training sample c i ∈C,i∈[0,M-1], where M is the number of samples in the training data set. Sample c i = {I i ,P i ,K i ,R i ,T i}, where I i is the image of the i-th object, P i is the point cloud of the i-th object, K i is the camera’s intrinsic parameter matrix, R i is the rotation matrix of the i-th object, T i is the translation vector of the i-th object. Finally, model training is performed based on each physical entity traffic participant. θ :(I,P,K,R,T)→f(R,G,B). Calculate loss, back propagate, and optimize parameters A better reconstructed image is obtained. The neural network field of the physical entity in the driving scene can be obtained by the dynamic traffic participant reconstruction dedicated neural rendering network trained in the above embodiment of the present application. θ .

[0052] Based on the above Figure 3 and Figure 4 For the dedicated neural rendering network for static driving scene reconstruction and the dedicated neural rendering network for dynamic traffic participant reconstruction, the method of inputting the target information into the dedicated neural rendering network for static driving scene reconstruction and the dedicated neural rendering network for dynamic traffic participant reconstruction respectively in the above step 102 to obtain the output results of the dedicated neural rendering network for static driving scene reconstruction and the dedicated neural rendering network for dynamic traffic participant reconstruction, further as follows Figure 5 The shown may include:

[0053] Step 501, inputting traffic road data into a neural rendering network dedicated to static driving scene reconstruction to obtain a first output result, wherein the first output result represents a static scene of the autonomous driving scene;

[0054] Step 502: input the traffic road data into a neural rendering network dedicated to dynamic traffic participant reconstruction to obtain a second output result, wherein the second output result represents the traffic participants in the autonomous driving scenario.

[0055] It can be seen that the static scene of the autonomous driving scene and the data information of the traffic participants in the autonomous driving scene can be obtained through the above-mentioned neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction. Therefore, the target information obtained by combining the text information input by the user can generate the required video data of the autonomous driving scene. Therefore, the output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction involved in the above-mentioned step 103 are fused to obtain the video data of the autonomous driving scene associated with the target information and associated with the self-vehicle. Figure 6 As shown, the method may include the following steps:

[0056] Step 601, generating a camera perspective associated with the vehicle based on the vehicle trajectory;

[0057] Step 602, transforming the traffic participant to the camera perspective through the traffic participant trajectory, the rotation matrix and translation vector of the traffic participant in the second output result, and the bounding box of the traffic participant, to obtain the world coordinates of the traffic participant relative to the vehicle in the first output result;

[0058] Step 603, fusing the output of the neural rendering network dedicated to reconstructing static driving scenes with the output of the neural rendering network dedicated to reconstructing dynamic traffic participants based on world coordinates, and rendering the fusion result to obtain each frame of perspective image under the camera perspective;

[0059] Step 604, based on each frame of perspective image under the camera perspective, stitch each frame of image data output into video data of the autonomous driving scene associated with the target information and associated with the self-vehicle.

[0060] In a specific example, the above steps 601 to 604 may be: generating a camera angle V according to the vehicle trajectory. i ∈V,i∈[0,J-1], where J is the number of frames. View angle v i ={c1,c2,…,c k} is the view group of the jth frame, where c1 is the first view and there are k views. The traffic participant c is transformed into i Transformed to the world coordinates of the static background under the camera perspective V: R ‘ vi =R vi c i Then, according to the position and direction of the Bbox, the neural network field of each traffic participant is connected to the neural network field of the driving static scene in chronological order, and finally according to the input camera perspective V and R ‘ vi , output each frame of image data, and splice each frame of image data into video data.

[0061] The present application is explained below in conjunction with a specific implementation of an embodiment of the present application, which provides a method for generating image data in a driving scenario based on LLM assistance. Figure 7 is a schematic diagram of the algorithm framework in this specific implementation mode, Figure 8 is a flowchart of the algorithm in this specific implementation method. Figure 7 and Figure 8 In this specific implementation, the steps of a method for generating image data in a driving scenario based on LLM assistance are as follows: Fig. 9 Shown include:

[0062] Step 901, trajectory generation: collect various traffic road data sets, including images and point clouds under different scenes, different lighting conditions, and different sensor parameters; extract the trajectory of the vehicle, the trajectory of traffic participants, semantic information, and sensor parameters; fine-tune the LLM; generate the trajectory of the vehicle, the trajectory of traffic participants, semantic information, and sensor parameters.

[0063] The step 901 may further include the following steps:

[0064] Step 11, using temporally continuous time series data, i.e., multiple images and point clouds, the positions and deflection angles of the vehicle and traffic participants at each time series are obtained through a labeling system.

[0065] Step 12: Input the LLM large model and fine-tune the LLM large model through time series prediction.

[0066] Step 13, the user describes the desired scenario. For example, "I am driving in the middle of the road, and there is a Mercedes in front of me. It drives forward to the next intersection and turns left."

[0067] Step 14, generate a richer and more detailed description, such as: "I am driving in the middle of the road, and there is a black Mercedes Benz 5 meters in front of me. It is moving forward at a constant speed of 40 kilometers and will turn left at the next intersection. The moving coordinate system satisfies the northeast sky coordinate system, that is, x points to the front, y points to the left, and z points to the sky. The rotating coordinate system is to rotate theta angle counterclockwise around the z axis."

[0068] Step 15, according to the detailed description, generate JSON, including key-value pairs of x, y, z and θ. For example: {'ego':{'x':0,'y':0,'z':0.2,'θ':30,…},'car':{'id':00001,'attribute':'black',location:{'x':0,'y':0,'z':0.2,'θ':30,…},'walker':{'id':00001,'attribute':'black',location:{'x':0,'y':0,'z':0.2,'θ':30,…},…}.

[0069] Step 902, scene reconstruction: using temporally continuous time series data, multiple images and point clouds are used as input after removing the vehicle and traffic participants, and the deep learning neural network is trained to obtain the trained neural network, and the neural network field representation of the static scene of autonomous driving is obtained.

[0070] Further, step 902 may include the following steps:

[0071] Step 21, using the temporally continuous time series data, remove the vehicle and traffic participants from multiple images and point clouds to obtain a training data set X, training sample x i ∈X,i∈[0,N-1], where N is the number of samples in the training data set. Sample x i = {I i ,P i ,K i ,R i ,Ti}, where I i is the i-th frame image, P i is the point cloud of the i-th frame, K i is the camera’s intrinsic parameter matrix, R i is the rotation matrix of the i-th frame, T i is the translation vector of the i-th frame.

[0072] Step 22, preprocess the data, x i ′=Preprocess(x i ), including image dedistortion, scaling, cropping, pixel sampling, and point cloud fusion, stitching and downsampling.

[0073] Step 23, training model, s θ :(I,P,K,R,T)→b(R,G,B). Calculate loss, back propagate, and optimize parameters Get better reconstructed images.

[0074] Step 24, obtain the neural network field s of the driving static scene θ .

[0075] Step 903, object reconstruction: Reconstruct the traffic participants through the extracted images and point clouds using a multi-view implicit surface reconstruction method to obtain a neural network field representation of the physical entities in the driving scene.

[0076] Furthermore, step 903 may further include the following steps:

[0077] Step 31, obtain the Bbox and pose transformation matrix of the vehicle and traffic participants through the annotation system.

[0078] Step 32: extract the image and point cloud of the traffic participants through Bbox, transform the image and point cloud to the same coordinate system through rotation and translation matrix, scale them to the same size, and obtain the physical entity traffic participant dataset C in the driving scene. Training sample c i ∈C,i∈[0,M-1], where M is the number of samples in the training data set. Sample c i = {I i ,P i ,K i ,R i ,T i}, where I i is the image of the i-th object, P i is the point cloud of the i-th object, K i is the camera’s intrinsic parameter matrix, R i is the rotation matrix of the i-th object, T i is the translation vector of the ith object.

[0079] Step 33, for each physical entity traffic participant, train the model, θ :(I,P,K,R,T)→f(R,G,B). Calculate loss, back propagate, and optimize parameters Get better reconstructed images.

[0080] Step 34, obtaining the neural network field of the physical entity in the driving scene θ .

[0081] Step 904, autonomous driving scene data generation: based on the generated vehicle trajectory, traffic participant trajectory, semantic information and sensor parameters, combined with the neural network field of the driving static scene and the neural network field representation of the physical entity in the driving scene, generate image data for the new driving scene.

[0082] Furthermore, step 904 may further include the following steps:

[0083] Step 41, generate the camera view angle V according to the vehicle trajectory. i ∈V,i∈[0,J-1], where J is the number of frames. View angle v i ={c1,c2,…,c k} is the view group of the jth frame, where c1 is the first view and there are k views.

[0084] Step 42, the traffic participant c is transformed into i Transformed to the world coordinates of the static background under the camera perspective V: R ‘ vi =R vi c i

[0085] Step 43, according to the position and orientation of the bbox, the neural network field of each traffic participant is connected to the neural network field of the driving static scene in chronological order.

[0086] Step 44: according to the input camera angle V and R ‘ vi , output each frame of image data and splice it into video data.

[0087] It can be seen that in this specific implementation, a deep learning neural network is used to reconstruct the driving scene through the collected discrete images and point cloud data, and a continuous neural network field expression of the driving scene is obtained; then the LLM is trained through the collected motion trajectories of the vehicle and traffic participants, and more trajectories are generated through the trained LLM; finally, the trajectories are used to generate more data from the continuous neural network field of the scene, so as to obtain the ability to batch generate training data sets for autonomous driving perception algorithms.

[0088] Corresponding to the above-mentioned method embodiments of the present application, the following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application.

[0089] Fig.10 This is a schematic diagram of the structure of a video data generation device for an autonomous driving scenario provided in an embodiment of the present application. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Fig.10 As shown, the device comprises:

[0090] The first processing module 1002 is used to input text information used to describe the driving scene of the ego vehicle into the large language model LLM, and output target information used to construct the autonomous driving scene; wherein the LLM is a vertical domain LLM used to generate the trajectory of traffic participants; the target information includes information of the static driving scene associated with the ego vehicle and information of other physical entities associated with the ego vehicle and the ego vehicle in the driving scene;

[0091] The second processing module 1004 is used to input the target information into the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction, respectively, to obtain output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction; wherein the neural rendering network dedicated to static driving scene reconstruction is trained based on data associated with static driving scenes in the traffic road data collected in the early stage, and the neural rendering network dedicated to dynamic traffic participant reconstruction is trained based on data associated with the vehicle and other physical entities associated with the vehicle in the traffic road data collected in the early stage;

[0092] The third processing module 1006 is used to fuse the output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction to obtain video data of the automatic driving scene associated with the target information and the self-vehicle.

[0093] Through the device of the embodiment of the present application, after the text information used to describe the driving scene of the self-vehicle is input into the large language model LLM, target information including information of the static driving scene associated with the self-vehicle and information of other physical entities associated with the self-vehicle and the driving scene can be generated, and then combined with the neural network field representation of the static scene of autonomous driving constructed by the neural rendering network dedicated to static driving scene reconstruction, and the neural network field representation of the physical entities in the driving scene constructed by the neural rendering network dedicated to dynamic traffic participant reconstruction, image data under new driving scenes can be generated, because the LLM, the neural rendering network dedicated to static driving scene reconstruction, and the neural rendering network model dedicated to dynamic traffic participant reconstruction are all obtained by fine-tuning or training traffic road data of different scenes. Therefore, based on the above-mentioned method in the embodiment of the present application, the video data of the corresponding autonomous driving scene can be generated for the text information in different application scenarios, that is, the LLM, the neural rendering network dedicated to static driving scene reconstruction, and the neural rendering network dedicated to dynamic traffic participant reconstruction in the embodiment of the present application can generate video data of the autonomous driving scene in batches, thereby solving the technical problem that the method of generating driving scenes in the prior art cannot meet the large-scale acquisition of large-scale data required by the autonomous driving perception algorithm, and improves the efficiency of generating autonomous driving scenes.

[0094] In an optional implementation manner of the embodiment of the present application, the device in the embodiment of the present application may also include: an acquisition module, used to collect traffic road data, wherein the traffic road data includes image data and point cloud data associated with the vehicle under different scenarios, different lighting conditions and different sensors of road traffic; an extraction module, used to extract first data from the traffic road data, wherein the first data includes the vehicle trajectory, traffic participant trajectory, semantic information and sensor parameters associated with the vehicle; a third processing module, used to fine-tune the base large language model LLM based on the first data, wherein a vertical domain LLM for generating traffic participant trajectories is obtained, wherein the vertical domain LLM for generating traffic participant trajectories has the ability to generate time series posture.

[0095] In an optional implementation manner of an embodiment of the present application, the first processing module in the embodiment of the present application includes: a first processing unit, used to input text information into a vertical LLM for generating traffic participant trajectories, and output target information for constructing an autonomous driving scenario, wherein the information of the static driving scene associated with the ego vehicle in the target information includes semantic information and sensor parameters associated with the ego vehicle, and the information of the ego vehicle and other physical entities associated with the ego vehicle in the target information includes the ego vehicle trajectory and the traffic participant trajectory.

[0096] In an optional implementation manner of an embodiment of the present application, the device in the embodiment of the present application also includes: a first construction module, used to construct a first training data set based on the collected traffic road data, wherein the first training data set includes multiple first training samples, and each first training sample includes image data of the current frame, point cloud data of the current frame, a camera intrinsic parameter matrix associated with the current frame, a rotation matrix of the current frame, and a translation vector of the current frame; a first training module, used to preprocess the first training samples, and train a neural rendering network dedicated to static driving scene reconstruction based on the preprocessed first training samples.

[0097] In an optional implementation manner of an embodiment of the present application, the device in the embodiment of the present application also includes: a fourth processing module, used to obtain the bounding box and posture transformation matrix of the vehicle and traffic participants in the collected traffic road data by automatic annotation; an extraction module, used to extract the image data and point cloud data of the traffic participants through the bounding box; a fifth processing module, used to convert the image data and point cloud data to the same coordinate system through the posture transformation matrix to obtain a second training set; wherein the second training set includes multiple second training samples, and the second training samples include the image data of the traffic participant, the point cloud data of the traffic participant, the camera intrinsic parameter matrix associated with the traffic participant, the rotation matrix of the traffic participant, and the translation vector of the traffic participant; a second training module, used to train a neural rendering network dedicated to dynamic traffic participant reconstruction based on the second training samples in the second training set.

[0098] In an optional implementation manner of an embodiment of the present application, the second processing module in the embodiment of the present application may further include: a first output unit, used to input traffic road data into a neural rendering network dedicated to static driving scene reconstruction, to obtain a first output result, wherein the first output result represents the static scene of the autonomous driving scene; a second output unit, used to input traffic road data into a neural rendering network dedicated to dynamic traffic participant reconstruction, to obtain a second output result, wherein the second output result represents the traffic participants in the autonomous driving scene.

[0099] In an optional implementation manner of the embodiment of the present application, the third processing module in the embodiment of the present application may further include: a generation unit, used to generate a camera perspective associated with the ego-vehicle based on the ego-vehicle trajectory; a second processing unit, used to transform the traffic participant to the camera perspective through the traffic participant trajectory, the rotation matrix and translation vector of the traffic participant in the second output result, and the bounding box of the traffic participant, to obtain the world coordinates of the traffic participant relative to the ego-vehicle in the first output result; a third processing unit, used to fuse the output of the neural rendering network dedicated to static driving scene reconstruction with the output of the neural rendering network dedicated to dynamic traffic participant reconstruction based on the world coordinates, and render the fusion result to obtain each frame of perspective image under the camera perspective; a fourth processing unit, used to, based on each frame of perspective image under the camera perspective, splice each frame of the output image data into video data of the autonomous driving scene associated with the target information and associated with the ego-vehicle.

[0100] like Fig.11 As shown, an embodiment of the present application provides an electronic device, including a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0101] Memory 1103, used for storing computer programs;

[0102] In one embodiment of the present application, the processor 1101 is used to execute the program stored in the memory 1103 to implement the steps of a method for generating video data for an autonomous driving scene provided by any of the aforementioned method embodiments, and the effects achieved are similar and will not be repeated here.

[0103] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a method for generating video data for an autonomous driving scenario provided in any of the aforementioned method embodiments are implemented.

[0104] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0106] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0107] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating video data for autonomous driving scenarios, It is characterized in that include: Input text information used to describe the driving scene of the self-vehicle into a large language model LLM, and output target information for constructing the autonomous driving scene; wherein the LLM is a vertical domain LLM for generating the trajectory of traffic participants; the target information includes information of the static driving scene associated with the self-vehicle, and information of other physical entities associated with the self-vehicle and the self-vehicle in the driving scene; The target information is respectively input into a neural rendering network dedicated to static driving scene reconstruction and a neural rendering network dedicated to dynamic traffic participant reconstruction, to obtain output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction; wherein the neural rendering network dedicated to static driving scene reconstruction is trained based on data associated with static driving scenes in previously collected traffic road data, and the neural rendering network dedicated to dynamic traffic participant reconstruction is trained based on data of the self-vehicle and other associated physical entities in previously collected traffic road data; The output results of the neural rendering network dedicated to reconstructing static driving scenes and the neural rendering network dedicated to reconstructing dynamic traffic participants are fused to obtain video data of the autonomous driving scene associated with the target information and the ego vehicle.

2. The method according to claim 1, It is characterized in that The vertical LLM used to generate the trajectories of traffic participants is obtained, including: Collecting the traffic road data, wherein the traffic road data includes image data and point cloud data associated with the vehicle under different scenes, different lighting conditions, and different sensors; Extracting first data from the traffic road data, wherein the first data includes a self-vehicle trajectory, a traffic participant trajectory, semantic information, and sensor parameters associated with the self-vehicle; The base large language model LLM is fine-tuned based on the first data to obtain a vertical domain LLM for generating traffic participant trajectories, wherein the vertical domain LLM for generating traffic participant trajectories has the ability to generate time-series postures.

3. The method according to claim 2, It is characterized in that The text information is input into the vertical domain LLM for generating the trajectory of the traffic participants, and the target information output for constructing the autonomous driving scenario includes: The text information is input into a vertical LLM for generating traffic participant trajectories, and target information for constructing an autonomous driving scenario is output, wherein the information of the static driving scene associated with the ego vehicle in the target information includes semantic information and sensor parameters associated with the ego vehicle, and the information of the ego vehicle and other physical entities associated with the ego vehicle in the target information includes the ego vehicle trajectory and the traffic participant trajectory.

4. The method according to claim 2, It is characterized in that Training the neural rendering network dedicated to the static driving scene reconstruction includes: Constructing a first training data set based on the collected traffic road data, wherein the first training data set includes a plurality of first training samples, each of which includes image data of a current frame, point cloud data of the current frame, a camera intrinsic parameter matrix associated with the current frame, a rotation matrix of the current frame, and a translation vector of the current frame; The first training samples are preprocessed, and the neural rendering network dedicated to static driving scene reconstruction is trained based on the preprocessed first training samples.

5. The method according to claim 4, It is characterized in that Training a neural rendering network dedicated to dynamic traffic participant reconstruction involves: Obtain the bounding boxes and position transformation matrices of the vehicle and traffic participants in the collected traffic road data through automatic annotation; Extracting the image data and point cloud data of the traffic participant through the bounding box; The image data and the point cloud data are converted to the same coordinate system by using the posture transformation matrix to obtain a second training set; wherein the second training set includes a plurality of second training samples, and the second training samples include the image data of the traffic participant, the point cloud data of the traffic participant, the camera intrinsic parameter matrix associated with the traffic participant, the rotation matrix of the traffic participant, and the translation vector of the traffic participant; The neural rendering network dedicated to dynamic traffic participant reconstruction is trained based on the second training samples in the second training set.

6. The method according to claim 5, It is characterized in that The target information is input into a neural rendering network dedicated to static driving scene reconstruction and a neural rendering network dedicated to dynamic traffic participant reconstruction respectively, and the output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction are obtained, including: Inputting the traffic road data into a neural rendering network dedicated to static driving scene reconstruction to obtain a first output result, wherein the first output result represents a static scene of the autonomous driving scene; The traffic road data is input into a neural rendering network dedicated to dynamic traffic participant reconstruction to obtain a second output result, wherein the second output result represents the traffic participants in the autonomous driving scenario.

7. The method according to claim 6, It is characterized in that The output results of the neural rendering network dedicated to reconstructing static driving scenes and the neural rendering network dedicated to reconstructing dynamic traffic participants are fused to obtain video data of the autonomous driving scene associated with the target information and associated with the self-driving vehicle, including: generating a camera perspective associated with the ego vehicle based on the ego vehicle trajectory; The traffic participant is transformed to the camera perspective by using the traffic participant trajectory, the rotation matrix and translation vector of the traffic participant in the second output result, and the bounding box of the traffic participant to obtain the world coordinates of the traffic participant relative to the vehicle in the first output result; The output of the neural rendering network dedicated to reconstructing static driving scenes is fused with the output of the neural rendering network dedicated to reconstructing dynamic traffic participants based on the world coordinates, and the fusion result is rendered to obtain each frame of the perspective image under the perspective of the camera; Based on each frame of perspective image under the camera perspective, each frame of image data output is spliced ​​into video data of an automatic driving scene associated with the target information and associated with the self-vehicle.

8. A video data generation device for an autonomous driving scene, It is characterized in that include: A first processing module is used to input text information used to describe the driving scene of the self-vehicle into a large language model LLM, and output target information used to construct the autonomous driving scene; wherein the LLM is a vertical domain LLM used to generate the trajectory of traffic participants; the target information includes information of a static driving scene associated with the self-vehicle, and information of other physical entities associated with the self-vehicle and the self-vehicle in the driving scene; The second processing module is used to input the target information into a neural rendering network dedicated to static driving scene reconstruction and a neural rendering network dedicated to dynamic traffic participant reconstruction, respectively, to obtain output results of the neural rendering network dedicated to static driving scene reconstruction and the neural rendering network dedicated to dynamic traffic participant reconstruction; wherein the neural rendering network dedicated to static driving scene reconstruction is trained based on data associated with static driving scenes in traffic road data collected in the early stage, and the neural rendering network dedicated to dynamic traffic participant reconstruction is trained based on data associated with the self-vehicle and other physical entities associated with the self-vehicle in traffic road data collected in the early stage; The third processing module is used to fuse the output results of the neural rendering network dedicated to reconstructing the static driving scene and the neural rendering network dedicated to reconstructing the dynamic traffic participants to obtain video data of the automatic driving scene associated with the target information and the self-driving vehicle.

9. The device according to claim 8, It is characterized in that The device also includes: A collection module, used to collect the traffic road data, wherein the traffic road data includes image data and point cloud data associated with the vehicle under different scenes, different lighting conditions and different sensors; An extraction module, configured to extract first data from the traffic road data, wherein the first data includes a trajectory of the vehicle, a trajectory of a traffic participant, semantic information, and sensor parameters associated with the vehicle; The third processing module is used to fine-tune the base large language model LLM based on the first data, wherein a vertical domain LLM for generating traffic participant trajectories is obtained, wherein the vertical domain LLM for generating traffic participant trajectories has the ability to generate time-series postures.

10. The device according to claim 9, It is characterized in that The first processing module comprises: The first processing unit is used to input the text information into a vertical LLM for generating traffic participant trajectories, and output target information for constructing an autonomous driving scenario, wherein the information of the static driving scene associated with the ego vehicle in the target information includes semantic information and sensor parameters associated with the ego vehicle, and the information of the ego vehicle and other physical entities associated with the ego vehicle in the target information includes the ego vehicle trajectory and the traffic participant trajectory.

11. An electronic device, include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to execute the video data generation method for the autonomous driving scene according to any one of claims 1-7.

12. A computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the video data generation method for an autonomous driving scene as described in any one of claims 1-7.

Images