Automatic driving data synthesis method based on point cloud and semantic graph

Abstract

The invention discloses an automatic driving data synthesis method based on a point cloud and a semantic graph, relates to the technical field of AI automatic driving, and solves the technical problems in the prior art that a large amount of difference exists between a synthesized picture and a real picture, and a video generation model is unstable and cannot be accurately controlled. Comprising the following steps: carrying out geometric representation on static features and dynamic features in a traffic scene; rendering the geometric representation into a continuous point cloud graph and a semantic graph at a fixed frame rate and at a fixed interval according to the position of the own vehicle camera; then, rendering synthesized point cloud data and a semantic graph are used as condition injection of a generative model to guide data synthesis, and more real automatic driving data are obtained; according to the method, the point cloud data and the semantic graph are introduced, so that the problems that a video generation model is unstable and cannot be accurately controlled are effectively solved; and the difference between a simulation scene and a real scene in the automatic driving field can be reduced on the premise of not depending on any real data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of AI automatic driving, and particularly relates to an automatic driving data synthesis method based on a point cloud and a semantic graph. BACKGROUND

[0002] Sim2Real is a technology widely used in intelligent simulation to provide virtual data for AI training, which can be well used for robot training but performs poorly in the field of automatic driving. Compared with the training environment of robots, the environment required for automatic driving is more complex and open, and there are more noises, and the visual model of automatic driving is more dependent on the details of the picture. Although the rendering engine based on Unity, Unreal, etc. can well simulate real scenes, there is still a gap in light, modeling, and texture distribution with the real world, and the cost of synthesizing data conforming to the sample distribution of the real world is very huge; generative models are AI models that can generate video data by specifying input conditions such as prompt words and semantic graphs, and are mainly divided into diffusion models and autoregressive models. The picture realism of the video data generated by the generative model is higher than that of the rendering engine, and is more consistent with the picture distribution of the real world and has a very low cost, but there are unstable generated content and uncontrollable situations. SUMMARY

[0003] The application provides an automatic driving data synthesis method based on a point cloud and a semantic graph, which solves the technical problems of a large number of differences between the synthesized picture and the real picture and the instability and uncontrollable of the video generation model in the prior art.

[0004] To achieve the above purpose, the application adopts the following technical solutions: In a first aspect, an automatic driving data synthesis method based on a point cloud and a semantic graph is provided, comprising: S1: geometrically representing static features and dynamic features in a traffic scene; S2: rendering the geometric representation into continuous point cloud graphs and semantic graphs at a fixed frame rate and interval from the position of the camera of the ego vehicle; S3: generating a generative model based on a DiT architecture, namely a DiT model; S4: inputting the point cloud graph data and the semantic graph data generated in step S2 and the prompt words used to describe the scene into the DiT model, and outputting a piece of automatic driving video data; The prompt words are used to describe the weather, light, and color information in the scene; the point cloud graph data is used to guide the model to generate basic shape information, and the semantic graph data is used to provide information of lane lines and more accurate control of vehicle positions that cannot be included in the point cloud data.

[0005] Based on the above technical scheme, in the automatic driving data synthesis method based on point cloud and semantic graph provided in the application, the point cloud data and the semantic graph are generated by rendering the geometric representation of the static features and the dynamic features of the automatic driving data, and finally a generative model (DiT model) is used to synthesize more realistic automatic driving data; the point cloud data and the semantic graph rendered and synthesized are used as the condition injection of the generative model to guide data synthesis, effectively solving the problems of instability and inability to accurately control of the video generation model; meanwhile, the simulation and rendering of the point cloud data and the semantic graph by the self-defined data can be generated without relying on existing data, effectively reducing the differences between the simulation scene and the real scene in the automatic driving field, increasing the generalization of the simulation scene, also reducing the time and economic cost required for the construction of the simulation scene, and a large amount of automatic driving long-tail data can be synthesized to construct a high-quality and controllable automatic driving data set.

[0006] Further, the geometric representation of the static features and the dynamic features specifically includes: S11: geometric representation of the road center line: the geometric shape of the road center line is represented by a plurality of curves with a fixed curvature radius R; a road center line P(x, y)=RoadRefLine(s) with a curvature radius R and a length len is drawn; s is the distance of any point on the center line from the starting point of the road; The calculation method of x, y at any position is as follows: x=(cos(hdg0+angle_s-π / 2)-cos(hdg0-π / 2)) R+x0; y=(sin(hdg0+angle_s-π / 2)-sin(hdg0-π / 2)) R+y0; Where s0≤s≤s0+len, angle_s=(s-s0) / R; x0, y0 are the starting world coordinates of the curve, hdg0 is the heading angle at P(x0, y0), len is the length of the road, and s0 is the length of the starting point of the curve from the starting point of the road. S12: geometric representation of the lane line: the representation of the lane line and the road edge is represented by the width of the corresponding lane, and the starting position and the ending position of the road; the world coordinates of the lane line at the lane center line s are defined by calculating the lane width at s; the specific representation is: width=Laneline(s); Laneline(s)=a+b d_s+c d_s²+d d_s³ifs_start≤s<s_endelse0; where s start, s end are the start and end positions of the lane line relative to the road center line; a, b, c, d are preset coefficients of the multiple function for calculating the lane width; d s = s - s start, s is the distance from the lane line to the starting point of the road; Zebra crossing, road marking is represented by a rectangle composed of four point coordinates in the world coordinate system, and the stop line is represented by a line segment composed of two points; S13: Motion state representation of traffic participants: by defining the Bezier curve of the action trajectory , and the cubic spline curve s(t) to specify the position to which the traffic participant moves at different times; where the calculation method of s(t) is:

[0007] where is the initial position of the vehicle, is related to the initial speed of the vehicle, is related to the initial acceleration of the vehicle, is positively related to the change of the vehicle acceleration; represents the time value corresponding to the starting time of the kth curve; for the kth curve, the distance s is obtained according to the time t at any time; the s is brought into the Bezier curve in the motion trajectory for interpolation, that is, the position information of the traffic participant at any time can be represented.

[0008] Further, the geometric representation is rendered into continuous point cloud and semantic graph at the position of the self-vehicle camera at a fixed frame rate and interval, specifically including: Semantic graph rendering: for lane line data, a line segment set is obtained by sampling at an interval of 0.2m; for traffic participants, a line segment set is also obtained according to the pre-set 3D hollow bounding box; the line segment set in the 3D space is projected onto the two-dimensional plane through the camera projection matrix Mproj, and then rendered; wherein the Mproj matrix is calculated through the calibration parameters of the camera; After projection transformation, for each pixel point, the color of the pixel point is judged by calculating whether the line segment passes through the pixel point, and the transparency of the color is calculated according to the distance between the line segment and the pixel point; Point cloud rendering: the intersection points of the ray and the road, traffic participants, and triangular faces of building trees are calculated to simulate point cloud data; After obtaining the point set of the point cloud, the point set is projected through the Mproj matrix, and the depth d of each point is calculated, and d / 200 is written into the pixel, and then a 3 3 convolution kernel is used to denoise and fill the holes of the point cloud.

[0009] Further, before rendering the semantic graph and the point cloud data, preparation work is also needed, including: According to the fixed delta_t interval, the dynamic characteristics of the dynamic object are sampled, and the position and rotation information of each entity in each frame are calculated; the position and rotation information of the ego vehicle is processed separately, and the position information of the virtual vehicle-mounted camera is combined to calculate the camera projection matrix Mproj; Processing of road geometry information; the geometry information of the road is represented by an implicit function, which needs to be discretized before rendering.

[0010] Further, the specific steps of discretizing the road geometry information are as follows: The Roadrefline(S) and Laneline(S) are sampled at a fixed delta_t interval, and the point set Points of the lane line is calculated for road-related semantic graph rendering; The leftmost lane line point and the rightmost lane line point of the road and the leftmost and rightmost points of the next sampling point are used to generate two triangular surfaces to generate the road mesh for point cloud data generation.

[0011] Further, the architecture of the DiT model in step S3 is as follows: The original video is processed into tokens by the tokenizer, then added noise, then blocked and flattened into one-dimensional tokens; after adding absolute position encoding to the tokens, the tokens are calculated by the multi-head self-attention layer; The embedding vector of the noise adding step is also used as a token, which is added to the multi-head self-attention layer, so that the model can make different predictions according to different noise adding steps; Then pass through a multi-head cross-attention layer to fuse the tokens of the prompt words; after several times of prediction and denoising, the completely denoised tokens are obtained; Then the one-dimensional tokens are spatially reorganized according to the blocking and flattening rules before denoising to restore the TxCxHxW structure; then the tokenizer decoder is used to decode to generate video data; where T is the number of image frames, C is the dimension of the token, H and W are the height and width of the video.

[0012] Further, step S3 also includes training of the DiT model, and the specific steps are as follows: Before training, the traffic scene video is labeled to produce the prompt words of the scene; Then the video data is processed by the tokenizer, and Gaussian noise is added to the data in the forward process; , wherein βt is the noise schedule parameter, controlling the strength of noise added at each step; x t is the result of adding noise t times in the diffusion model, x t-1 is the result of adding noise t-1 times; After several iterations, the data is completely degraded into random noise, and then the DiT denoiser is used in the reverse process to predict the completely degraded noise, to predict the noise at the corresponding step, and to evaluate the prediction through the loss function.

[0013] Further, the loss function is defined as , where D θ is the noise prediction model, n is the noise in the forward noise adding process, σ is the standard deviation of the noise, and x0 is the tokens that are expected to be restored.

[0014] Further, the DiT denoiser replaces the main network of the UNet used to predict noise in the DDPMs with a Transformer architecture to denoise the tokens; the DiT denoiser predicts the denoised image through a multi-head self-attention mechanism, and then fuses the tokens of the prompt through a cross-attention layer to serve as the input of the next layer.

[0015] Further, the ControlNet architecture is used in the DiT model to fuse the point cloud image and the semantic image to control the denoising process, which specifically includes: The point cloud image and the semantic image are also input in the form of tokens after using a tokenizer, and after passing through a linear layer, they are also formed into a noise image by superimposing the noise of the video tokens; then the DiT denoiser is used for denoising, and then the result predicted by the ControlNet is changed through a linear layer before guiding the result of the video tokens; specifically, n=D(x σ ,σ,c) where c refers to the injected control condition, x σ is the current noise image, σ is the standard deviation of the noise, and n is the predicted noise.

[0016] The application provides an automatic driving data synthesis method based on point cloud and semantic image, which can use multiple curves to represent a simulated traffic road scene. By defining the curvature, curve starting point, curve length, and starting point heading angle, any road center line can be accurately represented. Then, according to the road center line, the road width at different positions is defined, and then the road lane and lane line are defined. In addition, the Bezier curve and the cubic spline curve are used to simulate the vehicle motion state.

[0017] Subsequently, at a fixed frame rate and interval, the geometric representation is rendered into continuous point cloud maps and semantic maps based on the position of the ego camera. Through the simulation and rendering of point cloud data and semantic maps by custom data, it can be generated without relying on existing data, effectively reducing the gap between simulation scenes and real scenes in the field of autonomous driving, improving the generalization ability of simulation scenes, and significantly reducing the time and economic benefits required for simulation scene construction.

[0018] At the same time, the rendered synthesized point cloud data and semantic maps are injected as conditions of the generative model to guide data synthesis, effectively solving the instability and difficulty of precise control of the video generation model. In addition, this method can also synthesize a large amount of autonomous driving long-tail data, thereby constructing a high-quality and highly controllable autonomous driving dataset.

[0019] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of a feature or beneficial effect means that the specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and beneficial effects described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creating any inventive labor.

[0021] Figure 1 The framework diagram of the present application, a point cloud and semantic map based autonomous driving data synthesis method.

[0022] Figure 2 The flowchart of the present application, a point cloud and semantic map based autonomous driving data synthesis method.

[0023] Figure 3 The flowchart of another point cloud and semantic map based autonomous driving data synthesis method of the present application.

[0024] Figure 4 A schematic diagram of a traffic road scene of the present application. Figure 5 A flowchart of the ControlNet architecture in the present application. DETAILED DESCRIPTION

[0025] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this paper only describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean: A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0026] It should be noted that in this application, "exemplary" or "for example" means to serve as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0027] To solve the technical problems of the difference between the synthesized picture and the real picture in the prior art and the instability and inability to accurately control of the video generation model, the embodiments of the present application provide an automatic driving data synthesis method based on point cloud and semantic graph, which comprises: Geometric representation of static features and dynamic features in a traffic scene; Rendering of point cloud and semantic graph; that is, rendering the geometric representation into continuous point cloud and semantic graph data at the position of the self-car camera at a fixed frame rate and interval; Generating a generative model based on DiT architecture, namely DiT model; Wherein the input of the DiT model is the point cloud and semantic graph data generated in the previous step and the prompt words used to describe the scene, and the prompt words are used to describe the weather, light, color and other information in the scene; the output is a piece of automatic driving video data; wherein the point cloud data is used to guide the model to generate basic shape information, and the semantic graph data is used to provide information of lane lines and more accurate control of vehicle position which cannot be contained in the point cloud data.

[0028] Based on this, the method generates point cloud maps and semantic map data by rendering the geometric representations of the static features and dynamic features of the autonomous driving data, and finally uses a generative model (DiT model) to synthesize more realistic autonomous driving data; the point cloud map and semantic map data rendered and synthesized are used as the conditional injection of the generative model to guide data synthesis, so that the differences between the simulation scene and the real scene in the field of autonomous driving can be reduced without relying on any real data, the generalization of the simulation scene is increased, the time and economic cost required for the construction of the simulation scene can be reduced, a large amount of autonomous driving long-tail data can be synthesized to construct a high-quality and controllable autonomous driving data set; As Figures 1 to 2 shown, the application embodiment provides an autonomous driving data synthesis method based on point cloud and semantic map, comprising: S1: geometric representation of static features and dynamic features in the traffic scene; Among them, the static features include traffic road elements, traffic participants and non-traffic elements; the data of the traffic road elements is expanded to both sides with the road center line as the reference point; Among them, the dynamic features are the motion state of the traffic participants, which is used to describe the position and rotation of the traffic participants at any time point t; Specifically, the traffic road elements include but are not limited to lane lines, zebra crossings, stop lines, road edges and road signs; the traffic participants include motor vehicles, pedestrians and non-motor vehicles; the non-traffic elements include trees, buildings and street lamps, etc.; In some implementations, the geometric features of the traffic participants can be divided into 3D bounding boxes and three-dimensional mesh bodies representing shapes, and the position information is represented by dynamic features; In some implementations, the non-traffic elements include trees, buildings, street lamps, etc.; the geometric features of the non-traffic elements are represented by three-dimensional mesh bodies and position rotation scaling in world coordinates; S2: render the geometric representation into continuous point cloud maps and semantic map data at the position of the self-car camera with a fixed frame rate and interval; Specifically, in this way, virtual data highly similar to real-world scenes can be generated. When generating point cloud maps, according to the position, shape and other information of each element in the geometric representation, the distribution of each point in space can be accurately simulated to present rich details such as road undulations and vehicle contours. For semantic maps, according to the semantic information of different elements given by the geometric representation, the points in the point cloud map are classified and labeled to clearly distinguish traffic road elements, traffic participants and non-traffic elements, etc., providing accurate and comprehensive data support for the training of subsequent autonomous driving algorithms. At the same time, the setting of fixed frame rate and interval ensures the coherence and stability of the generated data, making it more consistent with the data collection characteristics in the actual driving process; In some implementations, the semantic map is mainly composed of line segments, 3D hollow bounding boxes and 2D rectangular planes, which are used to mark the traffic conditions of the scene; for the signs on the road, a rectangle is drawn using its four vertices to mark it; for different elements in the scene, different colors are used to draw to mark them; It is worth noting that the point cloud data is a set of points in three-dimensional space, and the rendering of the point cloud map is relatively complex. In order to simulate real point cloud data, the virtual point cloud is first simulated by relying on the mesh data of the entity and the constructed laser calibration data. The point cloud data is defined as a set of 3D coordinates. The obtained point cloud data is finally rendered on an image. After camera projection transformation is performed on the point cloud data, the depth value is written into the pixel; In some implementations, due to the problem of point cloud density, there may be empty pixels, so a convolution kernel is used to convolve the image to reduce the generation of empty pixels; S3: generate a generative model based on the DiT architecture, namely the DiT model; Specifically, the tokenizer is the basis of the DiT architecture, which encodes continuous video data into tokens that can be processed by the Transformer model, also known as the latent representation of video data, and can convert tokens back into video data after denoising; In some implementations, the tokenizer is an encoder-decoder architecture. For any given , the tokenizer encoder ( ) can output tokens , and then the decoder ( ) is used to restore the tokens to video data, i.e. ; wherein H, W, and T are the height, width, and frame number of the video, and C is the dimension of a single token; In some implementations, the DiT model uses a continuous tokenizer to process the original data and encode the video into TxCxHxW tokens; It is worth noting that the value range of H (video height) and W (video width) is usually set according to specific requirements and hardware conditions, for example: the video height H is set between 240 pixels and 1080 pixels, and the video width W is set between 320 pixels and 1920 pixels; such value range can meet the requirements of most automatic driving algorithms for image resolution, and can avoid the problem of excessive consumption of computing resources caused by too large image; Specifically, T represents the frame number related parameter in the time dimension, which is set between 1 and 10, and is used to simulate the vehicle driving situation in a short time; It is worth noting that since tokens is a (T, H, W) shaped cube and the Transformer receives 1-dimensional data, it is necessary to flatten the data into 1-dimensional data by performing a blocking process; Specifically, the blocking process first divides the (T, H, W) shaped cube data into multiple small blocks according to certain rules, and these small blocks maintain certain continuity in the spatial and temporal dimensions; then these small blocks are connected in turn to form a 1-dimensional data sequence, so that the Transformer can receive and process these data; Specifically, the size of the block is set to t x h x w; where t is between 1 and 3, h is between 1 and 4, and w is between 1 and 5; then the original latent representation is finally flattened into a 1-dimensional sequence with a length of T x H x W ÷ (t x h x w); Through this block flattening method, the dimension complexity of the data can be effectively reduced while retaining key spatial and temporal feature information, providing more efficient and representative input data for subsequent model processing; In some implementations, the original latent representation is flattened into a 1-dimensional sequence with a length of T x H x W ÷ 4 using a block with T = 1, H = 2, and W = 2; Specifically, the original latent representation is blocked according to the time dimension T = 1, the height dimension H = 2, and the width dimension W = 2, and each block has a size of 1 x 2 x 2. After feature extraction and processing of each block, the results of these blocks are connected in turn to form a 1-dimensional sequence with a length of (T x H x W) ÷ 4; Specifically, the DiT-based denoiser is the core structure of the DiT model; the DiT denoiser is a variant of the DDPMs-based denoising model, which works by predicting the noise of a pure noise image through the model, then subtracting the predicted noise from the noise image, and then restoring the original image through multiple predictions; In some implementations, the DiT denoiser replaces the backbone network of the UNet used to predict noise in DDPMs with a Transformer architecture, allowing it to denoise tokens instead of 2-dimensional images; the DiT denoiser predicts the denoised image through a multi-head self-attention mechanism, and then fuses the tokens of the prompt words through a cross-attention layer to serve as input for the next layer; S4: input the point cloud graph and semantic graph data generated in step S2 and the prompt words used to describe the scene into the DiT model, and output an automatic driving video data; Specifically, after inputting the point cloud map data, semantic map data, and prompt words into the DiT model, the model first preprocesses these input data to ensure they meet the input requirements of the model. Then, using the powerful capabilities of the DiT denoiser, the model gradually denoises and extracts features from these data, deeply understanding the complex relationships in the data through multi-head self-attention mechanisms. After fusing the tokens with the prompt words, the model generates intermediate representations that are closer to real-world scenarios. Finally, after a series of processing and optimization, the DiT model outputs a high-quality autonomous driving video data that not only contains rich scene information but also accurately reflects the behavior of autonomous vehicles in various situations. Compared with traditional rendering-based methods, this method effectively reduces the difference between the synthesized picture and the real picture. Moreover, by introducing point cloud maps and semantic maps, the instability and lack of precise control of video generation models are effectively addressed. Furthermore, the simulation and rendering of point cloud maps and semantic maps using custom data can be generated without relying on existing data. In a possible implementation manner of the embodiment of the present application, in combination with Figure 2 Compared with Figure 3 The S1 can be implemented through the following S11, S12, and S13, which are described in detail as follows. S11: Geometric representation of the road center line Specifically, the geometric shape of the road center line can be represented by a plurality of curves with a fixed curvature radius R. By defining the curvature, the curve starting point, the curve length, and the starting heading angle, any road center line can be represented. For example, a road center line with a curvature radius R and a length len is drawn, where s is the distance of any point on the center line from the starting point of the road; P(x, y) = RoadRefLine(s). where x and y at any position are as follows: x = (cos(hdg0 + angle_s - π / 2) - cos(hdg0 - π / 2)) R + x0 y = (sin(hdg0 + angle_s - π / 2) - sin(hdg0 - π / 2)) R + y0 where s0 ≤ s ≤ s0 + len, angle_s = (s - s0) / R; x0 and y0 are the starting point world coordinates of the curve, hdg0 is the heading angle at P(x0, y0), len is the length of the road, and s0 is the length of the curve starting point from the road starting point. S12: Geometric representation of the lane line Specifically, the representation of the lane line and the road edge is represented by the width of the corresponding lane, and the starting position and the ending position of the road where it is located; the world coordinates of the lane line at the lane center line s are defined by calculating the lane width at s; and the specific representation is as follows: width=Laneline(s); Laneline(s)=a+b d_s+c d_s²+d d_s³ifs_start≤s<s_endelse0; wherein s_start, s_end are the starting and ending positions of the lane line relative to the road center line; d_s=s-s_start, s is the distance from the starting point of the road where the lane line is located; Specifically, a, b, c, and d are preset coefficients of a multiple function for calculating the lane width; these preset coefficients are set according to the actual road conditions, and different road types, design standards, etc. will make the coefficient values different. Through such function calculation, the width of the lane line at different positions s can be accurately obtained, and then the position and shape of the lane line in the overall road scene can be accurately determined by combining the world coordinates of the lane center line, thereby providing key lane line geometric information for automatic driving data synthesis.

[0029] In addition, as shown in Figure 4 a road can have multiple lanes, and multiple lane lines LaneLinei can be defined, i is the i-th lane of the current road, i=0 is the center lane line, i>0 is the lane line in the same direction as the road, and i<0 is the reverse lane line; RoadRefLine represents the road center line with a curvature radius R and a length len; then the coordinates of any lane line at any position s can be calculated; Exemplarily, the coordinate calculation method of the lane line with i=2 at s is as follows: x=x0+cos(hdq0+angle_s-π / 2) width; y=y0+cos(hdq0+angle_s-π / 2) width wherein width=Laneline1(s-s_start)+Laneline2(s-s_end) is the sum of the widths of lane 1 and lane 2; angle_s=(s-s0) / R; x0, y0 are the starting world coordinates of the curve; hdq0 is the heading angle at P(x0, y0); In some implementations, the zebra crossing and the road marking are represented by a rectangle composed of four point coordinates in the world coordinate system, and the stop line is represented by a line segment composed of two points. S13: the motion state of the traffic participant represents: a Bezier curve defining the action trajectory and a cubic spline curve s(t) to specify the position to which the traffic participant moves at different times; It should be noted that the facing direction of all traffic participants is consistent with the motion direction, so the first derivative of the Bezier curve is the motion direction at time t, which is used to represent the rotation information of the traffic participant at any time; wherein the calculation method of s(t) is:

[0030] It should be noted that the advantage of using a cubic spline curve is that the polynomial coefficients therein have a characteristic meaning for the position, speed, and acceleration of the vehicle; wherein is the initial position of the vehicle, is related to the initial speed of the vehicle, is related to the initial acceleration of the vehicle, is positively related to the change in acceleration of the vehicle; Specifically, k generally represents the number of curve segments, which is a positive integer, starting from 1 and increasing sequentially, for distinguishing different curve segments; represents the time value corresponding to the start time of the kth curve segment; wherein k = 0, 1, …, n-1; n is a positive integer and n≥1; It should be noted that when n takes different values, the constructed curve can simulate the motion trajectories of traffic participants with different complexity levels. When n is small, it is suitable for describing relatively simple and regular motion trajectories, such as straight line motion or uniform circular motion, etc., at this time the curve can present the motion characteristics more simply; and when n is large, it can accurately depict motion trajectories with complex changes, including multiple acceleration, deceleration, turning, etc., to meet the accuracy requirements of traffic participant motion simulation in different scenarios.

[0031] Therefore, the above formula 2 can also be written as q k (t)=s k +v k (t-t k )+a k (t-t k ) 2 / 2+j k (t-t k ) 3 / 3; for the kth curve segment, according to t at any time, the distance s can be obtained; by bringing s into the Bezier curve in the motion trajectory for interpolation, the position information of the traffic participant at any time can be represented; Because of the continuity of the distance and time of motion, for n curves, by defining v0 and a0 when k=0, s0=0, t0=0 and s1 and t1 when k=1, s1=v0 (t1-t0)+a0 (t1-t0) 2 / 2+j0 (t1-t0) 3 / 3, j0 can be solved; According to the kinematics of vehicles, it is believed that the speed and acceleration of vehicles also have continuity, that is, for any two adjacent curves, the acceleration and speed at the end of the previous curve are the same as those of the next curve; According to the above restrictions, v k+1 =v k +a k t d +j k t d 2 / 2, a k+1 =a k +j k t d . Through recursion, the parameters of all curves are obtained, that is, q k (t) is represented by two points p(s k ,t k ) and p(s k+1 ,t k+1 ); Therefore, for any curve s(t), it can be represented by defining n+1 control points p(s k ,t k ) and initial velocity v0 and acceleration a0 at k=0; these control points not only determine the shape of the curve, but also, in combination with the initial velocity v0 and acceleration a0, completely describe the motion state of the traffic participant on the curve. In this way, the invention can accurately simulate the position, speed and acceleration of the traffic participant at different time points, providing accurate data support for the decision of the automatic driving system. At the same time, this representation method greatly simplifies the calculation process and improves the efficiency of data processing; Exemplarily, first, set delta_t=0.0333, sample the cubic curve to obtain the distance s moved by all traffic participants in each frame; then, sample the corresponding Bezier curve according to s to determine its position; then, derive the Bezier curve to obtain its direction F; for the Bezier curve, the DeCasteljau algorithm is used to calculate the coordinates of the point at s.

[0032] After obtaining the position and orientation information, these data are integrated in time sequence to form a complete motion state description of all traffic participants in each frame. Then, according to the motion state description, the position, posture and other information of the traffic participants in each frame are presented in the form of an image by using a graphic rendering technique. For each frame image, rendering is performed according to the resolution requirement of 720P to ensure the definition and quality of the image. Finally, the rendered images are combined in sequence according to the frame rate requirement of 30 FPS to generate a 720P video with a duration of 10s, which can truly and accurately simulate the motion of the traffic participants in the actual scene.

[0033] In a possible implementation manner of the embodiment of the present application, the S2 can be implemented through the following S21 and S22, which are specifically described as follows. S21: semantic graph rendering Specifically, for the lane line data, a line segment set is obtained by sampling at intervals of 0.2m; for the traffic participants, a line segment set is also obtained according to a pre-set 3D hollow bounding box; the line segment set in the 3D space is projected onto a two-dimensional plane through a camera projection matrix Mproj, and then rendering is performed; wherein the Mproj matrix is calculated through the calibration parameters of the camera; It should be noted that the camera calibration parameters include intrinsic parameters and extrinsic parameters, the intrinsic parameters mainly describe the internal geometry and optical characteristics of the camera, such as focal length, principal point position, etc., and the extrinsic parameters represent the position and attitude of the camera in the three-dimensional space. By accurately obtaining these calibration parameters and using a specific algorithm to calculate, an accurate Mproj matrix can be obtained, which accurately projects the points in the three-dimensional space onto the two-dimensional plane, providing a reliable basis for subsequent semantic graph rendering; After projection transformation, for each pixel point, the color of the pixel point is determined by calculating whether the line segment passes through the pixel point, and the transparency of the color is calculated according to the distance between the line segment and the pixel point; Specifically, according to the geometric information of the lane line, the range of the pixel points whose colors need to be determined is determined; for each pixel point in the range, the line segment and pixel point position relationship is calculated by using the coordinate information of the pixel point and the line segment in the lane line point set; if the line segment passes through the pixel point, the pixel point is assigned a corresponding color value according to a pre-set rule, thereby completing the determination of the color of the pixel point; For example, the pixel point is assigned a corresponding color according to the object type represented by the line segment (such as a lane line or a traffic participant), for example, the lane line is set to yellow, and the traffic participants are set to different colors according to their categories; It should be noted that if the line segment passes through the pixel point, the transparency of the color of the pixel point is calculated according to the distance between the line segment and the pixel point, and a preset transparency calculation model, such as a linear interpolation model, is combined to calculate the transparency of the color of the pixel point, so as to determine the final color presented by the pixel point; the closer the distance, the lower the transparency, the more obvious the color, the farther the distance, the higher the transparency, the lighter the color, and the visual effect of the object in the scene is more realistically simulated in this way, so that the rendered semantic map is closer to the actual scene. S22: Point cloud map rendering: Specifically, the intersection points of the triangular faces of the mesh of the ray, the road, the traffic participants, the buildings and the trees are simulated. It should be noted that before calculating the intersection points, an octree is first used to construct an acceleration structure for the triangular faces. The specific method is to uniformly divide the space into eight cubes with a fixed size, and then recursively further subdivide each child cube into eight smaller cubes until a suitable size is reached, thereby forming a tree structure, in which the leaf nodes represent the smallest child cubes. By judging the child cube in which the triangular face is located, if the triangular face spans multiple child cubes, it is considered to exist in each child cube.

[0034] Taking the parameters of the Velodyne AT128 laser radar as an example, the point cloud is generated. The field of view angle of AT128 is 120°x25.4°, and the angular resolution is 0.1°x0.2°. According to this, 152,400 rays of different angles are generated, and the intersection calculation is performed with the triangular faces of the mesh. With the aid of the pre-constructed octree, starting from the cube of the root node, it is gradually downward to the leaf node, and finally the intersection with the triangular faces contained in the leaf node cube is calculated, so as to determine the intersection point of each ray and the triangular face, and only the nearest intersection point with a distance less than 200 meters is retained.

[0035] After obtaining the point set of the point cloud, the point set is projected by the same Mproj matrix in S21, and the depth d of each point is calculated. d / 200 is written into the pixel, and then a 3 3 convolution kernel is used to denoise and fill the holes of the point cloud map; Specifically, the convolution kernel is a 3x3 matrix with a center weight of 4 and eight adjacent point weights of 1. Such a design effectively smooths noise points and fills data missing areas while preserving the main features of the point cloud, making the point cloud map more complete and accurate; In a possible implementation manner of the embodiment of the present application, preparation work is also required before the semantic map and the point cloud map are rendered, specifically including: The dynamic characteristics of the dynamic object are sampled according to the interval of the fixed delta_t, and the position and rotation information of each entity in each frame are calculated; the position and rotation information of the ego vehicle is processed separately, and the position information of the virtual vehicle-mounted camera is combined to calculate the camera projection matrix Mproj; Processing of road geometry information; the geometry information of the road is represented by an implicit function, which needs to be discretized before rendering; In a possible implementation manner of the embodiment of the application, the specific steps of discretizing the road geometry information are as follows: For the representation method of the road data in S1, the Roadrefline(S) and Laneline(S) are sampled at a fixed delta_t interval, and the point set Points of the lane line is calculated, which is used for rendering of the road-related semantic graph; Two triangular surfaces are generated using the points of the leftmost lane line and the rightmost lane line and the leftmost and rightmost points of the next sampling point to generate a mesh body of the road for generating point cloud data; The application can realize complete video generation from scratch by introducing implicit representation of traffic scenes and combining rendering technology. This generation method does not depend on existing autonomous driving data, but instead precisely models various elements of traffic scenes such as road layout, vehicle driving trajectory, and pedestrian activity by means of implicit representation, and then uses rendering technology to vividly present these modeling elements as video content. In this way, various complex and diverse traffic scene videos can be flexibly generated to meet different training needs and scene simulation requirements; In addition, by using point cloud maps and semantic graph data, the application can not only accurately control object types, but also to some extent, regulate object shapes; for example, for the object type of vehicles, by adjusting the distribution density of point cloud data and the annotation information in the semantic graph, the type of generated vehicles such as cars, trucks, or buses can be accurately controlled. At the same time, the shape of the vehicle can also be fine-tuned, such as changing the length, width, or height of the vehicle body, so that it is more consistent with the actual vehicle characteristics in a specific scenario. This precise control capability greatly improves the usability and reliability of the synthesized data, providing more abundant and diverse data support for subsequent model training; In a possible implementation manner of the embodiment of the application, the above S3 can be implemented through the following S31 and S32, which are specifically described as follows: S31: Structure of DiT model Specifically, the architecture of the DiT model is as follows: The original video is processed by the tokenizer into tokens, then added with noise, and then blocked and flattened into one-dimensional tokens. The tokens are added with absolute position encoding and calculated through the multi-head self-attention layer. It should be noted that absolute position encoding is a technique for injecting sequence element position information into the model. When processing sequence data, the model itself usually does not have the ability to perceive the position of the elements, and the absolute position encoding assigns a unique encoding vector to each position element, so that the model can identify and utilize the position information of the elements in the sequence. These encoding vectors are usually predefined and sent into the model along with the input data for processing, helping the model better understand the structure and semantics of the sequence data. It should be noted that the embedding vectors of the noise addition step are also tokens, which are added to the multi-head self-attention layer so that the model can make different predictions according to different noise addition steps. Then, the tokens of the prompt words are fused through a multi-head cross-attention layer. After several rounds of prediction and denoising, the fully denoised tokens are obtained. Then, the one-dimensional tokens are spatially reorganized to restore the TxCxHxW structure according to the blocking and flattening rules before denoising. Then, the tokenizer decoder is used to generate video data. S32: Training of DiT model It should be noted that real-world video data including traffic scenes are used for training. Before training, the Qwen2.5-VL model is used to annotate the traffic scene video to generate prompt words for the scene. It should be noted that Qwen2.5-VL is a visual language model open sourced by Ali Tongyiqian on January 28, 2025. Then, the video data is processed by the tokenizer, and Gaussian noise is added to the data during the forward process. where β t is the noise scheduling parameter that controls the strength of noise addition at each step; x t is the result of adding noise t times in the diffusion model, x t-1 is the result after t-1 times of noise addition. After several iterations, the data completely degenerates into random noise, and then the DiT denoiser is used in the reverse process to predict the noise corresponding to each step. The model is evaluated through the loss function. In one possible implementation of the embodiments of the present application, the loss function is defined as , where D θNoise prediction model, n is the noise in the forward noise adding process, sigma is the standard deviation of the noise, x0 is the tokens that you want to restore; In a possible implementation manner of the embodiment of the application, the DiT model uses a ControlNet architecture to fuse the point cloud graph and the semantic graph to control the denoising process, and the specific steps include: Specifically, the structure of the ControlNet is as shown in the figure Figure 5 The point cloud graph and the semantic graph are also used as input in the form of tokens after using a tokenizer, pass through a linear layer, and then form a noise graph by superimposing the noise of the video tokens; then use the DiT denoiser to predict the noise, change the predicted result through a linear layer, and then guide the result of the video tokens; then the comprehensive formula is as follows: n=D(x σ ,σ,c) Here, c refers to the injected control condition, x σ is the current noise graph, sigma is a certain disturbance added by prediction, and the disturbance is a standard deviation of random noise specially added in the model training process to enhance the robustness of the model. The addition of such noise enables the model to exhibit better adaptability and stability when facing various uncertainties in actual data. By adjusting the size of sigma, the degree of disturbance can be controlled, thereby affecting the denoising effect of the model and the quality of the finally generated data. n is the predicted noise; In some implementation manners, the continuous point cloud graph and the semantic graph are also encoded through the tokenizer and then denoised in the DiT denoiser, and then the result predicted by the ControlNet is superimposed on the original denoised result; It should be noted that the guided data source is only used in the first three denoising processes, and in addition, for different data sources and different batches of denoising, a weight is used to guide the influence of different data sources on the original noise graph; For example, for the point cloud graph and the semantic graph, two hyperparameters w1 and w2 are set to respectively control the weights of the point cloud graph and the semantic graph; meanwhile , the hyperparameters are used to control the weight in each superimposition process; h1 and h2 respectively represent the weights of the first three semantic graphs and point cloud graphs in each superimposition with the original noise, and are both hyperparameters; It should be noted that the values of h1 and h2 are adjusted according to the specific task requirements and experimental results, and they have a mutual coordination relationship with w1 and w2; w1 and w2 control the proportion of point cloud graph and semantic graph in the final result as a whole, while h1 and h2 more finely adjust the influence of semantic graph and point cloud graph on the original noise graph in the first three superposition processes. In the training process, by continuously optimizing the combination of these hyperparameters, the synthesized autonomous driving data can better meet the actual scene requirements, improve the quality and usability of the data, and thus enhance the performance and robustness of the DiT model trained based on these data.

[0036] Specifically, the value range of w1 and w2 can be set between 0 and 1 to represent the relative importance of point cloud graph and semantic graph in the superposition process, or grid search, random search or Bayesian optimization can be used to find the hyperparameter combination that optimizes the model performance, or these hyperparameters are set by the skilled person according to the actual situation or obtained through a large amount of data simulation.

[0037] The present application not only integrates advanced real-time rendering technology and cutting-edge artificial intelligence algorithm, but also realizes precise control of content quality and style through highly automated programmatic generation method, while greatly reducing the time input and resource cost required in data construction and content production process. This method can realize complete video generation from scratch by introducing implicit representation of traffic scene and combining rendering technology. In this way, various complex and diversified traffic scene videos can be generated flexibly to meet different training needs and scene simulation requirements. In addition, the DiT architecture used in the present application exhibits excellent spatiotemporal coherence and generation effect in the field of video generation. By rendering and synthesizing point cloud data and semantic graph as the conditional injection of generative model to guide data synthesis, not only can the object type be accurately controlled, but also the object shape can be regulated to a certain extent. This precise control capability greatly improves the usability and reliability of the synthesized data. At the same time, since this method does not rely on real data, it can flexibly generate autonomous driving data in various extreme or rare scenarios, effectively solving the problem of long-tail scene data scarcity and high data collection risk in the field of autonomous driving. In addition, this method is also highly efficient and scalable, capable of quickly generating a large amount of high-quality autonomous driving data to meet the growing needs of autonomous driving research and development, and helping to discover more corner cases in autonomous driving scenarios.

[0038] In the description of the specification, reference to "one embodiment", "an example", "a specific example" or the like means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The appearances of the phrases "in one embodiment", "an example", "a specific example" or the like in various places in the specification are not necessarily referring to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0039] The preferred embodiments of the application disclosed above are only to help explain the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments. Obviously, many modifications and variations can be made in light of the contents of the specification. The specification selects and specifically describes these embodiments in order to better explain the principles and practical application of the application, so that those skilled in the art can well understand and utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. A method for synthesizing autonomous driving data based on point clouds and semantic graphs, characterized in that, Includes the following steps: S1: Geometric representation of static and dynamic features in traffic scenarios; S2: Render the geometric representation into continuous point cloud and semantic graph data at a fixed frame rate and interval based on the position of the vehicle camera; S3: Generate generative models based on the DiT architecture, i.e., DiT models; S4: Take the point cloud map and semantic map data, as well as the prompt words used to describe the scene, as input to the DiT model, and output a segment of autonomous driving video data; The prompt words are used to describe the weather, lighting, and color information in the scene; the point cloud data is used to guide the model to generate basic shape information; and the semantic graph data is used to provide lane line information that cannot be included in the point cloud data and to control the vehicle position more accurately.

2. The method for synthesizing autonomous driving data based on point clouds and semantic graphs according to claim 1, characterized in that, Geometric representation of static and dynamic features, specifically including: S11: Geometric representation of the road centerline: Draw a road centerline P(x,y)=RoadRefLine(s) with radius of curvature R and length len; s is the distance of any point on the centerline from the starting point of the road; The calculation method for x and y at any position is as follows: x=(cos(hdg0+angle_s-π / 2)-cos(hdg0-π / 2)) R+x0; y=(sin(hdg0+angle_s-π / 2)-sin(hdg0-π / 2)) R+y0; Where s0≤s≤s0+len, angle_s=(s-s0) / R; x0,y0 are the world coordinates of the starting point of the curve, hdg0 is the heading angle at P(x0,y0), len is the length of the road, and s0 is the distance between the starting point of the curve and the starting point of the road. S12: Geometric Representation of Lane Lines: Lane lines and curbs are represented by the width of the corresponding lane, as well as the start and end positions of the road. The world coordinates of the lane line at the lane centerline s are defined by calculating the lane width at point s. Specifically, it is represented as follows: width = Laneline(s); Line(s)=a+b d_s+c d_s2+d d_s³ifs_start≤s<s_endelse0; Where s_start and s_end are the start and end positions of the lane line relative to the road centerline; a, b, c, and d are preset coefficients of the multiplication function used to calculate the lane width; d_s = s - s_start, where s is the distance from the starting point of the lane line on the road. S13: Representation of the motion state of traffic participants: The positions moved to by traffic participants at different times are specified by defining the Bézier curve of the movement trajectory and the cubic spline curve s(t); specifically: in This refers to the initial location of the vehicle. Related to the vehicle's initial speed, It is related to the vehicle's initial acceleration. Positively correlated with changes in vehicle acceleration; This represents the time value corresponding to the starting moment of the k-th curve segment; for the k-th curve segment, the distance s is obtained based on t at any time; by substituting s into the Bézier curve in the trajectory for interpolation, the position information of traffic participants at any time can be represented.

3. The method for synthesizing autonomous driving data based on point clouds and semantic graphs according to claim 2, characterized in that, At a fixed frame rate and interval, the geometric representation is rendered into continuous point cloud and semantic graph data based on the position of the vehicle's camera, specifically including: Semantic graph rendering: For lane line data, a set of line segments is obtained by sampling at intervals of 0.2m; for traffic participants, a set of line segments is also obtained based on a pre-defined 3D hollow bounding box; the set of line segments in 3D space is projected onto a two-dimensional plane by the camera projection matrix Mproj for rendering; where the Mproj matrix is ​​calculated using the camera calibration parameters. After projection transformation, for each pixel, the color of the pixel is determined by calculating whether the line segment passes through the pixel, and the transparency of the pixel color is calculated based on the distance between the line segment and the pixel. Point cloud rendering: Point cloud data is simulated by finding the intersection points of rays with the triangular faces of meshes such as roads, traffic participants, buildings, and trees; after obtaining the point set, the point set is projected using the Mproj matrix, and the depth d of each point is calculated, with d / 200 written to the pixel; a 3D matrix is ​​then used. A 3-fold convolutional kernel is used to denoise and fill holes in the point cloud map.

4. The method for synthesizing autonomous driving data based on point cloud and semantic graph according to claim 3, characterized in that, Preparatory work must be done before rendering semantic graphs and point cloud data, including: The dynamic features of the dynamic object are sampled at fixed intervals of delta_t to calculate the position and rotation information of each entity in each frame; the position and rotation information of the vehicle are processed separately and combined with the position information of the virtual vehicle camera to calculate the camera projection matrix Mproj. Processing of road geometry information: The geometric information of the road is represented by implicit functions, which need to be discretized before rendering.

5. The method for synthesizing autonomous driving data based on point cloud and semantic graph according to claim 4, characterized in that, The specific steps for discretizing road geometry information are as follows: Roadrefline(S) and Laneline(S) are sampled at fixed intervals of delta_t to calculate the point set Points of the lane lines, which is used for rendering the road-related semantic graph. Two triangular faces are generated using the points on the leftmost and rightmost lane lines of the road, and the leftmost and rightmost points of the next sampling point, to generate the road mesh for point cloud data generation.

6. The method for synthesizing autonomous driving data based on point cloud and semantic graph according to claim 1, characterized in that, The architecture of the DiT model in step S3 is as follows: The original video is processed into tokens by a tokenizer, then noise is added, and the tokens are divided into blocks and flattened into one-dimensional tokens. After adding absolute position encoding to the tokens, the calculation is performed through a multi-head self-attention layer. The embedding vector from the noise-adding step is also used as a token and added to the multi-head self-attention layer; the tokens of the prompt words are fused through a multi-head cross-attention layer; after several predictions and denoising, the fully denoised tokens are obtained. One-dimensional tokens are spatially reorganized into a T×C×H×W structure using the rules of segmentation and flattening before denoising; the tokenizer's decoder is used to generate video data. Where T is the number of image frames, C is the dimension of the token, and H and W are the height and width of the video.

7. The method for synthesizing autonomous driving data based on point cloud and semantic graph according to claim 6, characterized in that, Step S3 also includes training the DiT model, the specific steps of which are as follows: Before training, the videos of traffic scenes were annotated to generate scene cue words; After the video data is processed by the word segmenter, Gaussian noise is added to the data during the forward pass. ,in These are noise scheduling parameters that control the intensity of noise addition at each step. This is the result of adding noise t times in the diffusion model. This is the result after adding noise t-1 times; After several iterations, the data degenerates into random noise. In the reverse process, the DiT denoiser is used to predict the noise of the completely degenerated noise, and the noise of the corresponding step is predicted and evaluated by the loss function.

8. The method for synthesizing autonomous driving data based on point cloud and semantic graph according to claim 7, characterized in that, The loss function is defined as ,in It is a noise prediction model, where n is the noise in the forward noise addition process, σ is the standard deviation of the noise, and x0 is the tokens that we want to restore.

9. The method for synthesizing autonomous driving data based on point cloud and semantic graph according to claim 7, characterized in that, The DiT denoiser replaces the UNet backbone network used to predict noise in DDPMs with a Transformer architecture to denoise the tokens. The DiT denoiser predicts the denoised map through a multi-head self-attention mechanism, and then fuses the tokens of the cue words through a cross-attention layer as the input of the next layer.

10. The method for synthesizing autonomous driving data based on point cloud and semantic graph according to claim 6, characterized in that, The DiT model uses the ControlNet architecture to fuse point cloud maps and semantic maps to control the denoising process, specifically including: Both the point cloud map and the semantic map are segmented and input as tokens. After passing through a linear layer, noise from the video tokens is superimposed to form a noise map, which is then denoised using a DiT denoiser. The result predicted by ControlNet is transformed through a linear layer before being used to guide the result of the video tokens. Specifically, n = D(x σ ,σ,c); Where c refers to the injection control condition, x σ Here is the current noise map, where σ is the standard deviation of the noise and n is the predicted noise.

Images