Road scene simulation video generation method based on diffusion model

Abstract

The invention discloses a road scene simulation video generation method based on a diffusion model, and the method comprises the steps: mapping a content-controllable video synthesized by simulation software into a simulation video with lifelike images and dynamic coherence through combining an automatic driving simulation software data synthesis method and a video simulation algorithm; an expansion self-attention module is adopted to decompose a video simulation algorithm into a frame-by-frame image simulation conversion process, so that the simulation difficulty of a long video is reduced; a combination of a semantic segmentation map, a depth estimation map and a content description text is used as abstract representation of image semantic content and a space structure, the content and style of the image are decoupled, and content information of an original image is effectively kept in the image simulation conversion process; an open-source model is adopted to generate a semantic segmentation map, a depth estimation map and a content description text of a real image as training data, so that the construction difficulty of a training data set is reduced. The method has the advantages that the generated simulated video image is vivid, dynamic and coherent, and the video content is controllable.

Description

Technical Field

[0001] The invention relates to the technical field of digital video generation, and in particular to a method for generating simulated videos of road scenes based on a diffusion model. Background Art

[0002] Research on autonomous driving technology based on deep learning increasingly relies on large-scale and high-quality road scene video training data. However, it is very expensive to collect and annotate real road video data in the real world to build a training data set, and it is difficult to reproduce certain accident scenes that are very helpful for training. The emergence of simulation software can partially make up for this deficiency. For example, Carla, an autonomous driving simulation software based on the game engine Unreal Engine, can simulate various driving scenarios under user control and generate synthetic data of multiple simulated sensors. However, there are obvious differences between this type of synthetic data and real data, including differences in visual texture and data distribution. This makes it difficult for autonomous driving algorithms to adapt to real scenes after training with synthetic data. Simulated data refers to synthetic data that has reduced the difference with real data through simulation processing. The use of simulated data can make autonomous driving algorithms better adapt to real scenes and show performance similar to or even better than when trained entirely with real data. Therefore, the simulated data generation method is considered to be an effective way to solve the problem of lack of real data and has been widely used in the fields of natural language processing and computer vision.

[0003] There are two main types of methods for generating realistic video data based on diffusion models, one based on video generation models and the other based on image generation models. The method using the video generation model generates realistic videos with the advantages of high inter-frame consistency and video dynamic coherence, but is limited by the performance bottleneck of existing computers and cannot generate long videos. The method based on the image generation model generates single-frame images with good quality and strong controllability, but lacks semantic consistency and dynamic coherence between multiple frames. At the same time, both types of methods based on generative models face serious "hallucination" problems, that is, phenomena that do not conform to the laws of the physical world or violate human cognition often appear in the video.

[0004] The above-mentioned problems have resulted in a lack of realistic video data generation methods suitable for road scenes. In this context, the present invention proposes a realistic video data generation method that combines autonomous driving simulation software and a generative model: first, the autonomous driving simulation software is used to collect synthetic video data and corresponding annotations, and then a video simulation algorithm that combines an image generation model and an extended self-attention module is used to convert the synthetic video into a realistic video, while retaining its content and structural information, reducing its domain difference with the real video, and finally, the annotated data collected in the simulation software is combined to form a realistic video dataset. Summary of the invention

[0005] The purpose of the present invention is to propose a method for generating road scene simulated videos based on a diffusion model, in view of the shortcomings of existing methods for generating road scene simulated videos in terms of video duration and semantic consistency between frames.

[0006] The main technical problem solved by the present invention is how to generate large-scale, scene-rich road scene simulated video data to form an autonomous driving simulated video data set. This goal can be broken down into three sub-problems: first, how to generate realistic video data and high-precision data annotations at low cost; second, how to improve the quality of videos generated by the algorithm in the absence of a large amount of real data with high-precision annotations as training samples; and finally, how to effectively constrain the model during the generation process so that the data it generates meets the input conditional information, including content consistency constraints and video dynamic coherence constraints. The present invention proposes a method for generating road scene simulated videos based on a diffusion model. By combining a simulation software sampling method and a deep learning conditional generation method, large-scale and high-quality road scene simulated video data and annotations are generated to provide data support for the research of autonomous driving technology and alleviate data shortage and long-tail problems.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A method for generating a road scene simulated video based on a diffusion model, the method comprising the following steps:

[0009] S101: Collect and clean real image data of road scenes;

[0010] S102: extracting a semantic segmentation map, a depth estimation map and content description text annotations from the real image processed in step S101;

[0011] S103: using the real image data of the road scene obtained in step S101 and the annotations obtained in step S102, training a multi-condition image generation model;

[0012] S104: Building a road scene in the autonomous driving simulation software and collecting virtual sensor data to obtain a synthetic video of the road scene and a corresponding semantic segmentation map sequence and depth estimation map sequence;

[0013] S105: extracting key frames from the synthesized video obtained in step S104, and extracting content description text from the key frames using an open source image description model;

[0014] S106: Through the video simulation algorithm, use the multi-conditional image generation model trained in step S103 as a simulation model for a single-frame image, and use the synthetic video and semantic segmentation map sequence, depth estimation map sequence, and key frame content description text obtained in steps S104 and S105 as input to generate a simulated video that is consistent with the semantic content and spatial structure of the synthetic video.

[0015] Furthermore, in step S103, the network structure of the multi-conditional image generation model includes: a pre-trained image generation network, a conditional control network, and a conditional feature fusion module; the conditional feature fusion module includes a semantic label mapping table and a multi-level feature fusion network.

[0016] Furthermore, the conditional feature fusion module is used to perform feature conversion and multi-level feature fusion on the input semantic segmentation map and the normalized depth estimation map, and construct a fused feature tensor that has both semantic information and spatial structure information; the conditional control network is used to guide the image generation process of the pre-trained image generation network according to the fused feature tensor, and help the model generate realistic images that meet the input control conditions in terms of semantic content and spatial structure.

[0017] Furthermore, in step S103, the multi-conditional image generation model obtains the latent space feature code z0 for the input original real image after passing through the VAE encoder during the single-step training process, randomly samples a denoising step t∈[1,999], and adds random noise that obeys the standard Gaussian distribution to z0 according to the forward process formula of DDPM. Get the noise code in is the preset noise coefficient determined by the denoising step t; the noise code z t The denoised time feature vector time_emb corresponding to the denoising step t, the text feature vector text_emb obtained by the content description text c after the CLIP-text encoder, the semantic segmentation map seg, and the depth estimation map depth are input into the multi-conditional image generation model. The pre-trained image generation network of the model outputs the predicted value ∈ of random noise ∈ under various control conditions θ (z t , t, c, sem, depth), use the predicted value and the actual added random noise ∈ to calculate the mean square error loss; the loss function is as follows:

[0018]

[0019] Use the loss function to perform reverse gradient propagation and update the parameter weights of the conditional control network and the conditional feature fusion module, thus completing the single-step training process. Repeat the above process until the loss function converges and the model training is completed.

[0020] Furthermore, in step S104, the synthesized video data, semantic segmentation map sequence, and depth estimation map sequence data are all generated by autonomous driving simulation software.

[0021] Furthermore, the step S106 includes the following sub-steps:

[0022] S106-1: Replace the self-attention module in the multi-conditional image generation model with an extended self-attention module; the operation process of the extended self-attention module is expressed as:

[0023]

[0024] where h i represents the state quantity when the i-th frame image in the video enters the module; Q(·), K(·), V(·) represent the linear layers that map the state quantity to Query, Key, and Value tensors respectively; d represents the number of feature channels of Query, Key, and Value tensors, and f represents the number of image frames that effectively participate in the operation;

[0025] S106-2: Synthesize the video V s =[I 0 , I 1 , ..., I n ]After passing through the VAE encoder frame by frame, the latent space feature encoding sequence is obtained Where [·] represents an ordered set, I i represents the i-th frame image of the video, Indicates the noise code corresponding to the i-th frame image at the t-th denoising step, when t = 0 Also known as the latent space feature encoding of the image, ε encoder (·) represents the VAE encoder; the latent space feature encoding of the image is pre-noised using the DDIM inversion method to obtain a noisy coding sequence The semantic segmentation map sequence corresponding to the synthetic video is denoted as M s =[sem 0 , sem 1 , ..., sem n ], the depth estimation map sequence is recorded as D s =[depth 0 , depth 1 , ..., depth n ], the content description text sequence is recorded as C s =[c 0 , c 1 , ..., c n ], where sem i , depth i , c iRespectively represent the semantic segmentation map, depth estimation map, and content description text corresponding to the i-th frame image;

[0026] S106-3: Represents the input of the i-th frame image in the video, and inputs the first k frames of the synthesized video into [inps 0 , ..., inps k-1 ] are spliced ​​in the batch dimension and input into the model, and the denoising process of the diffusion model is performed. The cache state of all self-attention modules in the model is set to a copy of the input state, and the input state is saved as a temporary file at the same time. The model outputs the first k frames of the realistic video [I sim 0 , ..., I sim k-1 ], the subscript sim represents a simulated image;

[0027] S106-4: Perform simulation conversion on the synthesized video batch by batch to obtain a complete simulation video: input k frames of images into the model, and read the temporary file saved by the previous batch of images as an additional input of the expansion-self-attention module; the model outputs k frames of simulation video, and repeats until all frames of the video are processed to obtain a complete simulation video V sim =[I sim 0 , ..., I sim n ];

[0028] S106-5: For the synthetic video data obtained in step S104, the processes from step S106-2 to step S106-4 are performed one by one, and the obtained simulated video data and the corresponding annotated data constitute a complete simulated video data set.

[0029] The beneficial effect of the present invention is that a method for generating realistic videos of road scenes based on a diffusion model is obtained. The present invention combines autonomous driving simulation software and generative model technology, and uses a video editing method based on an image generation model and an extended self-attention module to improve the realistic effect of the realistic video and the semantic consistency between video frames. The present invention has the advantages of realistic and dynamically coherent realistic videos and controllable video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a block diagram of the implementation architecture of the road scene simulation video generation method proposed by the present invention;

[0031] Figure 2 This is a flow chart of the method for generating road scene simulation video proposed by the present invention;

[0032] Figure 3A schematic diagram of using a large language model to extract content description text of an image in the present invention;

[0033] Figure 4 This is a network structure diagram of the multi-conditional image generation model proposed in the present invention;

[0034] Figure 5 Schematic diagram of the training process of the multi-conditional image generation model proposed in the present invention;

[0035] Figure 6 This is a schematic diagram of the algorithm flow of the video simulation algorithm proposed by the present invention;

[0036] Figure 7 Schematic diagram of the operation process of the extended self-attention module used in the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementations. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation and specific operation process. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0038] The main parts of the implementation steps of the present invention are as follows Figure 1As shown, it includes: 1. Autonomous driving simulation software, such as open source Carla and other simulation software. This type of autonomous driving simulation software usually has multiple road scene maps, which can simulate the road scene and the behavior of virtual vehicles and virtual pedestrians therein, and generate synthetic images and precise annotations of road scenes through virtual sensors, such as semantic segmentation maps and depth estimation maps. In the present invention, this type of autonomous driving simulation software is used to generate synthetic videos of road scenes and semantic segmentation map sequences and depth estimation map sequences; 2. Real image datasets of road scenes, such as open source datasets such as Cityscapes and BDD100K, or real image datasets collected by themselves; 3. Open source models and tools, including semantic segmentation models, depth estimation models and image description models, which are used by the present invention to extract semantic segmentation maps, depth estimation maps and image content description texts required in the model training process. The semantic segmentation model used in this embodiment is Mask2Former, the depth estimation model is DepthAnything, and the image description model is GPT4-o; 4. Multi-conditional image generation model. The present invention proposes a multi-conditional image generation model based on the Stable Diffusion model with semantic segmentation map, depth estimation map, and image content description text as control conditions, which is used to generate a realistic image with similar semantic content and spatial structure to the reference image; 5. Video simulation algorithm. The present invention proposes a video simulation algorithm based on the above-mentioned multi-conditional image generation model. The algorithm combines the extended self-attention module and the DDIM inversion method to generate realistic videos.

[0039] For more information about the autonomous driving simulation software Carla, please refer to: Dosovitskiy A, Ros G, Codevilla F, et al. CARLA: An open urban driving simulator [C] / / Conference on robot learning. PMLR, 2017: 1-16.

[0040] For more information about the Cityscapes dataset, please refer to: Cordts M, Omran M, Ramos S, et al. The cityscapes dataset for semantic urban scene understanding[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 3213-3223.

[0041] For the open source driving dataset BDD100k, please refer to: Yu F, Chen H, Wang X, et al. Bdd100k: A diverse driving dataset for heterogeneous multitask learning [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 2636-2645.

[0042] For detailed information and usage of the semantic segmentation model Mask2Former, please refer to the paper: Cheng B, Misra I, Schwing AG, et al. Masked-attention mask transformer for universal image segmentation [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 1290-1299.

[0043] For detailed information and usage of the depth estimation model DepthAnything, please refer to the paper: Yang L, Kang B, Huang Z, et al. Depth anything: Unleashing the power of large-scale unlabeled data [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 10371-10381.

[0044] For detailed information and usage of the image description model GPT4-o, please refer to the website: https: / / platform.openai.com / organization / usage.

[0045] For the Stable Diffusion model and its specific network structure, please refer to the paper: Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[0046] For detailed information about the DDIM inversion method, please refer to the paper: Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in neural information processing systems, 2020, 33: 6840-6851.

[0047] The present invention provides a method for generating road scene simulation video based on a diffusion model, the flow chart is as follows: Figure 2 As shown, the specific steps include:

[0048] like Figure 2 As shown, in step S101, a real road scene image dataset is collected and cleaned, which specifically includes the following sub-steps:

[0049] S101-1: Collect open source real image data of road scenes. In this embodiment, the subsets of the BDD100K dataset, BDD10K and Cityscapes dataset, are collected, with about 10,000 real images. The road scene refers to the road scene from the driving perspective taken by the on-board camera.

[0050] S101-2: Cleaning the data set, manually screening the collected real image data set, selecting images with clear weather, soft light, stable and clear image quality, and removing images with serious artifacts or light spots. In this embodiment, about 8,500 real images are retained after screening.

[0051] After step S101, a real image dataset with high sample quality is obtained.

[0052] like Figure 2As shown, in step S102, the semantic segmentation map, depth estimation map and content description text of the real image are obtained. The real image uses the real image data collected and cleaned in step S101; if the original open source real data set provides a certain type of annotation data, it can be directly obtained from the original data set; otherwise, open source models and tools can be used to extract annotations for the real image. Specifically, it includes the following sub-steps:

[0053] S102-1: Obtain the semantic segmentation map of the real image and save it in the same representation method. The BDD10K dataset and Cityscapes dataset collected in this embodiment have provided semantic segmentation map annotations, which can be directly obtained from the dataset; for open source real image data that lacks semantic segmentation map annotations, a semantic segmentation model, such as Mask2Former, can be used to process the real image and obtain it. Since the semantic categories and category numbers of the semantic segmentation maps in different open source datasets are different, in order to unify the representation method for use in subsequent steps, this embodiment uses the semantic segmentation map representation method in the Cityscapes dataset as the standard, and saves the acquired semantic segmentation maps as RGB images with 20 colors, where each color represents a semantic category.

[0054] Step S102-2: Obtain the depth estimation map of the real image, uniformly represent it using relative depth, and save it as a grayscale image. For real image samples that lack depth estimation maps in open source datasets, an open source depth estimation model, such as DepthAnything, can be used to extract relative depth estimation maps for real images.

[0055] Step S102-3: Obtain image content description text of the real image. Use an open source image description model or a large language model with image understanding capabilities, such as GPT4-o, to generate a comprehensive description of the image content under the prompt. The prompt words used in this embodiment and the generated content description text are as follows: Figure 3 shown.

[0056] After step S102, the semantic segmentation map, depth estimation map and content description text corresponding to each image in the real image dataset are obtained, and the representation methods of the three types of annotations are unified.

[0057] In step S102, when the real image data is incompletely or inaccurately labeled, an open source model is used to obtain the semantic segmentation picture sequence and depth estimation picture sequence of the image data, as well as the content description text, so that the real data can be combined with these labeled data to effectively train the multi-conditional image generation model.

[0058] like Figure 2As shown, in step S103, the real road scene image dataset collected in step S101 and the three types of labeled data obtained in step S102 are used to train a multi-conditional image generation model. The multi-conditional image generation model specifically refers to the model proposed in the present invention, and its model structure is as follows Figure 4 As shown in Figure 2, the single-step training process of the model is as follows: Figure 5 shown.

[0059] like Figure 4 As shown in Figure 1, the model mainly consists of three parts: pre-trained image generation network, conditional control network, and conditional feature fusion module. The pre-trained image generation network uses the pre-trained Stable Diffusion 1.5 model; the conditional feature fusion module contains a semantic label mapping table (in Figure 4 The corresponding sem2fea module in the image processing module and a network structure for multi-level feature fusion are used. The module performs feature conversion and multi-level feature fusion on the input semantic segmentation map and the normalized depth estimation map to construct a fused feature tensor with both semantic information and spatial structure information. The conditional control network guides the image generation process of the pre-trained image generation network according to the fused feature tensor, and helps the model to generate realistic images that meet the input control conditions in terms of semantic content and spatial structure, thereby solving the problem of semantic changes and spatial structural distortion of the generated image. Among them, the semantic label mapping table sem2fea needs to use the Clip-text encoder to extract text features from the semantic category, and then use the principal component analysis method to reduce the high-order feature vector to a lower dimension. In this embodiment, the semantic label of each pixel in the semantic segmentation map is mapped to a 32-dimensional feature vector after passing through the mapping table.

[0060] like Figure 5 As shown in the figure, in the single-step training process, the multi-conditional image generation model obtains the latent space feature code z0 after the input original real image passes through the VAE encoder, randomly samples a denoising step t∈[1,999], and adds random noise that follows the standard Gaussian distribution to z0 according to the forward process formula of DDPM. Get the noise code in is the preset noise coefficient determined by the denoising step t. t The denoised time feature vector time_emb corresponding to the denoising step t, the text feature vector text_emb obtained by the content description text c through the CLIP-text encoder, the semantic segmentation map seg, and the depth estimation map depth are input into the above-mentioned multi-conditional image generation model. The Unet network of the model (the Unet network is part of the pre-trained image generation network stablediffusionv 1.5) outputs the predicted value ∈ of random noise ∈ under various control conditions θ (zt , t, c, sem, depth), use the predicted value and the actual added random noise ∈ to calculate the mean square error loss, the loss function is as follows:

[0061]

[0062] Use this loss function to perform reverse gradient propagation and update the parameter weights of the conditional control network and feature fusion module, completing the single-step training process. Repeat the above process until the loss function converges and the model training is completed.

[0063] For detailed information on the specific training process of the diffusion model and the forward and backward processes of DDPM, please refer to the paper Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models [J]. Advances in neural information processing systems, 2020, 33: 6840-6851.

[0064] After step S103, the present invention constructs a multi-condition image generation model that can generate a simulated image of a road scene using a semantic segmentation map, a depth estimation map, and an image content description text as input conditions.

[0065] In step S103, the conditional feature fusion module takes the semantic segmentation map and the normalized depth estimation map as module input, uses the semantic label mapping table to convert meaningless semantic labels into high-dimensional feature vectors with semantic features, and adopts a multi-level feature fusion network to fuse semantic features and depth estimation features. The feature map output by the module realizes the effective fusion of semantic information and spatial structure information.

[0066] In step S103, the semantic label mapping table obtained based on the pre-trained multimodal model and the high-dimensional feature dimensionality reduction method can select the feature vector of the corresponding label in the mapping table as the semantic feature vector of the pixel according to the semantic label of each pixel in the semantic annotation map, and then construct a semantic feature map with stronger spatial and semantic representation capabilities from the semantic annotation map.

[0067] like Figure 2 As shown, in step S104, the road scene is built using the autonomous driving simulation software and the required sensor data is collected, including the synthetic video data from the vehicle camera perspective, the synthetic semantic segmentation map sequence, and the synthetic depth estimation map sequence. The autonomous driving simulation software used in this embodiment is Carla, version 0.9.14. It specifically includes the following sub-steps:

[0068] S104-1: Build a road scene, set the weather and time of the scene, and generate virtual pedestrians and virtual vehicles in the scene. For example, call the open source city scene map Town02 in Carla as the road scene, set the weather to sunny, the time to afternoon, and the number of virtual pedestrians and virtual vehicles in the scene to 50.

[0069] S104-2: Control pedestrians and vehicles in the scene to simulate traffic behavior in real road scenes, such as calling the automatic path-finding algorithm in Carla to make each pedestrian and vehicle move towards the destination randomly selected by the algorithm in the scene.

[0070] S104-3: Mount a virtual camera on a designated vehicle, configure virtual camera parameters, and obtain a synthetic video from the perspective of the vehicle-mounted camera and a corresponding semantic segmentation map sequence and depth estimation map sequence.

[0071] For detailed information about the maps, algorithms, and virtual cameras provided by Carla 0.9.14, see the online documentation: https: / / carla.readthedocs.io / en / 0.9.14 / .

[0072] After step S104, the synthetic video in the road scene and the corresponding semantic segmentation map sequence and depth estimation map sequence are obtained through the autonomous driving simulation software.

[0073] In step S104, the synthesized video data is collected from the autonomous driving simulation software, and the related annotation data is also generated by the simulation software.

[0074] like Figure 2 As shown, in step S105, the content description text of the video key frame is obtained. The video key frame refers to the key frame extracted by the equal interval sampling method from the synthetic video in step S104. In this embodiment, the virtual camera frame rate is set to 20 frames per second, so the key frame is extracted from the synthetic video by extracting one frame every five frames. The method for extracting the content description text from the key frame image is the same as that in step S102-3. For details, see Figure 2 .

[0075] like Figure 2 As shown, in step S106, a video simulation algorithm is used to generate simulated video data to construct a simulated video data set. The algorithm flow of the video simulation algorithm proposed in the present invention is as follows: Figure 6 As shown, the algorithm uses the multi-conditional image generation model trained in step S103 as a simulation model for a single-frame image, and uses the synthetic video and three types of annotations obtained in steps S104 and S105 as input to generate a simulation video that is consistent with the semantic content and spatial structure of the synthetic video. Specifically, it includes the following sub-steps:

[0076] S106-1: Replace all self-attention modules in the multi-conditional image generation model with extended self-attention modules. The parameters of the latter are the same as the former, but the operation process within the module is different. The operation process is as follows Figure 7 As shown in the figure, the additional input "cached hidden states" are passed through the linear layer of the self-attention module to obtain the corresponding key tensor and value tensor, which are concatenated with the key tensor and value tensor corresponding to the original state, and then the self-attention operation is performed. The original state is the state of the current frame image input to the algorithm, and the additional input cache state is the corresponding state of the f-1 frame image before the current frame image. The operation process of the extended self-attention module can be expressed as:

[0077]

[0078] where h i represents the state quantity when the i-th frame image in the video enters the module; Q(·), K(·), V(·) represent the linear layers that map the state quantity to Query, Key, and Value tensors, respectively; d represents the number of feature channels of Query, Key, and Value tensors, and f represents the number of image frames that effectively participate in the operation.

[0079] S106-2: Prepare the input data of the algorithm. s =[I 0 , I 1 , .., I n ]After passing through the VAE encoder frame by frame, the latent space feature encoding sequence is obtained Where [·] represents an ordered set, I i represents the i-th frame image of the video, Indicates the noise code corresponding to the i-th frame image at the t-th denoising step, when t = 0 Also known as the latent space feature encoding of the image, ε encoder (·) represents the VAE encoder. The latent space feature encoding of the image is pre-noised using the DDIM inversion method to obtain a noise code sequence The semantic segmentation map sequence corresponding to the synthetic video is denoted as M s =[sem 0 , sem 1 , ..., sem n ], the depth estimation map sequence is recorded as D s =[depth 0 , depth 1 , ..., depth n ], the content description text sequence is recorded as C s =[c0 , c 1 , ..., c n ], where sem i , depth i , c i They respectively represent the semantic segmentation map, depth estimation map, and content description text corresponding to the i-th frame image.

[0080] S106-3: Warm up the model and handle the edge cases of the algorithm. Represents the input of the i-th frame image in the video, and inputs the first k frames of the synthesized video into [inps 0 , ..., inps k-1 ] are spliced ​​in the batch dimension and input into the model, and the denoising process of the diffusion model is performed. The cache state of all self-attention modules in the model is set to a copy of the input state (excluding the corresponding current frame), and the input state is saved as a temporary file at the same time. The model outputs the first k frames of the realistic video [I sim 0 , ..., I sim k-1 ], the subscript sim indicates that it is a simulated image.

[0081] S106-4: Perform simulation conversion on the synthesized video batch by batch to obtain a complete simulation video. Each time, k frames of image input are sent to the model, and the temporary file saved by the previous batch of images is read as an additional input to the extended self-attention module. The model outputs k frames of simulation video each time, and after repeated execution, the complete simulation video V is obtained. sim =[I sim 0 , ..., I sim n ].

[0082] S106-5: Execute the process from step S106-2 to step S106-4 one by one on the synthetic video data generated in step S104 to obtain a large amount of simulated video data, which together with the corresponding annotated data constitute a complete simulated video data set.

[0083] It should be noted that each time the k frames of the composite video are converted, the process is repeated until the video ends. If the video has n frames, the process is performed (n / / k+1) times until all frames of the input video are processed.

[0084] After step S106, the present invention implements a method for generating road scene simulated videos based on a diffusion model, which combines the data acquisition method of the autonomous driving simulation software and the generative model data generation method. The method converts the synthetic video data sampled from the simulation software into simulated video data with better visual texture, and together with the corresponding annotated data, forms a simulated video data set.

[0085] In step S106, the video simulation algorithm mentioned uses the intermediate state quantities generated during the conversion of the first few frames of images to guide the conversion process of subsequent image frames, and adopts an extended self-attention module and a state quantity caching method to process the video frames in batches, effectively improving the inter-frame consistency during video simulation conversion, and can realize the simulation conversion of long videos.

[0086] In step S106, a large amount of synthetic video data of road scenes is used to generate a large amount of simulated road video data through the method, and a simulated video data set of road scenes is formed. Compared with the synthetic video generated by simulation software, the simulated video has better visual texture and smaller difference with the real video data, and can be effectively used in the training and evaluation of autonomous driving algorithms.

[0087] The above are only specific implementation methods of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A method for generating realistic video of road scenes based on a diffusion model, characterized in that: The method comprises the following steps: S101: Collect and clean real image data of road scenes; S102: extracting a semantic segmentation map, a depth estimation map and content description text annotations from the real image processed in step S101; S103: using the real image data of the road scene obtained in step S101 and the annotations obtained in step S102, training a multi-condition image generation model; S104: Building a road scene in the autonomous driving simulation software and collecting virtual sensor data to obtain a synthetic video of the road scene and a corresponding semantic segmentation map sequence and depth estimation map sequence; S105: extracting key frames from the synthesized video obtained in step S104, and extracting content description text from the key frames using an open source image description model; S106: Through the video simulation algorithm, use the multi-conditional image generation model trained in step S103 as a simulation model for a single-frame image, and use the synthetic video and semantic segmentation map sequence, depth estimation map sequence, and key frame content description text obtained in steps S104 and S105 as input to generate a simulated video that is consistent with the semantic content and spatial structure of the synthetic video.

2. The method for generating road scene simulated video based on diffusion model according to claim 1, characterized in that: In step S103, the network structure of the multi-conditional image generation model includes: a pre-trained image generation network, a conditional control network, and a conditional feature fusion module; the conditional feature fusion module includes a semantic label mapping table and a multi-level feature fusion network.

3. The method for generating road scene simulated video based on diffusion model according to claim 2, characterized in that: The conditional feature fusion module is used to perform feature conversion and multi-level feature fusion on the input semantic segmentation map and the normalized depth estimation map, and construct a fused feature tensor with both semantic information and spatial structure information; The conditional control network is used to guide the image generation process of the pre-trained image generation network according to the fused feature tensor, and help the model generate realistic images that meet the input control conditions in terms of semantic content and spatial structure.

4. The method for generating road scene simulated video based on diffusion model according to claim 3, characterized in that: In step S103, during the single-step training process, the multi-conditional image generation model obtains the latent space feature code z0 for the input original real image after passing through the VAE encoder, randomly samples a denoising step t∈[1,999], and adds random noise that follows the standard Gaussian distribution to z0 according to the forward process formula of DDPM. Get the noise code in is the preset noise coefficient determined by the denoising step t; the noise code z t The denoised time feature vector time_emb corresponding to the denoising step t, the text feature vector text_emb obtained by the content description text c after the CLIP-text encoder, the semantic segmentation map seg, and the depth estimation map depth are input into the multi-conditional image generation model. The pre-trained image generation network of the model outputs the predicted value ∈ of random noise ∈ under various control conditions θ (z t , t, c, sem, depth), use the predicted value and the actual added random noise ∈ to calculate the mean square error loss; the loss function is as follows: Use the loss function to perform reverse gradient propagation and update the parameter weights of the conditional control network and the conditional feature fusion module, thus completing the single-step training process. Repeat the above process until the loss function converges and the model training is completed.

5. The method for generating road scene simulated video based on diffusion model according to claim 1, characterized in that: In step S104, the synthesized video data, semantic segmentation map sequence, and depth estimation map sequence data are all generated by autonomous driving simulation software.

6. The method for generating road scene simulated video based on diffusion model according to claim 1, characterized in that: The step S106 includes the following sub-steps: S106-1: Replace the self-attention module in the multi-conditional image generation model with an extended self-attention module; the operation process of the extended self-attention module is expressed as: where h i represents the state quantity when the i-th frame image in the video enters the module; Q(·), K(·), V(·) represent the linear layers that map the state quantity to Query, Key, and Value tensors respectively; d represents the number of feature channels of Query, Key, and Value tensors, and f represents the number of image frames that effectively participate in the operation; S106-2: Synthesize the video V s =[I 0 , I 1 , ..., I n ]After passing through the VAE encoder frame by frame, the latent space feature encoding sequence is obtained Where [·] represents an ordered set, I i represents the i-th frame image of the video, Indicates the noise code corresponding to the i-th frame image at the t-th denoising step, when t = 0 Also known as the latent space feature encoding of the image, ε encoder (·) represents the VAE encoder; the latent space feature encoding of the image is pre-noised using the DDIM inversion method to obtain a noisy coding sequence The semantic segmentation map sequence corresponding to the synthetic video is denoted as M s =[sem 0 , sem 1 , ..., sem n ], the depth estimation map sequence is recorded as D s =[depth 0 , depth 1 , ..., depth n ], the content description text sequence is recorded as C s =[c 0 , c 1 , ..., c n ], where sem i , depth i , c i Respectively represent the semantic segmentation map, depth estimation map, and content description text corresponding to the i-th frame image; S106-3: Represents the input of the i-th frame image in the video, and inputs the first k frames of the synthesized video into [inps 0 , ..., inps k-1 ] are spliced ​​in the batch dimension and input into the model, and the denoising process of the diffusion model is performed. The cache state of all self-attention modules in the model is set to a copy of the input state, and the input state is saved as a temporary file at the same time. The model outputs the first k frames of the realistic video [I sim 0 , ..., I sim k-1 ], the subscript sim represents the simulated image; S106-4: Send k frames of image input to the model, and read the temporary file saved by the previous batch of images as additional input to the expansion-self-attention module; the model outputs k frames of realistic video, and repeats until all frames of the video are processed to obtain a complete realistic video V sim =[I sim 0 , ..., I sim n ]; S106-5: For the synthetic video data obtained in step S104, the processes from step S106-2 to step S106-4 are performed one by one, and the obtained simulated video data and the corresponding annotated data constitute a complete simulated video data set.

Images