Simulation image generation method for automatic driving scene

Abstract

The invention discloses a simulation image generation method for an automatic driving scene. The simulation image generation method is used for generating a simulation automatic driving composite image data set. According to the method, an automatic driving simulation platform and an image simulation algorithm are combined, a multi-modal domain combining a semantic segmentation map, a contour map and text description is constructed between a real image domain and a synthetic image domain, and an image simulation task is converted from mapping from the synthetic image domain to the real image domain to mapping from the multi-modal domain to the real image domain. Therefore, the domain deviation is reduced, and the simulation difficulty is reduced. According to the method, an open source model is adopted to generate a semantic segmentation map, a contour map and text description of a real image as training data, so that a training data set corresponding to a multi-modal domain and a real image domain is constructed, and the training difficulty is reduced. During generation, a semantic segmentation map, a contour map and text description of a synthetic data set are used for generating a simulated image. The method has the advantages that the generated image is good in simulation and controllable in content.

Description

Technical Field

[0001] The present invention relates to the technical field of realistic image data generation, and in particular to a realistic image generation method for an autonomous driving scenario. Background Art

[0003] Real data (especially traffic accident scene data) is difficult to obtain, small in scale, low in quality, difficult to use, and requires a lot of manual annotation. This has led more and more researchers and manufacturers to turn to synthetic data generated by open source autonomous driving simulation platforms to easily generate rich and diverse data.

[0004] The problem with synthetic datasets is that they are not realistic and generalizable enough, and there is a domain deviation from real images. To solve this problem, one approach is to build a more realistic rendering engine to reduce the domain deviation by improving the physical realism of the image. However, engine development often requires a lot of money and human resources. Another approach is to use a deep learning-based image simulation method to convert low-realistic synthetic images into high-realistic simulated images.

[0005] In recent years, artificial intelligence-based generative models have developed rapidly. In the field of image simulation, there are mainly three methods: Generative Adversarial Model (GAN), Diffusion Model, and structural models (such as WCT, PhotoWCT). GAN-type methods can modify textures, but they are highly dependent on the quality of the dataset, lack generalization ability, and cause distortion of contours and semantic information. Most of the existing Diffusion Model-based methods use semantic supervision information to generate images in some weakly supervised scenarios, and the supervision is weak. Structural models can change the tone of the entire picture, which is suitable for real style transfer and artistic style conversion, but cannot replace textures, and the simulation is weak. The simulation task of the autonomous driving dataset requires not only texture changes, but also the preservation of lines, semantics and other information. In the final analysis, more supervision information and a more complex training framework are required.

[0006] In general, in addition to the lack of simulation effect, existing methods are also difficult to recognize image semantics and easily cause semantic distortion.

[0007] The above defects have resulted in a lack of methods for generating realistic image datasets suitable for autonomous driving scenarios. In this context, the present invention proposes a method for generating realistic images by combining an autonomous driving simulation platform with an image simulation algorithm: the autonomous driving simulation platform provides synthetic images, and the image simulation algorithm reduces the domain deviation between the synthetic images and the real images. Summary of the invention

[0008] The purpose of the present invention is to provide a method for generating a simulated image for an autonomous driving scenario in view of the deficiencies in the prior art.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] A method for generating a realistic image for an autonomous driving scene, the method comprising the following steps:

[0011] S101: Setting an autonomous driving simulation platform to ensure that the relative position and parameters of the camera and the vehicle in the autonomous driving simulator are substantially consistent with the real data set used for training; using the autonomous driving method of the autonomous driving simulation platform; setting the camera shutter frequency so that the camera generates synthetic image data as the vehicle travels;

[0012] S102: Using the synthetic image dataset collected in step S101 and the open source generation model, generate its corresponding semantic segmentation map, fine contour map, coarse contour map, and text description to construct a synthetic multimodal dataset;

[0013] S103: Use the open source real autonomous driving dataset and the open source generation model in step S102 to generate corresponding semantic segmentation maps, fine contour maps, coarse contour maps, and text descriptions to construct a real multimodal dataset;

[0014] S104: Using the real multimodal dataset constructed in step S103 as input, construct and train a simulation model of multi-head control;

[0015] S105: Input the multimodal data set obtained in step S102 into the simulated model trained in step S104 to generate a simulated image.

[0016] Furthermore, the training data of the model is a multimodal dataset constructed from a real dataset, and the dataset used when the final realistic image is generated is a synthetic multimodal dataset obtained by an autonomous driving simulation platform.

[0017] Furthermore, in step S103, the open source generative model in step S102 is used to generate a semantic segmentation map, a fine contour map, and a text description corresponding to the real dataset image; the generated semantic segmentation map is used to generate a coarse contour map; the position of the camera relative to the center of the vehicle is added to the text description; and a real multimodal dataset is constructed.

[0018] Furthermore, in step S104, the method for training a realistic model of multi-head control includes: fine-tuning the pre-trained central network model using the RGB image data and text description in the multimodal dataset generated in step S103; fixing the central network weights, and training the contour control network and the semantic segmentation control network using the real multimodal dataset generated in step S103.

[0019] Furthermore, in step S104, the network structure is divided into a central network, a semantic segmentation control network, and a contour control network, wherein the input of the contour control network is a feature fusion tensor of a coarse contour and a fine contour.

[0020] Furthermore, in step S104, the network training uses the following loss function:

[0021]

[0022] Among them, c seg represents the control information of the semantic graph after the encoding network, c hed1 represents the control information of the fine contour image after the encoding network, c hed2 represents the control information of the rough contour image after the encoding network; ∈ is the diffusion noise added by the diffusion model at the current time t, ∈ θ is the noise of ∈ predicted by the network.

[0023] The beneficial effect of the present invention is to obtain a method for generating realistic images for autonomous driving scenarios. The present invention reduces the domain deviation of synthetic data sets and real data sets by combining an autonomous driving simulation platform with an image simulation algorithm, thereby improving the image simulation effect. The present invention has the advantages of good simulation of generated images and controllable content. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a block diagram of the implementation architecture of the method for generating a realistic image for an autonomous driving scenario proposed in the present invention;

[0025] Figure 2 A flowchart of a method for generating a simulated image for an autonomous driving scenario proposed by the present invention;

[0026] Figure 3 Instance diagrams of fine contour maps and semantic segmentation maps used in the realistic image generation method for autonomous driving scenarios proposed in the present invention;

[0027] Figure 4 This is the overall network structure diagram of the image simulation model proposed in the present invention. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0029] There are two unavoidable problems in the current deep learning-based image simulation technology: 1. The domain deviation between the real image domain and the synthetic image domain is large; 2. The image simulation model lacks paired data sets (real images and synthetic images with exactly the same content, semantics, and contours, and one-to-one correspondence). Due to the high complexity of autonomous driving scenes and complex semantic information, paired training data is even more important.

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0031] The main parts of the implementation steps of the present invention are as follows Figure 1 As shown, it includes: 1. Autonomous driving simulation platform, such as open source Carla, Apollo and other autonomous driving platforms. Such autonomous driving simulation platforms usually have various basic autonomous driving algorithms for performance evaluation, various road scenes for virtual vehicles to drive, vehicles with different performance parameters for experiments, and various virtual sensors to provide data for autonomous driving algorithm evaluation. In the present invention, such autonomous driving simulation platform is used to generate synthetic image data sets; 2. Real autonomous driving image data sets, such as open source data sets such as cityscapes, KITTI, etc., can also be self-collected image data sets; 3. Module semantic segmentation map, contour map, text description generation module: In order to solve the problem of lack of paired data sets and large domain deviation between real image domain and synthetic image domain, the present invention uses semantic segmentation map, contour map, and text description as control conditions to construct a mapping between multimodal domain and real image domain. The present invention uses open source pre-trained generation models, such as Stable Diffusion model for semantic style map and contour map generation, and LLaVA-Lightning for image text description generation; 3. Image simulation model, the present invention proposes an image simulation model based on diffusion model with multiple rounds of training, which is used to generate simulated images.

[0032] For more information about Carla, please refer to: Dosovitskiy A., Ros G., Codevilla F., et al. CARLA: An open urban driving simulator [C] / / Conference on robot learning. PMLR, 2017: 1-16.

[0033] For detailed information about the cityscapes dataset, please refer to: Cordts M., Omran M., Ramos S., et al. The cityscapes dataset for semantic urban scene understanding[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 3213-3223.

[0034] For detailed information about the KITTI dataset, please refer to: Geiger A., ​​Lenz P., Urtasun R..Are weady for autonomous driving? the kitti vision benchmark suite[C] / / 2012IEEE conference on computer vision and pattern recognition.IEEE, 2012:3354-3361.

[0035] For detailed information about Stable Diffusion, please refer to: Rombach R., Blattmann A., Lorenz D., et al. High-resolution image synthesis with latent diffusion models [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[0036] For detailed information and usage of the LLaVa model, please refer to the paper: Liu H., Li C., Wu Q., et al. Visual instruction tuning [J]. Advances in neural information processing systems, 2024, 36.

[0037] The present invention provides a method for generating a simulated image for an autonomous driving scene, the flow chart of which is as follows: Figure 2 As shown, the specific steps include:

[0038] like Figure 2As shown, in step S101, the automatic driving platform is set, which specifically includes the following steps:

[0039] S101-1: Select a vehicle model in the autonomous driving platform. Build a road scene, such as calling open source city resources in the open source autonomous driving simulation platform.

[0040] S101-2: Set up a virtual RGB camera and place it on the car. The camera placement and orientation are as similar as possible to the camera in the real dataset. The internal parameters of the virtual camera are as similar as possible to the camera in the real dataset. Adjust the shutter frequency of the camera to control the number of datasets.

[0041] S101-3: Generate vehicles and crowds using the autonomous driving simulation platform.

[0042] S101-4: Call the autonomous driving algorithm of the autonomous driving simulation platform to start the vehicle.

[0043] S101-5: Start the camera, which will generate a composite image according to the set shutter frequency as the vehicle moves.

[0044] After step S101, a synthetic image dataset that is as consistent as possible with the selected real image dataset is generated through the autonomous driving simulation platform.

[0045] like Figure 2 As shown, in step S102, a semantic segmentation dataset, a coarse contour map, a fine contour map and a text description corresponding to the synthetic image dataset are generated to construct a synthetic multimodal dataset. The synthetic image dataset uses the synthetic image generated in step S101. Specifically, the following steps are included:

[0046] S102-1: Fine contour maps and semantic segmentation maps are generated using open source generative models, such as StableDiffusion. The generated contour maps and semantic segmentation maps are as follows: Figure 3 shown.

[0047] S102-2: The text description is generated using an open source multimodal language generation model, such as LLaVA-Lightning. An example of a generated prompt is: The image is a photo of cityscapes, featuring a street with a red bus driving down it. The image also includes a forest in the background. There are buildings on the left of image. The monochromatic colorscheme of the photo emphasizes the contrast between the vehicles and the urban environment, creating a visually striking and evocative representation of city life.

[0048] Use the script to add the pose information of the camera that generates the corresponding image at the end of the text description, such as: This image is generated by a camera with a position and orientation of (x, y, z, roll, pitch, yaw) relative to the center of the vehicle. x, y, z, roll, pitch, yaw are the pose information of the camera that generates the synthetic image relative to the center of the vehicle, representing the position and attitude angle respectively.

[0049] S102-3: Generate a coarse contour map using the semantic segmentation map. The coarse contour map describes the outer contour of the object and can filter out some interfering lines. The fine contour map can provide guidance for generating the inner contour of the object. The generation formula is:

[0050]

[0051] where x i,j Represents the pixel at position (i, j) in the original image, p i,j Represents the pixel at position (i,j) of the coarse contour. *,* Represents any pixel adjacent to (i, j).

[0052] like Figure 2As shown, in step S103, for the selected open source autonomous driving dataset (such as cityscapes, KITTI), the same model and method in step S102 are used to generate semantic segmentation datasets, coarse contour maps, fine contour maps and text descriptions. At the same time, the script is used to add camera position information (as described in step S102) to construct a real multimodal dataset.

[0053] like Figure 2 As shown, in step S104, the diffusion model is generated by training the multimodal dataset constructed in step S103. The model structure is as follows Figure 4 As shown in the figure, the model is mainly divided into three parts: the central network, the semantic segmentation control network, and the contour control network. The central network uses the pre-trained Stable Diffusion model. The semantic segmentation control network can help the network generate semantic matching and solve the problem of semantic changes in the generated image; the contour control network is used to solve the boundary line and contour distortion problems of the generated image. The fine contour map can be used to control the internal lines of the object, but the fine contour map will produce many redundant lines (such as shadow contours, etc.), so the coarse contour map and the fine contour map are used for joint control.

[0054] The network training process is as follows:

[0055] S104-1: Fine-tune the central network using image datasets and text descriptions from real multimodal datasets, so that image generation conforms to the overall style of the target dataset. Given a latent space tensor z0 of an image x0, randomly sample a time t(t∈(0,1000)) and Gaussian noise ∈(∈∈N(0,I)) using the diffusion model noise addition method:

[0056]

[0057] Get the noise map x t ,in α i is the preset hyperparameter. t , the encoded text description and t are input into the model, and the model predicts ∈ to get ∈ θ . Calculate the loss function:

[0058]

[0059] The model training epoch is 2. Too much training will cause the model to overfit and lose the prior ability of the original diffusion model.

[0060] For detailed information on the generation of ∈ and the training method of the diffusion model, please refer to the diffusion model paper: Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models [J]. Advances in neural information processing systems, 2020, 33: 6840-6851.

[0061] S104-2: Train the contour control network and the semantic segmentation control network. Fix the weight of the central network so that it is no longer updated during training. The input for training is the RGB image x and its corresponding rough contour map x in the real multimodal dataset. hed1 , fine contour map x hed2 , semantic segmentation map x seg , text description p. Using the pre-trained encoder, x hed1 、x hed2 、x seg , p are encoded into the latent space respectively, and the latent space control condition c is obtained hed1 、c hed2 、c seg 、c p .

[0062] In the contour control network, MLP is used to hed1 、c hed Feature fusion is performed because fine contours often contain interfering lines such as shadow lines, and coarse contours lack the internal contours of objects. Using a single contour control will cause line distortion.

[0063] The initial weights in the control network block are copied from the fine-tuned central network. After the input passes through the control network, it is input into the zero convolution layer, which makes the network's initial output equal to the central network output to obtain stable gradient information. During training, the parameters of the zero convolution layer will be updated accordingly, allowing the network to determine the control strength of the input control conditions by itself.

[0064] The training process follows the training process of the diffusion model, and the loss function is:

[0065]

[0066] Among them, c seg represents the control information of the semantic graph after the encoding network, c hed1 represents the control information of the fine contour image after the encoding network, c hed2Represents the control information of the rough contour image after the encoding network. When the control network is trained, the text conditions of the input control network will be randomly masked with a probability of 30%, so that the network is more dependent on the control information of the image. Pooling layer or mask can be used. ∈ is the diffusion noise added by the diffusion model at the current time t, ∈ θ is the noise of ∈ predicted by the network.

[0067] After step S104, the present invention constructs a diffusion model that can generate a single realistic picture with contour map, semantic segmentation map, prompt as control conditions and real RGB image as supervision information. The present invention constructs a multimodal domain composed of semantic segmentation map, contour map, and text description, and trains a multi-head control simulation model that can model the mapping relationship from the multimodal domain to the real image domain.

[0068] like Figure 2 As shown, in step S105, a simulated data set is generated using the simulated model trained in step S104 and the synthetic data set obtained in step S101, and the input data includes: semantic segmentation data, coarse contour data, fine contour data, and text description.

[0069] The above are only specific implementation methods of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A method for generating a realistic image for an autonomous driving scene, characterized in that: The method comprises the following steps: S101: Setting an autonomous driving simulation platform to ensure that the relative position and parameters of the camera and the vehicle in the autonomous driving simulator are substantially consistent with the real data set used for training; using the autonomous driving method of the autonomous driving simulation platform; setting the camera shutter frequency so that the camera generates synthetic image data as the vehicle travels; S102: Using the synthetic image dataset collected in step S101 and the open source generation model, generate its corresponding semantic segmentation map, fine contour map, coarse contour map, and text description to construct a synthetic multimodal dataset; S103: Use the open source real autonomous driving dataset and the open source generation model in step S102 to generate corresponding semantic segmentation maps, fine contour maps, coarse contour maps, and text descriptions to construct a real multimodal dataset; S104: Using the real multimodal dataset constructed in step S103 as input, construct and train a simulation model of multi-head control; S105: Input the multimodal data set obtained in step S102 into the simulated model trained in step S104 to generate a simulated image.

2. The method for generating a simulated image for an autonomous driving scene according to claim 1, characterized in that: The training data of the model is a multimodal dataset constructed from a real dataset, and the dataset used when generating the final realistic image is a synthetic multimodal dataset obtained from an autonomous driving simulation platform.

3. The method for generating a realistic image for an autonomous driving scene according to claim 1, characterized in that: In step S103, the open source generative model in step S102 is used to generate a semantic segmentation map, a fine contour map, and a text description corresponding to the real dataset image; the generated semantic segmentation map is used to generate a coarse contour map; the position of the camera relative to the center of the vehicle is added to the text description; and a real multimodal dataset is constructed.

4. The method for generating a simulated image for an autonomous driving scene according to claim 1, characterized in that: In the step S104, the method for training a realistic model of multi-head control includes: fine-tuning the pre-trained central network model using the RGB image data and text description in the multimodal dataset generated in step S103; fixing the central network weights, and training the contour control network and the semantic segmentation control network using the real multimodal dataset generated in step S103.

5. The method for generating a realistic image for an autonomous driving scene according to claim 4, characterized in that: In step S104, the network structure is divided into a central network, a semantic segmentation control network, and a contour control network, wherein the input of the contour control network is a feature fusion tensor of a coarse contour and a fine contour.

6. The method for generating a simulated image for an autonomous driving scene according to claim 1, characterized in that: In step S104, the network training uses the following loss function Among them, c seg represents the control information of the semantic graph after the encoding network, c hed1 represents the control information of the fine contour image after the encoding network, c hed2 represents the control information of the rough contour image after the encoding network; ∈ is the diffusion noise added by the diffusion model at the current time t, ∈ λ is the noise of ∈ predicted by the network.

Images