Image generation method and device of driving scene, electronic equipment and storage medium

Abstract

The invention relates to an image generation method and device for a driving scene, electronic equipment and a storage medium, and the method comprises the steps: obtaining a simulation image and an initial noise matrix of the driving scene, and enabling the initial noise matrix to be determined based on a real image of a real driving scene; inputting the simulation image into an image generation control network of an image generation model for coding to obtain a feature image; and inputting the feature image and the initial noise matrix into an image diffusion network of the image generation model for processing to obtain a driving scene image. By adopting the technical scheme, the real style of image generation is introduced through the initial noise matrix, and the initial noise matrix is determined based on the real image of the real driving scene, so that the generated driving scene image can restore the style of the real world, and the quality and visual fidelity of the driving scene image are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image generation, and in particular to a method, device, electronic device and storage medium for generating an image of a driving scene. Background Art

[0002] Digital simulation technology plays a key role in the development of autonomous driving technology. Digital simulation technology can build complex virtual environments and provide a large amount of simulation data for model training and verification, greatly improving R&D efficiency while also reducing safety risks and economic costs of testing.

[0003] However, existing simulation environments are usually unable to restore the complexity of the real-world environment in terms of natural lighting, detailed textures, and object contours, resulting in certain differences between the simulated images and the real images, which to some extent affects the credibility of the simulation results and the actual application effect. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, at least one embodiment of the present disclosure provides a method, device, electronic device and storage medium for generating an image of a driving scene.

[0005] In a first aspect, the present disclosure provides a method for generating an image of a driving scene, comprising:

[0006] Acquire a simulated image of a driving scene and an initial noise matrix, wherein the initial noise matrix is ​​determined based on a real image of a real driving scene;

[0007] Inputting the simulated image into an image generation control network of an image generation model for encoding to obtain a feature image;

[0008] The feature image and the initial noise matrix are input into the image diffusion network of the image generation model for processing to obtain a driving scene image.

[0009] In a second aspect, the present disclosure provides a device for generating an image of a driving scene, comprising:

[0010] A data acquisition module, used to acquire a simulated image of a driving scene and an initial noise matrix, wherein the initial noise matrix is ​​determined based on a real image of a real driving scene;

[0011] An encoding module, used for inputting the simulation image into an image generation control network of an image generation model for encoding to obtain a feature image;

[0012] The image generation module is used to input the feature image and the initial noise matrix into the image diffusion network of the image generation model for processing to obtain a driving scene image.

[0013] In a third aspect, the present disclosure provides an electronic device, including: a processor and a memory;

[0014] The processor is used to execute the image generation method of the driving scene as described in the first aspect by calling the program or instruction stored in the memory.

[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the method for generating an image of a driving scene as described in the first aspect.

[0016] In a fifth aspect, the present disclosure provides a computer program product, which is used to execute the method for generating an image of a driving scene as described in the first aspect.

[0017] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:

[0018] In the disclosed embodiment, a simulated image and an initial noise matrix of a driving scene are obtained, the initial noise matrix is ​​determined based on a real image of a real driving scene, the simulated image is input into an image generation control network of an image generation model for encoding, a feature image is obtained, and then the feature image and the initial noise matrix are input into an image diffusion network of an image generation model for processing, to obtain a driving scene image. The above technical solution is adopted to obtain an initial noise matrix and a simulated image of a driving scene and input them into an image generation model for generating a driving scene image, the spatial distribution and details of the image generation are controlled by the simulated image, and the real style of the image generation is introduced by the initial noise matrix. Since the initial noise matrix is ​​determined based on a real image of a real driving scene, the generated driving scene image can restore the style of the real world, thereby improving the quality and visual fidelity of the driving scene image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A schematic diagram of a process of generating an image of a driving scene provided by an exemplary embodiment of the present disclosure;

[0022] Figure 2 A schematic flow chart of a method for generating an image of a driving scene provided by another exemplary embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a training process of an image generation model provided for an exemplary embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of a model training phase of an exemplary embodiment of the present disclosure is shown;

[0025] Figure 5 A schematic diagram of the internal structure of ControlNet of an exemplary embodiment of the present disclosure is shown;

[0026] Figure 6 A schematic diagram of the structure of a device for generating an image of a driving scene provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understandable that the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments, and the specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present disclosure. In the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present disclosure.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0029] There is a focus in the simulation data scene, that is, we hope that the model can learn the corresponding content through the simulation scene. For example, in the driving scene, pedestrians crossing zebra crossings and roadside lights are all texture details that need to be retained after the style transfer of the simulation data. Therefore, the transfer model needs to understand and be able to retain the required details. In addition, the reason why simulation data is different from the real world is that the simulation data has certain differences in picture style such as lighting, texture, reflection, refraction, shadow, character details, architectural details, etc. from the real world. Therefore, it is particularly important to allow the transfer model to understand the style of the real world.

[0030] To solve this problem, advanced generative artificial intelligence technology (AI Generated Content, AIGC) came into being. This technology can optimize the presentation of the data while keeping the basic content of the data unchanged, making the generated data closer to the visual effects of the real world. By performing style conversion on the simulated data, the difference between the simulated data and the real data is significantly reduced, thereby improving the realism of the simulation. These adjusted simulated data can be widely used in model training and evaluation, which helps to improve the accuracy and adaptability of the model, making the evaluation results more realistic and better reflecting the performance in actual scenarios.

[0031] This scheme uses AIGC technology to implement an image generation scheme for driving scenes. By inputting simulated images as conditional data input to the image generation control network, the consistency of image content before and after style transfer is ensured. At the same time, the style of images before and after style transfer is changed by changing the noise distribution sampled by the image diffusion network, so that the finally generated driving scene image can restore the style of the real world.

[0032] The specific implementation methods, devices, electronic devices and storage media for generating images of driving scenes disclosed in the present invention are explained in detail below with reference to the accompanying drawings.

[0033] Figure 1 A flowchart of a method for generating an image of a driving scene provided in an exemplary embodiment of the present disclosure is provided. The method can be executed by a device for generating an image of a driving scene provided in an embodiment of the present disclosure. The device for generating an image of a driving scene can be implemented using software and / or hardware and can be integrated in an electronic device.

[0034] like Figure 1 As shown, the method for generating an image of a driving scene may include the following steps:

[0035] Step 101 : obtaining a simulated image of a driving scene and an initial noise matrix, where the initial noise matrix is ​​determined based on a real image of a real driving scene.

[0036] Among them, the simulated image of the driving scene refers to the virtual image of the driving scene generated by digital simulation technology, and the real image of the real driving scene refers to the real image of the real-world driving scene collected by image acquisition devices such as vehicle-mounted cameras.

[0037] In this embodiment, for the acquired real images, the potential features contained in the images can be acquired based on these real images, and then the initial noise matrix is ​​calculated based on the acquired potential features. Since the real images can reflect the style of the real environment such as lighting, texture, shadow, etc., the initial noise matrix obtained based on the real images also includes the style features of the real environment.

[0038] It should be noted that, in this embodiment, each time an image is generated, the initial noise matrix obtained is different, so that each generated image has a different style, thereby achieving the purpose of style transfer.

[0039] Step 102: Input the simulated image into the image generation control network of the image generation model for encoding to obtain a feature image.

[0040] Among them, the image generation model can be pre-trained. The image generation model includes two parts: an image generation control network and an image diffusion network. The input of the image generation control network determines the content and details of the final generated image, and is used to constrain the denoising process of the image diffusion network, so that the image diffusion network considers additional conditional requirements during the image generation process, that is, the content of the simulated image output by the image generation control network.

[0041] In this embodiment, the acquired simulation image is input into the image generation control network of the image generation model for encoding, and the simulation image is converted into a feature image through the image generation control network to be input into the image diffusion network of the image generation model, so as to constrain the image diffusion network to generate image content through the feature image.

[0042] Step 103: Input the feature image and the initial noise matrix into the image diffusion network of the image generation model for processing to obtain a driving scene image.

[0043] In this embodiment, the obtained initial noise matrix and the feature image output by the image generation control network are input together into the image diffusion network of the image generation model, and the initial noise matrix is ​​denoised by the image diffusion network. The content of the feature image needs to be considered in the denoising process, so that the driving scene image finally obtained by denoising contains both the restricted image content in the simulation image and the style information in the initial noise matrix, thereby improving the authenticity of the driving scene image.

[0044] The image generation method of the driving scene of the embodiment of the present disclosure obtains a simulated image and an initial noise matrix of the driving scene, the initial noise matrix is ​​determined based on a real image of the real driving scene, the simulated image is input into the image generation control network of the image generation model for encoding, and a feature image is obtained, and then the feature image and the initial noise matrix are input into the image diffusion network of the image generation model for processing to obtain a driving scene image. The above technical solution is adopted to obtain an initial noise matrix and a simulated image of the driving scene and input them into the image generation model for generating a driving scene image, the spatial distribution and details of the image generation are controlled by the simulated image, and the real style of the image generation is introduced by the initial noise matrix. Since the initial noise matrix is ​​determined based on the real image of the real driving scene, the generated driving scene image can restore the style of the real world, thereby improving the quality and visual fidelity of the driving scene image.

[0045] In an optional embodiment of the present disclosure, Figure 2 As shown, based on the above-mentioned embodiment, the step of obtaining the initial noise matrix in step 101 may include the following sub-steps:

[0046] Step 201, obtaining multiple real images of real driving scenes.

[0047] For example, multiple real images captured by on-board cameras of different vehicles may be obtained to obtain images of different styles, so that the determined spatial style prior distribution covers as much style information as possible.

[0048] Step 202: determining a spatial style prior distribution of a real driving scene based on a plurality of real images.

[0049] In this embodiment, for the multiple acquired real images, the spatial style prior distribution of the real driving scene can be determined.

[0050] In an optional embodiment of the present disclosure, when determining the spatial style prior distribution of a real driving scene based on real images, feature extraction can be performed on multiple real images to obtain potential features of the multiple real images. Then, the mean of the potential features of the multiple real images is calculated pixel by pixel to obtain a mean matrix, and the variance of the potential features of the multiple real images is calculated pixel by pixel to obtain a variance matrix. Then, based on the mean matrix and the variance matrix, the spatial style prior distribution is determined.

[0051] It should be noted that the potential features corresponding to different real images have the same size to facilitate the pixel-by-pixel position calculation of the mean and variance. In addition, the currently commonly used image feature extraction method can be used to extract the potential features in each real image. For example, the VAEEncoder module of the image diffusion network Stable Diffusion can be used to extract the potential features of each real image. The present disclosure does not limit the feature extraction method.

[0052] It can be understood that calculating the mean and variance of the potential features of multiple real images at each pixel position refers to calculating the mean and variance of the feature values ​​at the same pixel position of the potential features of all real images. For example, the mean and variance are calculated at the (0,0) position of the potential features of all real images, and the mean and variance are calculated at the (0,1) position, and so on, until the mean and variance of each pixel position are calculated. Therefore, the result obtained for the mean calculation is a matrix result (i.e., the mean matrix), and the result obtained for the variance calculation is also a matrix result (i.e., the variance matrix).

[0053] For example, assuming that the number of acquired real images is M, the potential features determined based on the M real images are recorded as The goal is to reduce it to a noise prior that is aligned with the noise distribution training trajectory. The size of is h×w, where h represents the width of the latent feature and w represents the height of the latent feature. The theoretical premise is that each It has an equal impact on the final learned training noise distribution, so it is necessary to calculate the spatial style prior distribution of each pixel position for the potential features of the real image, denoted as The mean and variance in this distribution are both in matrix form with a size of h×w. The spatial style prior distribution of each pixel position can be calculated by the following formula:

[0054]

[0055] Step 203: Determine an initial noise matrix based on the spatial style prior distribution.

[0056] In this embodiment, after the spatial style prior distribution of the driving scene is determined, the initial noise matrix can be determined based on the spatial style prior distribution.

[0057] In the traditional model reasoning process, the noise used by the image diffusion network obeys the standard normal distribution N(0,1). In the generation process of the driving scene image of the embodiment of the present disclosure, if the original standard normal distribution N(0,1) noise of the model is directly changed to a certain spatial style prior distribution This will cause a large distribution shift, which will make the model unable to generate reasonable images. Based on this, the disclosed embodiment designs a new style prior injection technology to avoid this problem, that is, the forward diffusion process of the image diffusion network is introduced into the style prior injection process. Specifically, the standard normal distribution is randomly sampled to obtain random noise, and then the spatial style matrix is ​​determined based on the random noise and the spatial style prior distribution. Finally, the spatial style matrix is ​​forward diffused for the target time step to obtain the initial noise matrix.

[0058] When determining the spatial style matrix, the obtained random noise can be multiplied by the mean matrix in the spatial style prior distribution, and then added to the variance matrix in the spatial style prior distribution. The result is a style prior instance, called the spatial style matrix.

[0059] In this embodiment, after obtaining the spatial style matrix, the spatial style matrix can be forward diffused with a target time step number, and the result of the forward diffusion (also in matrix form) is used as the initial noise matrix. The target time step number can be preset according to actual needs and experience. For example, the target time step number can be set to 0.8T (T is the total time step number of the original forward diffusion process of the image diffusion network, which is generally 1000). Through this style prior injection technology, the forward diffusion process of the image diffusion network is introduced into the style prior injection process, which can ensure style transfer while avoiding distribution offset.

[0060] The driving scene image generation method of the present embodiment obtains multiple real images of the real driving scene, determines the spatial style prior distribution of the real driving scene based on the multiple real images, and then determines the initial noise matrix based on the spatial style prior distribution. Therefore, through the spatial style prior injection technology, one-to-many style transfer can be achieved by only performing prior calculations on the real images of the driving scene without performing specific training on the style of the driving scene, and a variety of different initial noise matrices are obtained, with low calculation cost. The obtained initial noise matrix is ​​directly input into the image diffusion network of the image generation model for denoising processing. The model does not need to learn the style of the driving scene, has low training cost, and has high model freedom.

[0061] In an optional embodiment of the present disclosure, Figure 3 As shown, the image generation model of the embodiment of the present disclosure can be trained through the following steps:

[0062] Step 301, obtaining a pre-trained image diffusion network and an initial image generation control network.

[0063] The image diffusion network is pre-trained, for example, the image diffusion network can use StableDiffusion, and the initial image generation control network refers to an untrained image generation control network, for example, an untrained ControlNet can be used as the initial image generation control network.

[0064] Step 302: Initialize the second network parameters of the initial image generation control network according to the first network parameters of the image diffusion network.

[0065] In this embodiment, for the convenience of description and distinction, the network parameters (including weights and biases) of the image diffusion network are called first network parameters, the network parameters of the initial image generation control network are called second network parameters, and the second network parameters are initialized according to the first network parameters.

[0066] Among them, when initializing the network parameters, the first network parameters are copied to the initial image generation control network to realize the initialization of the second network parameters, and the first network parameters of the image diffusion network are locked, that is, the first network parameters of the image diffusion network will not be updated during the model training process, and only the second network parameters of the initial image generation control network will be updated.

[0067] Step 303: Input the sample simulation image into the initialized initial image generation control network for encoding to obtain a sample feature image.

[0068] The sample simulation image refers to a simulation image of a driving scene obtained and used as a model training sample.

[0069] In this embodiment, the acquired sample simulation image is input into the initialized initial image generation control network, and the initial image generation control network encodes the sample simulation image to obtain a sample feature image. The obtained sample feature image is used as a conditional input and input into the image diffusion network to constrain the prediction process of the image diffusion network.

[0070] Step 304: Input the sample feature image and the sample simulation image into the image diffusion network for noise prediction to obtain predicted noise, wherein the sample noise added to the sample simulation image by the image diffusion network obeys the standard normal distribution.

[0071] In this embodiment, the sample feature image generated by the initial image generation control network is input into the image diffusion network as a constraint on the image diffusion network for noise prediction. In the noise prediction process, the image diffusion network performs noise processing on the input sample simulation image, and the sample noise added at each time step obeys the standard normal distribution, and performs noise prediction under the constraint of the input sample feature image to obtain the predicted noise.

[0072] Step 305, calculating the current loss value based on the sample noise and the predicted noise.

[0073] In this embodiment, the loss is calculated based on the obtained prediction noise and the added sample noise to obtain the current loss value.

[0074] Exemplarily, when performing loss calculation, the mean square error (MSE) loss function may be used to calculate the loss value, or other loss functions may be used to calculate the loss value, and the present disclosure does not impose any limitation on this.

[0075] Step 306, when the current loss value is greater than the loss threshold, update the second network parameters, and input a new sample simulation image for iterative training until the current loss value is no greater than the loss threshold, the training is completed, and the image generation model is obtained.

[0076] Among them, the loss threshold can be pre-set according to actual needs. For example, the loss threshold can be set according to the expected model effect. The smaller the loss threshold is set, the better the model effect obtained by training.

[0077] In this embodiment, after the current loss value is calculated, the current loss value can be compared with the loss threshold. If the current loss value is greater than the loss threshold, the second network parameter is updated, and a new sample simulation image is input for iterative training, and the next loss value is calculated and compared with the loss threshold again. Repeat the above process until the current loss value is not greater than the loss threshold, the training is completed, and a trained image generation model is obtained.

[0078] The image generation method for driving scenes of this embodiment initializes the second network parameters of the initial image generation control network according to the first network parameters of the image diffusion network, and uses the sample simulation image as a conditional input to constrain the noise prediction of the image diffusion network. At the same time, the sample simulation image is input into the image diffusion network as supervision, so that the trained image generation network has good stability and effect.

[0079] Through a large number of ControlNet experiments, it is found that the noise distribution sampled by the Stable Diffusion model greatly affects the style of the synthesized image, and the conditional input of ControlNet determines the content of the synthesized image. Therefore, the present disclosure starts from the technical perspective of decoupling for the first time, by inputting the simulated image as the conditional data input of ControlNet to ensure the consistency of the image content before and after style transfer, and at the same time, by changing the noise distribution sampled by the Stable Diffusion model to change the style of the image before and after style transfer, further improving the image details and visual fidelity. Therefore, in an optional embodiment of the present disclosure, the initial image generation control network can use ControlNet, the image diffusion network can use StableDiffusion, and ControlNet is used to train the content consistency base model. The training goal is to completely reconstruct the simulated image (content, style) of the ControlNet conditional input by sampling the noise of the standard normal distribution N(0,1), and achieve the purpose of style transfer by changing the distribution of the initialization noise during the model reasoning process. Among them, the output layer of ControlNet is connected to StableDiffusion through a zero convolution layer, and the network parameters of the zero convolution layer are initialized to zero.

[0080] Figure 4 FIG. 1 shows a schematic diagram of a model training phase of an exemplary embodiment of the present disclosure. It can be understood that since the noise is predicted in the model training phase, the image generation control network ( Figure 4The network parameters are ControlNet, so Figure 4 Only part of the structure of the image diffusion network Stable Diffusion is shown. Figure 4 As shown, ControlNet is connected to Stable Diffusion through a zero convolution layer. ControlNet is initialized according to the network parameters of the pre-trained Stable Diffusion. The network parameters of Stable Diffusion will be locked. ControlNet includes the created trainable copy of Stable Diffusion. The purpose of creating a copy of the original network instead of directly training the original network is to prevent overfitting when the data set is small and to maintain the high quality of the trained large model. In the model training phase, the sample simulated images in the simulated image data set are input into ControlNet and Stable Diffusion respectively. The simulated images input into ControlNet are used as conditional inputs, and the simulated images are also used as supervision for Stable Diffusion. ControlNet is a neural network that injects additional conditional information into Stable Diffusion. Assume that F(; Θ) is a UNet with parameters Θ, which converts the input image x into another feature map y, y = F(x; Θ). The initial parameters of the trainable copy of ControlNet created by copying the network parameters of UNet are Θ.

[0081] like Figure 4 As shown, in this embodiment, the trainable copy of ControlNet is connected to the UNet of Stable Diffusion through a zero convolution layer, and the zero convolution layer is represented by Z(;). Specifically, the zero convolution layer is a 1×1 convolution layer, whose weights and biases are initialized to zero, and the zero convolution layer constitutes a part of ControlNet. Take ControlNet including two zero convolution layers and the created trainable copy as an example, as shown Figure 5 As shown, the parameters of the two zero convolution layers are Θ z1 and θ z2 , then the sample feature image obtained by ControlNet can be expressed as y=Z(F(x+Z(c;Θ z1 );Θ c );Θ z2 ), where c represents a sample simulation image input as an additional condition; x represents a deep feature in the network, and in this embodiment also represents a sample simulation image; Θ c Represents the parameters of the trainable copy, whose initial value is Θ. Under the constraints of ControlNet, the feature map conversion result of UNet is y c =F(x; Θ)+Z(F(x+Z(c; Θ)z1 );Θ c );Θ z2 ).

[0082] like Figure 4 As shown in the figure, when predicting noise, noise is added for T steps, and the noise added at each time step t follows the standard normal distribution N(0,1). Under the constraint of the input sample feature image, noise prediction is performed through UNet and the predicted noise is output. MSE loss (MSE Loss) is calculated based on the predicted noise and the added noise, and the network parameters of the zero convolution layer and the trainable copy are adjusted according to the calculated loss value. Among them, the prompt word is fixed, for example, the prompt word is set to driving scene style transfer, and the setting of the prompt word has little effect on the training and reasoning process of the model.

[0083] ControlNet allows the spatial layout and details of image generation to be controlled through additional conditional images (such as edge maps, segmentation maps, RGB images, etc.); by locking the network parameters of the original Stable Diffusion model and creating a trainable copy, ControlNet can introduce new control signals without destroying the performance of the original Stable Diffusion model, ensuring that the quality and diversity of the model trained on large-scale datasets are preserved; and, ControlNet uses zero convolutional layers to connect the original Stable Diffusion model and the trainable copy. This approach does not introduce noise in the early stages of training, helps prevent overfitting and catastrophic forgetting, and makes model training more stable.

[0084] Further, in an optional embodiment of the present disclosure, when the initial image generation control network is ControlNet and the image diffusion network is Stable Diffusion, when the simulated image is input into the image generation control network of the image generation model for encoding to obtain the feature image, the simulated image can be input into the trainable copy of ControlNet for processing to obtain an intermediate feature image, wherein the trainable copy is obtained by copying the network parameters of Stable Diffusion, and then the intermediate feature image is input into the zero convolution layer for processing to obtain the feature image. It should be noted that in this embodiment, the method of calculating the feature image is similar to the method of calculating the sample feature image in the aforementioned model training stage, and will not be repeated here. Further, when the feature image and the initial noise matrix are input into the image diffusion network of the image generation model for processing to obtain the driving scene image, the feature image and the initial noise matrix can be input into Stable Diffusion, and Stable Diffusion performs denoising on the initial noise matrix according to a preset time step (for example, set to T=1000) under the restriction of the feature image to obtain the driving scene image. Since the initial noise matrix is ​​obtained by forward diffusion of style prior instances obtained by random sampling, the forward diffusion process of Stable Diffusion is introduced. Therefore, driving scene images containing real scene styles can be generated by directly denoising the input initial noise matrix, which saves the operation of adding noise through forward diffusion in the image generation process, improves the speed and efficiency of image generation, and reduces the computational complexity of image generation.

[0085] In an optional embodiment of the present disclosure, if the potential features used to determine the initial noise matrix are obtained by encoding the real image through the VAE encoding (Encoder) module of Stable Diffusion, then after the initial noise matrix is ​​denoised, the denoised result needs to be converted through the VAE decoding (Decoder) module of Stable Diffusion to convert the low-dimensional image into a high-dimensional image, and the converted high-dimensional image is the final driving scene image. VAE encoding can compress the image into a low-dimensional latent space, reducing the computational complexity of the model denoising process.

[0086] Stable Diffusion improves the quality of image generation through a larger UNet network structure, self-attention, cross-attention mechanisms, and additional text encoders. By combining ControlNet, the details and visual fidelity of the image can be further improved.

[0087] In order to implement the above embodiments, the present disclosure further provides a device for generating an image of a driving scene. The device for generating an image of a driving scene can be implemented using software and / or hardware and can be integrated into an electronic device.

[0088] Figure 6 A schematic diagram of the structure of a driving scene image generation device provided by an embodiment of the present disclosure, such as Figure 6 As shown, the image generation device 40 of the driving scene may include: a data acquisition module 410 , an encoding module 420 and an image generation module 430 .

[0089] The data acquisition module 410 is used to acquire a simulated image of a driving scene and an initial noise matrix, where the initial noise matrix is ​​determined based on a real image of a real driving scene;

[0090] An encoding module 420 is used to input the simulation image into the image generation control network of the image generation model for encoding to obtain a feature image;

[0091] The image generation module 430 is used to input the feature image and the initial noise matrix into the image diffusion network of the image generation model for processing to obtain a driving scene image.

[0092] Optionally, the data acquisition module 410 includes:

[0093] A real image acquisition unit, used to acquire multiple real images of real driving scenes;

[0094] A distribution determination unit, used for determining a spatial style prior distribution of a real driving scene based on a plurality of real images;

[0095] The noise determination unit is used to determine an initial noise matrix based on a spatial style prior distribution.

[0096] Further optionally, the distribution determination unit is further configured to:

[0097] Perform feature extraction on multiple real images to obtain potential features of multiple real images;

[0098] The potential features of multiple real images are averaged pixel by pixel to obtain a mean matrix;

[0099] The variance of the potential features of multiple real images is calculated pixel by pixel to obtain the variance matrix;

[0100] Based on the mean matrix and variance matrix, the spatial style prior distribution is determined.

[0101] Further optionally, the noise determination unit is further configured to:

[0102] Random sampling of the standard normal distribution is performed to obtain random noise;

[0103] Determining a spatial style matrix based on random noise and the spatial style prior distribution;

[0104] Perform forward diffusion of the spatial style matrix for the target number of time steps to obtain the initial noise matrix.

[0105] Optionally, the driving scene image generation device 40 further includes a training module for:

[0106] Obtain a pre-trained image diffusion network and an initial image generation control network;

[0107] Initializing a second network parameter of an initial image generation control network according to a first network parameter of the image diffusion network;

[0108] Input the sample simulation image into the initialized initial image generation control network for encoding to obtain a sample feature image;

[0109] The sample feature image and the sample simulation image are input into the image diffusion network for noise prediction to obtain the predicted noise, wherein the sample noise added by the image diffusion network to the sample simulation image obeys the standard normal distribution;

[0110] Calculate the current loss value based on sample noise and prediction noise;

[0111] When the current loss value is greater than the loss threshold, the second network parameters are updated, and a new sample simulation image is input for iterative training until the current loss value is no greater than the loss threshold. The training is completed and the image generation model is obtained.

[0112] Optionally, the initial image generation control network is ControlNet, and the image diffusion network is StableDiffusion; the output layer of ControlNet is connected to Stable Diffusion through a zero convolution layer, and the network parameters of the zero convolution layer are initialized to zero.

[0113] Further optionally, the encoding module 420 is further configured to:

[0114] The simulated image is input into the trainable copy of ControlNet for processing to obtain an intermediate feature image;

[0115] Input the intermediate feature image into the zero convolution layer for processing to obtain the feature image;

[0116] The image generation module 430 is also used for:

[0117] The feature image and the initial noise matrix are input into Stable Diffusion. Stable Diffusion denoises the initial noise matrix according to the preset time steps under the constraint of the feature image to obtain the driving scene image.

[0118] The image generation device for driving scenes that can be applied to electronic devices provided in the embodiments of the present disclosure can execute the image generation method for driving scenes provided in the embodiments of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not fully described in the embodiments of the device of the present disclosure, reference can be made to the description in any method embodiment of the present disclosure.

[0119] The embodiments of the present disclosure also provide an electronic device, including a processor and a memory; the processor calls the program or instructions stored in the memory to execute the steps of each embodiment of the image generation method for the driving scene as described above, which will not be repeated here to avoid repeated description.

[0120] The embodiments of the present disclosure also provide a computer-readable storage medium, which is non-transitory and stores programs or instructions. The programs or instructions enable a computer to execute the steps of the embodiments of the method for generating an image of a driving scene as described above. To avoid repeated description, they will not be repeated here.

[0121] The embodiments of the present disclosure also provide a computer program product, which is used to execute the steps of each embodiment of the aforementioned method for generating an image of a driving scene.

[0122] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0123] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating an image of a driving scene, characterized in that: The method comprises: Acquire a simulated image of a driving scene and an initial noise matrix, wherein the initial noise matrix is ​​determined based on a real image of a real driving scene; Inputting the simulated image into an image generation control network of an image generation model for encoding to obtain a feature image; The feature image and the initial noise matrix are input into the image diffusion network of the image generation model for processing to obtain a driving scene image.

2. The method according to claim 1, characterized in that The obtaining of the initial noise matrix comprises: Obtain multiple real images of real driving scenes; Determining a spatial style prior distribution of the real driving scene based on the multiple real images; Based on the spatial style prior distribution, the initial noise matrix is ​​determined.

3. The method according to claim 2, characterized in that The determining, based on the plurality of real images, a spatial style prior distribution of the real driving scene comprises: Performing feature extraction on the multiple real images to obtain potential features of the multiple real images; Calculating the mean of the potential features of the multiple real images at each pixel position to obtain a mean matrix; Calculating the variance of the potential features of the multiple real images at each pixel position to obtain a variance matrix; The spatial style prior distribution is determined based on the mean matrix and the variance matrix.

4. The method according to claim 2, characterized in that: The determining the initial noise matrix based on the spatial style prior distribution includes: Random sampling of the standard normal distribution is performed to obtain random noise; Determine a spatial style matrix based on the random noise and the spatial style prior distribution; The spatial style matrix is ​​forward diffused for a target number of time steps to obtain the initial noise matrix.

5. The method according to any one of claims 1 to 4, characterized in that: The image generation model is trained in the following way: Obtain a pre-trained image diffusion network and an initial image generation control network; Initializing second network parameters of the initial image generation control network according to the first network parameters of the image diffusion network; Inputting the sample simulation image into the initialized initial image generation control network for encoding to obtain a sample feature image; Inputting the sample feature image and the sample simulation image into the image diffusion network for noise prediction to obtain predicted noise, wherein the sample noise added by the image diffusion network to the sample simulation image obeys a standard normal distribution; Calculating a current loss value based on the sample noise and the predicted noise; When the current loss value is greater than the loss threshold, the second network parameters are updated, and a new sample simulation image is input for iterative training until the current loss value is no greater than the loss threshold. The training is completed and the image generation model is obtained.

6. The method according to claim 5, characterized in that The initial image generation control network is ControlNet, and the image diffusion network is Stable Diffusion; The output layer of the ControlNet is connected to the Stable Diffusion through a zero convolution layer, and the network parameters of the zero convolution layer are initialized to zero.

7. The method according to claim 6, characterized in that The step of inputting the simulated image into an image generation control network of an image generation model for encoding to obtain a feature image comprises: Inputting the simulation image into the trainable copy of the ControlNet for processing to obtain an intermediate feature image; Inputting the intermediate feature image into the zero convolution layer for processing to obtain the feature image; The step of inputting the feature image and the initial noise matrix into the image diffusion network of the image generation model for processing to obtain a driving scene image comprises: The feature image and the initial noise matrix are input into the Stable Diffusion, and the Stable Diffusion performs denoising processing on the initial noise matrix according to a preset time step under the restriction of the feature image to obtain the driving scene image.

8. A driving scene image generation device, characterized in that: include: A data acquisition module, used to acquire a simulated image of a driving scene and an initial noise matrix, wherein the initial noise matrix is ​​determined based on a real image of a real driving scene; An encoding module, used for inputting the simulation image into an image generation control network of an image generation model for encoding to obtain a feature image; The image generation module is used to input the feature image and the initial noise matrix into the image diffusion network of the image generation model for processing to obtain a driving scene image.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the image generation method of the driving scene according to any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or an instruction, and the program or the instruction enables a computer to execute the method for generating an image of a driving scene according to any one of claims 1 to 7.

Images