Image generation model training method and device, electronic equipment and storage medium

Abstract

The invention provides an image generation model training method and device, electronic equipment and a storage medium. The invention relates to the technical field of artificial intelligence, in particular to the fields of computer vision, image processing, augmented reality and the like, and can be used for application scenes such as automatic driving test and verification, automatic driving simulation and training and the like. According to the specific implementation scheme, the method comprises the steps of obtaining a first simulation image of a source domain and at least one control signal related to the first simulation image; generating a second simulation image of the target domain based on the first simulation image and the at least one control signal; constructing a simulation image pair sample based on the first simulation image and the second simulation image; and training the sample based on the simulation image to obtain a first image generation model, wherein the first image generation model is used for generating a fourth simulation image of the target domain based on the third simulation image of the source domain. According to the disclosure of the scheme, the generalization ability of the model can be enhanced, and the fidelity of image generation is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of computer vision, image processing, augmented reality, etc., and can be used in application scenarios such as autonomous driving testing and verification, autonomous driving simulation and training, and specifically to a training method, device, electronic device and storage medium for an image generation model. Background Art

[0002] As autonomous driving simulation technology develops, it becomes essential to test and optimize systems in a safe environment and reduce the risk and cost of road testing. However, the difference between simulation and real environment leads to inconsistent test results. Narrowing the gap between the two becomes the key to improving the effectiveness of simulation. Summary of the invention

[0003] The present invention provides a training method, device, electronic device and storage medium for an image generation model.

[0004] According to a first aspect of the present disclosure, a method for training an image generation model is provided, the method comprising:

[0005] Acquire a first simulated image of a source domain and at least one control signal related to the first simulated image;

[0006] generating a second simulated image of the target domain based on the first simulated image and at least one control signal;

[0007] constructing a simulated image pair sample based on the first simulated image and the second simulated image;

[0008] A first image generation model is obtained by training samples based on the simulated image, and the first image generation model is used to generate a fourth simulated image in the target domain based on the third simulated image in the source domain.

[0009] According to a second aspect of the present disclosure, there is provided an image generation method, the method comprising:

[0010] Acquire a third simulated image of the source domain;

[0011] The third simulated image is input into the first image generation model to obtain a fourth simulated image of the target domain output by the first image generation model; wherein the first image generation model is obtained by training through the method of the first aspect mentioned above.

[0012] According to a third aspect of the present disclosure, a training device for an image generation model is provided, the device comprising:

[0013] A first acquisition module, used to acquire a first simulation image of a source domain and at least one control signal related to the first simulation image;

[0014] An image construction module, generating a second simulated image of the target domain based on the first simulated image and at least one control signal;

[0015] An image pairing module, used for constructing a simulated image pair sample based on the first simulated image and the second simulated image;

[0016] The training module trains samples based on the simulated image to obtain a first image generation model, where the first image generation model is used to generate a fourth simulated image in the target domain based on the third simulated image in the source domain.

[0017] According to a fourth aspect of the present disclosure, there is provided an image generating device, the device comprising:

[0018] A second acquisition module, used for acquiring a third simulated image of the source domain;

[0019] The image generation module is used to input the third simulated image into the first image generation model to obtain a fourth simulated image of the target domain output by the first image generation model; wherein the first image generation model is obtained by training through the method of the first aspect.

[0020] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0024] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0025] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0026] The solution disclosed in the present invention can enhance the generalization ability of the model and improve the realism of image generation.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0029] Figure 1 is a flowchart of a method for training an image generation model according to an embodiment of the present disclosure;

[0030] Figure 2 is a schematic diagram of a workflow of a second image generation model according to an embodiment of the present disclosure;

[0031] Figure 3 is a schematic diagram of a training process of a second image generation model according to an embodiment of the present disclosure;

[0032] Figure 4 is a schematic diagram of a training process of a first image generation model according to an embodiment of the present disclosure;

[0033] Figure 5 is a schematic diagram of a workflow of a first image generation model according to an embodiment of the present disclosure;

[0034] Figure 6 It is an overall architecture diagram generated based on the image of the present disclosure;

[0035] Figure 7 is a structural schematic diagram of a training device for an image generation model according to an embodiment of the present disclosure;

[0036] Figure 8 is a structural schematic diagram of an image generating device according to an embodiment of the present disclosure;

[0037] Fig. 9 is a schematic diagram of a scenario of a method for training an image generation model according to an embodiment of the present disclosure;

[0038] Fig.10 is a schematic diagram of a scene of an image generation method according to an embodiment of the present disclosure;

[0039] Fig.11 It is a structural schematic diagram of an electronic device used to implement the training method of the image generation model and / or the image generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0041] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0042] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0043] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0044] Simulation to Reality (Sim2Real): refers to the technology of migrating models and algorithms trained in a simulation environment to real-world applications, aiming to narrow the gap between simulation and reality and make the results in simulation equally effective in the real environment.

[0045] Diffusion model: refers to a generative deep learning model based on probabilistic reasoning, which is used to gradually generate data (such as images, sounds, etc.). Its core idea is to learn how to generate complex data from simple random noise through a gradual process of adding and denoising.

[0046] Gaussian noise: is a noise model that means the noise values ​​follow a normal distribution, with most values ​​concentrated near the mean and fewer extreme values. It is often used in image processing to simulate natural noise and help the model introduce disturbances during the generation process to learn denoising and generate more realistic images.

[0047] In the related art, although the existing Sim2Real migration method can narrow the gap, it has limited effect when dealing with complex features, which may cause loss of details and affect the reliability of the results. Therefore, there is an urgent need for a solution that can effectively narrow the domain gap, adapt to complex scenarios and maintain the integrity of details.

[0048] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a training method for an image generation model, which enhances the accurate migration of simulated images to high-fidelity real images, narrows the visual gap between simulation and reality, and effectively solves the problems of detail loss and insufficient processing of complex features in the prior art.

[0049] The present disclosure provides a method for training an image generation model. Figure 1 It is a flowchart of a method for training an image generation model according to an embodiment of the present disclosure. The method for training an image generation model can be applied to an image generation model training device. The image generation model training device is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the server can be a cloud server or an ordinary server. For example, mobile devices include but are not limited to: mobile phones, tablet computers. In some possible implementations, the method for training an image generation model can also be implemented by a processor calling computer-readable instructions stored in a memory. Figure 1 As shown, the training method of the image generation model includes:

[0050] S101, acquiring a first simulation image of a source domain and at least one control signal related to the first simulation image;

[0051] S102, generating a second simulation image of the target domain based on the first simulation image and at least one control signal;

[0052] S103, constructing a simulated image pair sample based on the first simulated image and the second simulated image;

[0053] S104 . Train the samples based on the simulated images to obtain a first image generation model, where the first image generation model is used to generate a fourth simulated image in the target domain based on the third simulated image in the source domain.

[0054] In the embodiments of the present disclosure, the source domain refers to the virtual or simulated environment for generating simulation images, which represents the basic data or environment used by the simulation system, including but not limited to: virtual simulation environment, computer-generated synthetic image scenes, scenes generated by game engines, industrial simulation environments, autonomous driving simulation scenes, drone flight simulation systems, and any other virtual or synthetic environments related to simulation data.

[0055] In the disclosed embodiment, the target domain refers to an environment or data that is closer to the characteristics of the real world than the source domain, including but not limited to: road scenes in the real world, images of objects in the real environment, real-life images collected by the camera of an unmanned vehicle, real urban environments, images in natural environments, industrial site images, and any other environments related to real data or high-fidelity images.

[0056] In the embodiment of the present disclosure, the first simulation image refers to an initial image obtained from a source domain (simulation environment), which is a basic image for subsequent generation and optimization.

[0057] In the embodiment of the present disclosure, the second simulation image refers to an optimized image that is generated based on the first simulation image and the control signal and is closer to the target domain.

[0058] In the disclosed embodiment, the control signal refers to auxiliary information or data used to guide or assist the image generation process, which can help generate a more accurate image or an image that better meets the requirements of the target domain, including but not limited to: depth map, edge map, semantic map, lighting information, material information, motion vector, etc.

[0059] In the disclosed embodiment, the first image generation model refers to a deep learning model generated through training, which can generate a high-fidelity image closer to the target domain based on the input source domain simulation image, including but not limited to: diffusion model, image generation model based on convolutional neural network, generative adversarial network, variational autoencoder, conditional image generation model, etc.

[0060] In some embodiments, obtaining a first simulated image of a source domain and at least one control signal related to the first simulated image includes: generating a first simulated image including scene objects, background, and lighting information through a simulation engine (such as a game engine (Unity), Unreal Engine, etc.), and extracting control signals related to the image, such as a depth map, an edge map, and a semantic map. At the same time, information such as lighting, shadows, and materials can also be extracted to ensure that the lighting effect and surface texture of the simulated image are closer to the real world.

[0061] In some embodiments, based on the first simulated image and at least one control signal, a second simulated image of the target domain is generated, including: based on the first simulated image and at least one control signal (such as a depth map, an edge map, a semantic map, etc.), inputting it into a pre-trained image generation model (such as a diffusion model). The control signal guides the generation process and adjusts the lighting, contour and semantic information in the image. Through iterative optimization and denoising of the model, the first simulated image is gradually refined to make it closer to the target domain characteristics in color, texture and structure, and generate a high-fidelity second simulated image.

[0062] In some embodiments, constructing a simulated image pair sample based on a first simulated image and a second simulated image includes: first, pairing the first simulated image (initial simulated image generated in the source domain) with the second simulated image (target domain optimized image generated based on the first simulated image and a control signal). In order to ensure the validity of the sample, it is necessary to perform quality detection on the pair of images, including checking their geometric consistency, semantic consistency, and visual differences. Specifically, the quality of the image pair can be evaluated by calculating image similarity indicators (such as structural similarity, peak signal-to-noise ratio, etc.), and screening out simulated image pairs that meet preset requirements. In this way, a high-quality simulated image pair sample is constructed for subsequent model training to learn how to accurately convert source domain images into target domain images.

[0063] It should be noted that, for the convenience of description, after the first image generation model is trained, the image from the source domain is called the third simulated image; after the third simulated image is input into the first image generation model, the image in the target domain output by the first image generation model based on the third simulated image is called the fourth simulated image; wherein the fourth simulated image is an image that is closer to the style of real-world images.

[0064] For example, assuming that the source domain is an autonomous driving simulation scene, a first simulated image of a city street scene is first generated through the Unity simulation engine, which contains elements such as roads, buildings, pedestrians, and vehicles. At the same time, control signals related to the image are extracted from the simulation engine, including a depth map of the scene, an edge map for identifying object contours, and a semantic map containing road, vehicle, and pedestrian categories. Based on these control signals, the first simulated image is processed using a pre-trained diffusion model to gradually adjust the lighting effects, texture details, and semantic relationships in the image to generate a second simulated image that is closer to the real road environment. Then, the first simulated image is paired with the second simulated image to construct a simulated image pair sample. In this process, indicators such as structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR) are used to evaluate the geometric and semantic consistency of the image pairs to ensure that they meet the preset standards. Finally, the simulated image pair samples are used to train the first image generation model, so that when the model receives a new third simulated image, it can generate a high-fidelity fourth simulated image to achieve an effect that matches the real-world scene.

[0065] The technical solution of the disclosed embodiment, through the construction of simulated image pairs, the model can learn the complex mapping relationship from the source domain to the target domain image conversion, thereby generating more realistic simulated images that conform to the characteristics of the target domain, narrowing the gap between the source domain and the target domain, and improving the realism of image generation. By using simulated image pairs containing multiple control signals for training, the model can learn how to perform image conversion under different conditions. This diverse training data can enhance the generalization ability of the model, so that it can still maintain stable performance when facing unknown or complex inputs.

[0066] In some embodiments, acquiring at least one control signal related to the first simulation image further includes: generating multiple related images corresponding to the first simulation image based on a simulation engine; and determining at least one control signal from the multiple related images of the first simulation image.

[0067] Here, simulation engine refers to a software platform used to generate virtual scenes and simulated images, which generates realistic images and related data, such as depth maps, edge maps, semantic maps and other control signals by simulating physical phenomena, lighting, materials, etc. It includes but is not limited to: Unity, Unreal Engine, physical simulation environment (Gazebo), autonomous driving vehicle simulation platform, drone and unmanned vehicle simulation platform, and other engines used for simulation image generation.

[0068] Here, multiple related images refer to multiple auxiliary images generated based on the first simulated image and associated with the image. These images provide different perspectives or feature information to help generate a more accurate target image, including but not limited to: depth map, edge map, semantic map, normal map, illumination map, etc.

[0069] In some embodiments, determining at least one control signal includes: first, generating a plurality of auxiliary images related to the first simulated image through a simulation engine, such as a depth map, an edge map, a semantic map, a normal map, and an illumination map. These related images provide information of different dimensions, which can reflect the geometric structure, edge features, illumination conditions, and semantic classification of objects in the scene. Next, the model analyzes these images to extract control signals that contribute to image generation. For example, by analyzing the depth map, the model can determine the distance information between the object and the camera as a control signal for subsequent image geometry correction; extract the contours of objects in the scene through the edge map to ensure the structural integrity of the generated image; determine the distribution of different object categories in the image through the semantic map to ensure the semantic consistency of the generated image. Finally, based on the information in these related images, at least one control signal is extracted and used to guide the generation of a second simulated image that is closer to the target domain features.

[0070] For example, suppose that in an autonomous driving simulation environment, a first simulated image of a city street is generated by the Unity engine, and the image contains scene elements such as vehicles, pedestrians, and buildings. Multiple auxiliary images related to this image are also generated, including depth maps, edge maps, and semantic maps. The depth map reflects the distance information between each object and the camera, the edge map marks the outline of the object, and the semantic map identifies different categories of areas such as roads, vehicles, and pedestrians. The model analyzes these auxiliary images and extracts control signals for generating a second simulated image. Specifically, the distance information of the depth map is used to adjust the geometric relationship in the scene to ensure the spatial sense and positional relationship of the objects in the image; the contour information of the edge map ensures that the boundary clarity of the object is retained during the image generation process; the semantic map ensures that the generated image is semantically consistent, for example, the road texture is only applied to the road surface, not to vehicles or buildings. Guided by these control signals, the model generates an image of a city street scene that is closer to reality, making the second simulated image more consistent with the real environment in terms of lighting, geometry, and semantic features.

[0071] In this way, this embodiment accurately guides the generation of simulated images by extracting control signals such as depth maps, edge maps, and semantic maps, so that the generated images are closer to reality in terms of geometry, lighting, and semantics, thereby improving the realism of the images.

[0072] In some embodiments, the at least one control signal further includes at least one of the following: a depth map, an edge map, and a semantic map.

[0073] In the disclosed embodiment, the depth map refers to the distance information between each pixel in the image and the camera or observation point. This map can be used to determine the distance relationship and spatial layout of objects in the scene, and is often used for three-dimensional reconstruction and geometric correction of objects.

[0074] In the disclosed embodiment, a semantic map refers to an image in which pixels in the image are classified and labeled, and each pixel in an area is assigned a category label (such as road, vehicle, pedestrian, etc.) to determine the object category and its distribution in the image, ensuring that the generated image is semantically consistent with expectations.

[0075] In the embodiment of the present disclosure, the edge map refers to an image that highlights the contours and boundaries of objects in the image and marks the obvious edges between objects, so as to ensure that the contours of the objects are clear and the structures are complete during the image generation process.

[0076] For example, in an autonomous driving simulation scenario, Unreal Engine generates depth maps, semantic maps, and edge maps to determine the distance, classification labels, and contours of objects. Using these control signals, the model generates a simulated image with accurate geometry, clear semantics, and clear boundaries, which is closer to the real urban environment.

[0077] In this way, image generation is guided by depth maps, semantic maps, and edge maps to ensure that the generated simulated images are closer to reality in terms of geometric structure, semantic classification, and object contours, effectively improving the authenticity and generation accuracy of the images.

[0078] In some embodiments, Figure 2 As shown, based on the first simulated image and at least one control signal, a second simulated image of the target domain is generated, and it also includes: inputting at least one control signal into a pre-trained second image generation model to obtain the second simulated image of the target domain output by the second image generation model.

[0079] In the disclosed embodiment, the second image generation model may be a dynamic snake convolution network (DSCNet) model.

[0080] Here, the DSCNet model refers to a deep learning model based on convolutional neural networks, which is mainly used for computer vision tasks, especially image segmentation or super-resolution tasks.

[0081] In some embodiments, obtaining a second simulated image of the target domain output by the second image generation model includes: first, taking Gaussian noise and control signals (such as depth maps, edge maps, semantic maps, etc.) as input. Gaussian noise is the initial state of image generation, similar to random seeds, representing the initial rough state of the image; the control signal provides clear guidance information for gradually removing noise and constructing the target image. In the second image generation process, the second image generation model, including but not limited to the DSCNet model, gradually processes the noise. The model passes through multiple iterative steps, and each step applies different weights to reduce noise, so that the image gradually changes from disorder to order, that is, from Gaussian noise to a clear image close to the target domain. Specifically, the DSCNet model uses a convolutional layer to process the image, and retains the key information in the control signal through denoising. The control signal guides noise removal during the generation process, and continuously provides the model with structural, illumination and semantic information, so that the generated image gradually approaches the target domain. In each iteration, the model not only removes noise, but also adjusts the image details according to the control signal. After multiple denoising and control signal guidance, the image gradually changes from disordered Gaussian noise to a simulated image close to reality. The control signal ensures that the generated image is consistent with the target domain in terms of geometry, lighting, and semantics. Ultimately, the second simulated image is not only visually close to the real scene, but also has accurate spatial and semantic structures, such as clear object outlines, correct lighting effects, and object categories.

[0082] For example, in an autonomous driving simulation system, the process of generating a second simulated image of the target domain begins with generating a first simulated image that includes elements such as streets, vehicles, and pedestrians. Subsequently, a depth map, an edge map, and a semantic map are extracted from the image as control signals. The depth map provides information about the distance between vehicles and pedestrians and the camera, the edge map marks the outlines of vehicles and buildings, and the semantic map ensures that elements such as roads and pedestrians in the image are correctly classified. These control signals are input into the pre-trained DSCNet model together with Gaussian noise. The DSCNet model gradually denoises and adjusts the image in combination with the control signal through multiple rounds of iterations. In each iteration, the Gaussian noise is gradually reduced, and the control signal guides the model to retain and enhance the geometric structure and semantic details in the image. For example, the depth map helps the model maintain the distance relationship of objects in the scene, ensuring that distant buildings are blurred and nearby vehicles are clear; the edge map ensures that the outlines of vehicles and buildings are clear and avoids blurring; the semantic map ensures that the road texture is only applied to the correct area and does not spread to the vehicle or pedestrian area by mistake. Through these iterative processes, the second simulated image generated by the DSCNet model is not only visually highly close to the real scene, but also consistent in terms of geometric structure, lighting, and object classification, generating a high-fidelity simulated image that is realistic with the target domain.

[0083] In this way, by introducing control signals such as depth maps, edge maps, and semantic maps, the DSCNet model can accurately retain the geometric structure and edge details of objects during Gaussian noise removal, ensuring that the generated image maintains the correct sense of distance and clear object contours in the three-dimensional space layout. At the same time, the semantic map guides the texture and lighting distribution in the image, so that the generated second simulation image not only conforms to the real scene logic in object classification, but also has a high degree of consistency with the target domain in visual effects.

[0084] In some embodiments, Figure 3 As shown, the second image generation model is a model trained with at least one control signal corresponding to a real image of the target domain as input data and the real image as reference data. The second image generation model may include a second control network (ControlNet-2) and a U-shaped structure network (U-Net). The input of the second control network includes a real image; the input of the U-shaped structure network includes a text description, and at least one control signal of the real image (such as a depth map, semantic map, edge map, etc. of the real image).

[0085] In the disclosed embodiments, a variational autoencoder (VAE) is a model component that encodes input data (such as an image) into a latent representation in a latent space. The role of the encoder is to compress complex high-dimensional data into a low-dimensional latent variable representation to provide an effective representation for the subsequent decoding step.

[0086] In the disclosed embodiment, the VAE decoder refers to a model component that reconstructs the original data from the latent representation in the latent space. The decoder completes the process of data generation or reconstruction by decoding the latent representation generated by the encoder into reconstructed data close to the original data.

[0087] In the disclosed embodiment, the contrastive language-image pretraining model (CLIP) can map images and texts into the same vector space at the same time, and train them through contrastive learning methods, so that the model can generate corresponding images according to natural language descriptions or extract semantic information from images.

[0088] In some embodiments, the training process of the second image generation model includes the following training formula:

[0089]

[0090] Among them, Z0=ε(I): This is the latent space representation Z0 obtained after encoding the input image I through VAE, which is the compressed feature representation of the image.

[0091] Among them, C text = CLIP(text): The text description text is encoded through the CLIP model to obtain the latent space representation C of the text text , represents the input text features. This part links the text description and the image generation process, so that the model can control the generated image according to the text description.

[0092] Among them, δ~N(0,1): This is the sampling of Gaussian noise, which means sampling noise from the standard normal distribution and adding it to the image during the generation process to simulate the random noise in the image.

[0093] in, Z t represents the latent space representation of the image after t time steps of noise addition. t represents the time step, which controls the degree of noise. C text Represents the text description control signal encoded by CLIP. θ(d) The control signal representing the depth map helps the model maintain the geometric information and spatial sense of the object during the generation process. The control signal representing the edge map ensures that the outlines of objects in the generated image are clear.

[0094] Among them, the loss function L: The loss function L represents the difference between the real noise δ and the noise δ generated by the model θ By minimizing this difference, the model learns to denoise and generate the target image. This process is repeated over and over again, allowing the model to learn to gradually remove noise from the noisy image while generating high-quality images based on the control signal.

[0095] For example, suppose that in an autonomous driving simulation system, the goal is to generate realistic images of urban road scenes. First, the system collects real images in the target domain, such as streets, vehicles, and pedestrians, and extracts control signals from these images, such as depth maps and edge maps. The depth map provides information about the distance between the object and the camera, helping the model maintain the spatial structure of objects in the scene; the edge map marks the contours of objects such as vehicles and pedestrians to ensure the clarity of the boundaries of objects in the generated image. During the training process, the input image is first encoded into a potential latent space representation Z0 through the VAE encoder, and the text description is encoded into a latent space representation C of the text through the CLIP model. text When generating an image, the model adds Gaussian noise at each time step t to blur the image, and then restores clarity through the denoising process. In this process, the depth map and edge map serve as control signals to guide the denoising step of the model, ensuring that the generated image remains consistent with the real image in terms of geometry. At each time step t, the depth map β θ (d) Maintain the distance relationship of objects in the scene, edge map Ensure that the contours of objects are preserved. After multiple rounds of iterations, the model continuously removes noise and gradually generates clearer images. The loss function L is used to measure the difference between the generated image and the target image. The model improves the image quality by optimizing this loss function to ensure that the final generated simulation image is not only visually close to the real world, but also maintains the accuracy of the geometric structure, presents clear contours and semantically logical object distribution. Ultimately, the model is able to generate high-fidelity simulation images that are highly consistent with the target domain based on text descriptions and control signals, such as an urban street scene with an accurate sense of distance, clear object contours, and lighting effects that conform to natural scenes. These images can not only be used for testing simulation systems, but also greatly improve the training effect of autonomous driving models.

[0096] In this way, by combining the VAE and CLIP models, under the guidance of input control signals (such as depth maps and edge maps) and text descriptions, noise is gradually removed to generate high-fidelity simulated images. This method ensures that the generated images are highly consistent with the real images in the target domain in terms of geometric structure, object contours, and semantic logic, significantly improving the image quality and detail fidelity.

[0097] In some embodiments, constructing a simulated image pair sample based on a first simulated image and a second simulated image also includes: performing quality inspection on each pair of simulated images to generate quality inspection results corresponding to each pair of simulated images; wherein each pair of simulated images includes a first simulated image and a second simulated image; and determining a simulated image pair whose quality inspection results meet preset quality requirements as a simulated image pair sample.

[0098] In some embodiments, quality inspection is performed on each pair of simulated images, including: quality inspection is performed on each pair of simulated images, including ensuring geometric structure and motion consistency through depth map matching and optical flow detection; comparing edge maps through edge detection algorithm to ensure the clarity of object contours; using semantic segmentation algorithm to detect semantic label matching to ensure object classification and region consistency; using structural SSIM and PSNR to evaluate the visual similarity and clarity of images; and ensuring that the generated image is consistent with the real image in color and lighting effects through color histogram analysis and lighting consistency detection.

[0099] For example, suppose the model generates a pair of simulated images: the first simulated image is a vehicle, pedestrian, building and other elements in a virtual city environment, and the second simulated image is a high-fidelity image generated by the DSCNet model. Next, the quality of the pair of simulated images is checked. First, by comparing the depth maps of the two images, ensure that the generated second simulated image is consistent with the first simulated image in terms of the spatial layout and distance of the objects. Then, the motion trajectory of the objects in the scene is evaluated by optical flow detection to verify whether the moving direction and speed of the vehicles and pedestrians are consistent with the actual situation. At the same time, the edge detection algorithm is used to compare the outline of the object to ensure that the edges of the vehicles and buildings in the generated image remain clear. Then, using the semantic segmentation algorithm, check whether the classification of roads, vehicles, pedestrians, etc. in the second simulated image is consistent with the first simulated image. To further evaluate the visual quality, the SSIM and PSNR values ​​are calculated to evaluate the similarity of the images in terms of brightness, contrast and noise level. Finally, the color distribution of the two images is compared by color histogram to ensure that the lighting and color of the generated image are consistent with the first simulated image. If these detection results all meet the preset quality standards, the pair of simulated images will be determined as a valid simulated image pair sample and ultimately used to train and optimize the autonomous driving model.

[0100] In this way, through precise quality detection, it is ensured that the second simulated image is highly consistent with the first simulated image in terms of geometric structure, motion, object classification and visual effect, and the quality of the image pair is comprehensively evaluated to ensure that the simulated image is not only realistic in appearance, but also consistent with the target domain at the detail level, thereby improving the reliability of the sample and facilitating the training of high-precision autonomous driving models.

[0101] In some embodiments, constructing a simulated image pair sample based on a first simulated image and a second simulated image also includes: performing a correlation test on each pair of simulated images to generate a correlation test result corresponding to each pair of simulated images; wherein each pair of simulated images includes a first simulated image and a second simulated image; and determining a simulated image pair whose correlation test result meets a preset correlation requirement as a simulated image pair sample.

[0102] In some embodiments, each pair of simulated images is subjected to correlation detection, including: ensuring that the spatial position and geometric structure of the objects are consistent through a depth map matching algorithm; evaluating the consistency of the object's motion trajectory through optical flow detection; comparing the outline and boundary of the object through edge detection to ensure that the shape and position are not offset; detecting the consistency of object classification and region through a semantic segmentation algorithm to ensure that the semantic labels match; and evaluating the visual similarity and noise level of the images through SSIM and PSNR to ensure that the two images are highly correlated at all levels.

[0103] For example, in an autonomous driving simulation scenario, a pair of simulated images is generated: the first simulated image is the initial virtual scene, including elements such as roads, vehicles, and pedestrians, and the second simulated image is an optimized image generated by the model. In order to ensure that the two have sufficient correlation, a detailed correlation test is performed on the pair of images. First, by comparing the depth maps of the two, the spatial position and geometric structure of the objects are verified to be consistent, ensuring that the distances of near and far objects remain the same. Subsequently, the optical flow detection algorithm is used to evaluate the motion trajectories of vehicles and pedestrians to ensure that their moving directions and speeds are as expected. Next, the outlines of the objects are compared by the edge detection algorithm to ensure that the shapes and positions of objects such as vehicles and buildings are not significantly offset. At the same time, the semantic segmentation algorithm is used to detect the classified areas in the image to ensure that the classification of objects such as roads and pedestrians is consistent and that semantic information is not lost. Finally, the SSIM and PSNR indicators are used to evaluate the visual similarity of the two images and detect their consistency in lighting, noise, and image details. If all the test results meet the preset standards, the pair of simulated images will be determined as a valid simulated image pair sample for subsequent model training and optimization.

[0104] In this way, through detailed correlation detection, the generated second simulated image is ensured to be highly consistent with the first simulated image in multiple dimensions such as geometric structure, motion trajectory, object classification and visual effect. This comprehensive correlation detection significantly improves the quality of the simulated image pair samples, ensuring that the model can more accurately capture the complex dynamic features in the real scene during subsequent training.

[0105] In some embodiments, Figure 4 As shown, a first image generation model is obtained by training samples based on simulated images, including:

[0106] The second simulated image in the simulated image pair sample is used as input data, and the first simulated image in the simulated image pair sample is used as reference data to train and obtain a first image generation model.

[0107] In the disclosed embodiment, the first image generation model may include a first control network (ControlNet-1) and a U-shaped structure network (U-Net). The input of the first control network includes a first simulation image; the input of the U-shaped structure network includes a text description and a second simulation image.

[0108] In this way, by using the second simulated image in the simulated image pair sample as input data and the first simulated image as reference data for training, the model can learn the key information required to convert from one simulated state to another. This training method significantly improves the model's understanding of specific tasks (such as image conversion, image editing, etc.) and can more accurately capture the mapping relationship between the input image and the reference image. Since the model is trained based on simulated image pairs, it can understand the intrinsic connection between image contents to a certain extent. Therefore, in addition to directly generating images similar to reference images, the model may also support flexible editing and conversion operations on input images, such as changing local features of images, adjusting the overall style of images, etc.

[0109] In some embodiments, a first image generation model is obtained based on sample training of a simulated image, including: obtaining an image latent space representation of a second simulated image; obtaining a text latent space representation of a text description; and training the first image generation model by combining the image latent space representation after noise addition, as well as the time step, the text latent space representation and the first simulated image.

[0110] Here, the text description defines which subject or domain the target image should belong to. For example, city street driving in a rainy scene, highway driving in a snowy scene, country road driving in a sunny scene, etc.

[0111] Here, the size of the sampling noise and the size of the time step can be set or adjusted according to actual needs such as accuracy requirements.

[0112] In this way, by encoding the text description into a text latent space representation and combining it with information such as the image latent space representation and time step, a simulated image that is highly matched to the text description can be generated. By introducing the noisy image latent space representation as one of the inputs and combining it with the concept of time step, this technology allows the model to gradually remove noise during the training process, thereby generating clearer, more delicate, and high-quality images. This denoising generation process helps the model learn the essential features of the image data, thereby improving the realism and naturalness of the generated images.

[0113] In some embodiments, obtaining the image latent space representation of the second simulated image includes: encoding the second simulated image through VAE to obtain the image latent space representation.

[0114] Here, the image latent space representation captures high-level, abstract features of the image that help the model generate or modify the image in subsequent steps.

[0115] Here, in addition to using VAE to obtain the latent space representation of the image, the latent space representation of the image can also be obtained through encoders in generative adversarial networks (GANs), autoregressive models, graph-based encoders, etc.

[0116] In this way, by encoding through VAE, the image latent space representation of the second simulated image is obtained, which helps to generate a high-quality target image according to the text description and control signal in the subsequent steps.

[0117] In some embodiments, obtaining a text latent space representation of a text description includes: encoding the text description through a CLIP model to obtain a text latent space representation.

[0118] Here, the text latent space representation (also called text embedding or text vector) can be obtained not only through the CLIP model, but also through a variety of other natural language processing (NLP) techniques or models.

[0119] In this way, the text description is encoded through the CLIP model to obtain the text latent space representation, which can capture the semantic connection between text and image at the same time, and help generate high-quality target images according to the text description and control signals in the subsequent steps.

[0120] In some embodiments, a first image generation model is trained by combining the latent space representation of the noisy image, the time step, the latent space representation of the text, and the first simulated image, including: inputting the latent space representation of the noisy image, the time step, the latent space representation of the text, and the first simulated image into the model to be trained; determining a loss function based on the sampled noise and the noise predicted by the model; and obtaining the first image generation model by training by minimizing the loss function, so that the first image generation model gradually removes noise from the noisy second simulated image, and generates a target image of the target domain according to the given text description and the first simulated image.

[0121] Here, the similarity between the target image and the first simulation image used as the control signal is greater than a preset threshold.

[0122] In some embodiments, the training process of the first image generation model includes the following training formula:

[0123]

[0124] Among them, Z0=ε(I): This is the latent space representation Z0 obtained after encoding the input image I through VAE, which is the compressed feature representation of the image.

[0125] Among them, C text = CLIP(text): The text description text is encoded through the CLIP model to obtain the latent space representation C of the text text , represents the input text features. This part links the text description and the image generation process, so that the model can control the generated image according to the text description.

[0126] Among them, δ~N(0,1): This is the sampling of Gaussian noise, which means sampling noise from the standard normal distribution and adding it to the image during the generation process to simulate the random noise in the image.

[0127] Among them, δ θ (Z t ,t,C text ,η θ (I sim )):Z t represents the latent space representation of the image after t time steps of noise addition. t represents the time step, which controls the degree of noise. C text Indicates the text description control signal encoded by CLIP. θ (I sim ) represents a control signal of the picture generated by the second image generation model, which is used to help guide the model to maintain the simulated image I during the generation process sim The consistency of features such as geometric structure, lighting effects, edge details, etc. between the target domain (real environment) image.

[0128] Among them, the loss function L: The loss function L represents the difference between the real noise δ and the noise δ generated by the model θ By minimizing this difference, the model learns to denoise and generate the target image. This process is repeated over and over again, allowing the model to learn to gradually remove noise from the noisy image while generating high-quality images based on the control signal.

[0129] For example, by constructing simulated image pair samples based on the first simulated image (initial simulated scene) and the second simulated image (an image that is closer to the target domain after optimization), the model is trained to capture the mapping relationship between the simulated image and the target domain image. During the training process, the system learns how to generate a higher fidelity target domain image from the simulated image by comparing the first simulated image (reference data) and the second simulated image (input data). The model gradually optimizes the generation process by identifying the differences between the two in terms of geometry, lighting, edges, and details, so that the final output image is more consistent with the visual effects and semantic consistency of the real scene. In this process, the input image is first encoded into a latent space representation Z0 through VAE. At the same time, the text description (such as a scene description of a vehicle or pedestrian) is encoded into a text latent space representation C through the CLIP model. text The model samples Gaussian noise from a standard normal distribution to simulate random interference in the generation process. As noise is added to the image, the model performs denoising through multiple rounds of iterations while generating images based on the input control signal. θ (I sim ) The control signal is responsible for ensuring the consistency of the generated image with the target domain image in terms of geometry, lighting, edge details, etc. For example, the first image generation model extracts this information from the simulated image to guide the adjustment of the model during the generation process, ensuring that the edges of objects remain clear, the lighting effects are consistent with the real scene, and the geometric positions of vehicles and pedestrians remain consistent. The model measures the difference in noise between the current generated image and the target image by calculating the loss function L. The minimization of the loss function guides the model to gradually remove noise, making the generated image closer and closer to the target domain image. After multiple rounds of iterations, the model learns how to generate high-fidelity target images while retaining the details of the simulated image, ensuring that the generated image is consistent with the target domain in terms of spatial layout, lighting, edges, and semantic information.

[0130] In this way, by taking the second simulated image in the simulated image pair sample as input data and combining iterative optimization with the control signal, the model can accurately capture the differences in geometric structure, lighting, and semantic information between the simulated image and the target domain image. Through multiple rounds of iterative denoising, the model ensures that the final output image maintains consistency and detail accuracy in complex scenes, significantly enhancing generation quality and training efficiency.

[0131] The present disclosure provides an image generation method. Figure 5 : is a flowchart of an image generation method according to an embodiment of the present disclosure. The image generation method can be applied to electronic devices, including but not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to: mobile phones, tablet computers, vehicle-mounted devices, personal computers, etc. In some possible implementations, the image generation method can also be implemented by a processor calling computer-readable instructions stored in a memory. Figure 5 As shown, the image generation method includes:

[0132] S501: Acquire a third simulation image of the source domain;

[0133] S502: Input the third simulated image into the first image generation model to obtain a fourth simulated image of the target domain output by the first image generation model.

[0134] In the disclosed embodiment, the third simulated image refers to a new simulated image generated from the source domain (simulated environment), which is usually an initial or unoptimized simulated image in the system generation process. It represents data generated in the simulation environment, which has not been optimized or adjusted, and may be significantly different from the image in the target domain (real environment).

[0135] In the disclosed embodiment, the fourth simulated image refers to an image generated by processing the third simulated image through the first image generation model. The fourth simulated image has been optimized, and the goal is to be a high-fidelity image that is closer to the real environment.

[0136] In some embodiments, obtaining a fourth simulated image of the target domain output by the first image generation model includes: first, inputting the simulated image and Gaussian noise into the first image generation model, the first image generation model including but not limited to the DSNet model. The simulated image is a third simulated image generated by the source domain, representing an image that has not been optimized. Gaussian noise is used as the initial state in the generation process to help simulate randomness. The DSNet model uses control signals (such as depth maps, edge maps, etc.) to gradually denoise and optimize the image through multiple rounds of iterations. In each iteration, the model adjusts the input according to feature information such as geometric structure, object edge, and lighting effect in the simulated image. DSNet combines the simulated image and noise and gradually converts it into a realistic target domain image. In this process, the control signal plays a guiding role to ensure that the generated image is consistent with the details, textures, and object contours of the real world. The generated image finally output is the optimized fourth simulated image, which is closer to the target domain in visual effect and has the lighting, structure, and semantic consistency in the real scene.

[0137] For example, in an autonomous driving simulation scenario, the system generates a third simulated image of a virtual city street, including unoptimized elements such as roads, vehicles, and pedestrians. Because this third simulated image is generated from a simulated environment, the visual effect is quite different from the street scene in the real world, the edges of objects are blurred, and the lighting and materials are not realistic enough. Next, the system inputs this third simulated image into the DSNet model together with Gaussian noise. Gaussian noise helps simulate the randomness in the generation process. The DSNet model performs multiple rounds of iterations by combining control signals (such as depth maps and edge maps), gradually removing noise in each iteration, and optimizing the image according to the geometric structure and lighting effects of the simulated image. The model makes detailed adjustments to the contours of vehicles, pedestrians, and roads based on the input geometry and edge information, and optimizes the lighting effects to make it closer to the real-world scene. Finally, the fourth simulated image generated by the DSNet model is highly similar to the city street scene in the real environment, with clear vehicle contours, natural lighting effects, and all elements are more realistic visually, meeting the requirements of high-fidelity simulated images.

[0138] In this way, by combining the randomness of Gaussian noise with precise control signals, the fidelity and authenticity of the details of the generated image are improved, making the simulated image not only more realistic visually, but also accurately reflecting the contours of objects and lighting changes in real scenes, thereby improving the overall effect of the image generation model. Through the above simulated image generation scheme, a large number of high-quality simulated images can be generated without relying on actual data, thereby greatly reducing the cost of data acquisition and accelerating the progress of related research and applications.

[0139] Figure 6 is an overall architecture diagram of image generation according to an embodiment of the present disclosure, such as Figure 6 As shown, the architecture is divided into two parts: construction of simulated image pairs based on the second image generation model and image migration based on the first image generation model.

[0140] Among them, based on the construction of the simulated image pair of the second image generation model, in this part, the input includes the control signals such as the depth map, edge map, semantic map and Gaussian noise of the first simulated image. By inputting these control signals and noise into the second image generation model, a second simulated image consistent with the target domain is generated. The first simulated image and the second simulated image constitute a simulated image pair for training and generating the first image generation model.

[0141] Among them, in the image migration based on the first image generation model, in this process, the input includes a third simulated image and Gaussian noise, and the first image generation model is responsible for converting the input third simulated image into a more realistic fourth simulated image, thereby realizing the migration from simulation to real scene.

[0142] It should be understood that Figures 2 to 6 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figures 2 to 6 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0143] The present disclosure provides a training device for an image generation model. Figure 7 As shown, the training device of the image generation model may include: a first acquisition module 701, used to acquire a first simulated image of a source domain and at least one control signal related to the first simulated image; an image construction module 702, used to generate a second simulated image of a target domain based on the first simulated image and the at least one control signal; an image pairing module 703, used to construct a simulated image pair sample based on the first simulated image and the second simulated image; a training module 704, used to obtain a first image generation model based on the simulated image pair sample training, and the first image generation model is used to generate a fourth simulated image of the target domain based on the third simulated image of the source domain.

[0144] In some embodiments, the first acquisition module 701 includes: a generation submodule for generating multiple related images corresponding to the first simulation image based on a simulation engine; and a determination submodule for determining at least one control signal from the multiple related images of the first simulation image.

[0145] In some embodiments, the at least one control signal comprises at least one of the following: a depth map, an edge map, and a semantic map.

[0146] In some embodiments, the image construction module 702 is used to: input at least one control signal into a pre-trained second image generation model to obtain a second simulated image of the target domain output by the second image generation model.

[0147] In some embodiments, the second image generation model is a model trained using at least one control signal corresponding to a real image of the target domain as input data and the real image as reference data.

[0148] In some embodiments, the image pairing module 703 includes: a first detection submodule, used to perform quality detection on each pair of simulated images and generate quality detection results corresponding to each pair of simulated images; wherein each pair of simulated images includes a first simulated image and a second simulated image; and a first pairing submodule, used to determine the simulated image pair whose quality detection results meet the preset quality requirements as a simulated image pair sample.

[0149] In some embodiments, the image pairing module 703 includes: a second detection submodule, used to perform correlation detection on each pair of simulated images, and generate correlation detection results corresponding to each pair of simulated images; wherein each pair of simulated images includes a first simulated image and a second simulated image; and a second pairing submodule, used to determine the simulated image pair whose correlation detection results meet the preset correlation requirements as a simulated image pair sample.

[0150] In some embodiments, the training module 704 is used to: use the second simulated image in the simulated image pair sample as input data and the first simulated image in the simulated image pair sample as reference data to train and obtain a first image generation model.

[0151] In some embodiments, the training module 704 includes: a first acquisition submodule, used to acquire the image latent space representation of the second simulated image; a second acquisition submodule, used to acquire the text latent space representation of the text description; and a training submodule, used to combine the noisy image latent space representation, the time step, the text latent space representation and the first simulated image to train a first image generation model.

[0152] In some embodiments, a training submodule is used to: input the latent space representation of the noisy image, the time step, the latent space representation of the text and the first simulated image into the model to be trained; determine the loss function based on the sampled noise and the noise predicted by the model; obtain the first image generation model by minimizing the loss function training, so that the first image generation model gradually removes noise from the noisy second simulated image, and generates a target image of the target domain according to the given text description and the first simulated image.

[0153] In some embodiments, the first acquisition submodule is used to: encode the second simulated image through VAE to obtain an image latent space representation.

[0154] In some embodiments, the second acquisition submodule is used to: encode the text description through the CLIP model to obtain a text latent space representation.

[0155] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0156] The training device for the image generation model of the disclosed embodiment can enhance the generalization ability of the model and improve the realism of image generation.

[0157] The present disclosure provides an image generating device, such as Figure 8As shown, the image generating device may include: a second acquisition module 801, used to acquire a third simulated image of a source domain; an image generating module 802, used to input the third simulated image into a first image generating model to obtain a fourth simulated image of a target domain output by the first image generating model; wherein the first image generating model is obtained by training using the method described above.

[0158] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0159] The image generation device of the disclosed embodiment can improve the detail fidelity and authenticity of the generated image, so that the simulated image is not only more realistic visually, but also can accurately reflect the contours of objects and lighting changes in real scenes.

[0160] The present disclosure provides a scenario diagram of a training method for an image generation model, such as Fig. 9 shown.

[0161] As mentioned above, the training method of the image generation model provided by the embodiment of the present disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0162] Specifically, the electronic device can perform the following operations:

[0163] Acquire a first simulated image of a source domain and at least one control signal related to the first simulated image;

[0164] generating a second simulated image of the target domain based on the first simulated image and at least one control signal;

[0165] constructing a simulated image pair sample based on the first simulated image and the second simulated image;

[0166] The first image generation model is obtained by training samples based on the simulated image.

[0167] It should be understood that Fig. 9 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig. 9 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0168] The present disclosure provides a scene schematic diagram of an image generation method, such as Fig.10 shown.

[0169] As mentioned above, the image generation method provided by the embodiment of the present disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0170] Specifically, the electronic device can perform the following operations:

[0171] Acquire a third simulated image of the source domain;

[0172] The third simulated image is input into the first image generation model to obtain a fourth simulated image of the target domain output by the first image generation model.

[0173] It should be understood that Fig.10 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig.10 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0174] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0176] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Fig.11As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0178] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the training method and / or image generation method of the image generation model. For example, in some embodiments, the training method and / or image generation method of the image generation model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the training method of the image generation model and / or the image generation method described above may be performed. Alternatively, in other embodiments, computing unit 1101 may be configured to perform the training method of the image generation model and / or the image generation method in any other appropriate manner (e.g., by means of firmware).

[0180] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0186] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0187] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for an image generation model, comprising: Acquire a first simulated image of a source domain and at least one control signal related to the first simulated image; generating a second simulated image of the target domain based on the first simulated image and the at least one control signal; constructing a simulated image pair sample based on the first simulated image and the second simulated image; A first image generation model is obtained by training samples based on the simulated image, and the first image generation model is used to generate a fourth simulated image of the target domain based on the third simulated image of the source domain.

2. The method according to claim 1, wherein: Acquiring at least one control signal related to the first simulation image, comprising: Generate a plurality of related images corresponding to the first simulation image based on a simulation engine; The at least one control signal is determined from a plurality of related images of the first simulated image.

3. The method according to claim 1 or 2, wherein: The at least one control signal comprises at least one of the following: Depth map, edge map, semantic map.

4. The method according to claim 1, wherein: The step of generating a second simulated image of the target domain based on the first simulated image and the at least one control signal includes: The at least one control signal is input into a pre-trained second image generation model to obtain the second simulated image of the target domain output by the second image generation model.

5. The method according to claim 4, wherein: The second image generation model is a model trained by using at least one control signal corresponding to a real image of the target domain as input data and the real image as reference data.

6. The method according to claim 1, wherein: The step of constructing a simulated image pair sample based on the first simulated image and the second simulated image comprises: Performing quality detection on each pair of simulation images to generate quality detection results corresponding to each pair of simulation images; wherein each pair of simulation images includes one of the first simulation images and one of the second simulation images; The simulated image pair whose quality detection result is that it meets the preset quality requirement is determined as the simulated image pair sample.

7. The method according to claim 1, wherein: The step of constructing a simulated image pair sample based on the first simulated image and the second simulated image comprises: Performing correlation detection on each pair of simulation images to generate correlation detection results corresponding to each pair of simulation images; wherein each pair of simulation images includes one of the first simulation images and one of the second simulation images; The simulated image pairs whose correlation detection results show that they meet the preset correlation requirements are determined as the simulated image pair samples.

8. The method according to claim 1, wherein: The step of training the sample based on the simulated image to obtain a first image generation model comprises: The first image generation model is obtained by training by taking the second simulated image in the simulated image pair sample as input data and taking the first simulated image in the simulated image pair sample as reference data.

9. The method according to claim 1, wherein: The step of training the sample based on the simulated image to obtain a first image generation model comprises: Acquire an image latent space representation of the second simulated image; Obtain text latent space representation of text description; The first image generation model is trained by combining the latent space representation of the image after noise addition, the time step, the latent space representation of the text and the first simulated image.

10. The method according to claim 9, wherein: The step of combining the latent space representation of the image after adding noise, the time step, the latent space representation of the text, and the first simulated image to train the first image generation model includes: Inputting the latent space representation of the image after adding noise, the time step, the latent space representation of the text and the first simulated image into the model to be trained; Determine the loss function based on the sampled noise and the model predicted noise; The first image generation model is obtained by training by minimizing the loss function, so that the first image generation model gradually removes noise from the noisy second simulated image, and generates a target image of a target domain according to the given text description and the first simulated image.

11. The method according to claim 9, wherein: The step of obtaining the image latent space representation of the second simulated image comprises: The second simulated image is encoded by a variational autoencoder (VAE) to obtain a latent space representation of the image.

12. The method according to claim 9, wherein: The step of obtaining the text latent space representation of the text description includes: The text description is encoded by comparing the language-image pre-trained CLIP model to obtain the text latent space representation.

13. A method for generating an image, comprising: Acquire a third simulated image of the source domain; The third simulated image is input into a first image generation model to obtain a fourth simulated image of the target domain output by the first image generation model; wherein the first image generation model is trained by the method described in any one of claims 1 to 12.

14. A training device for an image generation model, comprising: A first acquisition module, configured to acquire a first simulated image of a source domain and at least one control signal related to the first simulated image; an image construction module, configured to generate a second simulated image of a target domain based on the first simulated image and the at least one control signal; An image pairing module, configured to construct a simulated image pair sample based on the first simulated image and the second simulated image; A training module is used to train samples based on the simulated image to obtain a first image generation model, wherein the first image generation model is used to generate a fourth simulated image of the target domain based on the third simulated image of the source domain.

15. The device according to claim 14, wherein: The first acquisition module includes: A generating submodule, configured to generate a plurality of related images corresponding to the first simulation image based on a simulation engine; The determination submodule is configured to determine the at least one control signal from a plurality of related images of the first simulation image.

16. The device according to claim 14 or 15, wherein: The at least one control signal comprises at least one of the following: Depth map, edge map, semantic map.

17. The device according to claim 14, wherein: The image construction module is used to: The at least one control signal is input into a pre-trained second image generation model to obtain the second simulated image of the target domain output by the second image generation model.

18. The device according to claim 17, wherein: The second image generation model is a model trained by using at least one control signal corresponding to a real image of the target domain as input data and the real image as reference data.

19. The device according to claim 14, wherein: The image pairing module comprises: A first detection submodule, configured to perform quality detection on each pair of simulated images and generate a quality detection result corresponding to each pair of simulated images; wherein each pair of simulated images includes one of the first simulated images and one of the second simulated images; The first pairing submodule is used to determine the simulated image pair whose quality detection result is that it meets the preset quality requirement as the simulated image pair sample.

20. The device according to claim 14, wherein: The image pairing module comprises: A second detection submodule, configured to perform correlation detection on each pair of simulated images and generate a correlation detection result corresponding to each pair of simulated images; wherein each pair of simulated images includes one of the first simulated images and one of the second simulated images; The second pairing submodule is used to determine the simulation image pair whose correlation detection result meets the preset correlation requirement as the simulation image pair sample.

21. The device according to claim 14, wherein: The training module is used to: The first image generation model is obtained by training by taking the second simulated image in the simulated image pair sample as input data and taking the first simulated image in the simulated image pair sample as reference data.

22. The device according to claim 14, wherein: The training module comprises: A first acquisition submodule, used to acquire an image latent space representation of the second simulated image; The second acquisition submodule is used to obtain the text latent space representation of the text description; The training submodule is used to train the first image generation model by combining the latent space representation of the image after noise addition, the time step, the latent space representation of the text and the first simulated image.

23. The device according to claim 22, wherein: The training submodule is used to: Inputting the latent space representation of the image after adding noise, the time step, the latent space representation of the text and the first simulated image into the model to be trained; Determine the loss function based on the sampled noise and the model predicted noise; The first image generation model is obtained by training by minimizing the loss function, so that the first image generation model gradually removes noise from the noisy second simulated image, and generates a target image of a target domain according to the given text description and the first simulated image.

24. The device according to claim 22, wherein: The first acquisition submodule is used to: The second simulated image is encoded by a variational autoencoder (VAE) to obtain a latent space representation of the image.

25. The device according to claim 22, wherein: The second acquisition submodule is used to: The text description is encoded by comparing the language-image pre-trained CLIP model to obtain the text latent space representation.

26. An image generating device, comprising: A second acquisition module, used for acquiring a third simulated image of the source domain; An image generation module is used to input the third simulated image into a first image generation model to obtain a fourth simulated image of the target domain output by the first image generation model; wherein the first image generation model is trained by the method described in any one of claims 1 to 12.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to perform a method according to any one of claims 1-13.

29. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 13 when executed by a processor.

Images