Device and method of synthetic image generation

Abstract

To provide a computer implemented method for using conditional diffusion models to generate a synthetic image.SOLUTION: The computer-implemented method involves providing a neural conditioning, which is determined by a foundation model, as input to a ControlNet. The neural conditioning and a latent input representation are then propagated through the ControlNet, and outputs of the ControlNet are used as additional injections for the diffusion model. The latent input representation is further propagated through the diffusion model, with the additional injections from the ControlNet being injected into corresponding layers of the diffusion model during propagation.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a method for generating synthetic images using a conditional diffusion model, a method for operating an actuator, a computer program and machine-readable storage medium, a classifier, a control system, and a system for training a control system. [Background technology]

[0002] Prior art Recent large-scale text-to-image generation models, such as Stable Diffusion, have demonstrated the ability to generate diverse and creative images with surprising performance when given text prompts. However, fine-grained control over the generation process, such as controlling image layout and object position, is challenging. Recent work, ControlNet, builds on top of Stable Diffusion (SD) and introduces additional branches, i.e., trainable copies of the UNet encoder within SD, to accommodate additional conditions, such as segmentation label maps and depth. However, ground truth label maps or depth annotations can be costly. Furthermore, low-condition information may lack meaningful high-level semantic information, and the model may ignore the condition information, resulting in synthesized images that do not match the input conditions. Summary of the Invention [Problem to be solved by the invention]

[0003] Advantages of the invention We propose to use per-pixel neural representations of images, i.e., features extracted from pre-trained underlying models (e.g., DINO, stable diffusion), as condition information. By doing so, expensive annotation is not required (thus reducing annotation costs and allowing the use of large amounts of unlabeled data for training), and the extracted features also contain rich semantic information, which improves synthesis quality and matches the conditions well.

[0004] Our method does not require manual annotation to obtain condition information, significantly reducing the cost of creating datasets for fine-tuning and allowing large amounts of unlabeled data to be used for training. Despite the cost reduction, our method further allows for the specification of semantic layouts when neural representations are used to encode this information. Furthermore, neural representations capture additional scene information (e.g., object orientation) that cannot be described using semantic labels alone. This enables additional applications for downstream data preparation tasks, such as training neural networks. Because object orientation is preserved, synthetic data can reuse bounding box annotations from the original images to train a pose estimator in addition to training a segmenter. [Means for solving the problem]

[0005] Disclosure of the Invention In the first aspect of the present invention, a new format of condition information for fine-tuning a diffusion model is proposed, where the condition can be effectively utilized by the diffusion model, resulting in improved synthesis quality and alignment. Previous work on ControlNet uses a trainable copy of a UNet encoder and feeds conditions, such as a semantic label map, as input to the network. After each feature is added to a noisy latent state and processed through each layer stack, it is inserted back into the denoising UNet with stable diffusion, which is frozen during fine-tuning. However, controlling image generation using a label map requires costly fine-tuning and lacks a method for specifying additional details such as object orientation and 3D pose. Instead, we propose using a pixel-by-pixel neural representation of the image as conditioning information. Fine-tuning using such a representation does not require annotation, as it can be automatically extracted from the corresponding image using pre-trained base models (e.g., DINO features or stable diffusion features). During inference, neural representations can be edited to preserve desired information such as object orientation and scene layout, and distracting variables such as object texture and appearance can be randomized. Such edits include linear projection along semantically significant directions (e.g., obtained by principal component analysis of features), cut-and-paste conditioning information, and text-based editing.

[0006] In another aspect of the invention, a diffusion model is used to generate training data images that are used to train an image classifier.

[0007] In another aspect of the invention, receiving a sensor signal comprising data from a sensor; - determining an input signal dependent on said sensor signal; - feeding said input signal to said classifier to obtain an output signal characterizing a classification of said input signal; It is envisaged to use said classifier trained by one of the above methods by a method comprising:

[0008] The classifier or segmenter, e.g., a neural network, may comprise a structure that can be trained to, for example, identify and distinguish between pedestrians and / or vehicles and / or road signs and / or traffic lights and / or road surfaces and / or human faces and / or medical abnormalities in an image of an imaging sensor. Alternatively, the classifier, e.g., a neural network, may comprise a structure that can be trained to identify voice commands in an audio sensor signal.

[0009] In another aspect of the invention, a computer-implemented method for using a classifier trained by a method according to any one of the preceding aspects provides an actuator control signal for controlling an actuator, the actuator control signal being determined depending on an output signal of the classifier determinable as described in the preceding section. It is proposed that the actuator controls an at least partially autonomous robot and / or manufacturing machine and / or access control system.

[0010] In another aspect of the present invention, there is proposed a control system for operating an actuator, said control system comprising a classifier employed according to any one of the preceding aspects of the present invention and configured to operate said actuator according to an output of said classifier.

[0011] The present invention can be applied to various types of digital images, including video images, radar images, LiDAR images, ultrasound images, motion images, and thermal images. The present invention can be used to classify sensor data, or to detect the presence of objects in the sensor data, or to perform semantic segmentation on the sensor data, e.g., with respect to traffic signs, road surfaces, pedestrians, vehicles, and / or with respect to object classes that can be represented in a semantic segmentation task, e.g., trees, sky, etc.

[0012] The upstream core of the present invention involves active learning in a testbench setting. This can be utilized for a variety of purposes, such as selecting appropriate data points for training a machine learning system, testing / verification / validation of the machine learning system, or other specified use cases. The present invention interacts with the testbench in the following way, with the testbench being replaceable if deemed inappropriate.

[0013] In terms of active learning / testing and data curation, the present invention can be used to actively select data that a technical system sends to a back-end computer. Such a selection process helps reduce data traffic. The information obtained from this data curation can then be used to train machine learning systems, test / verify / validate machine learning systems, or other specified use cases.

[0014] This interaction occurs as follows: the present invention is used as a method / data for training, functioning as an upstream component in a machine learning toolchain. It does not directly enhance machine learning systems for the aforementioned applications. Instead, the interaction here is used as a method for training such machine learning systems, a method for generating training data for such training, a method for generating test data to ensure safe operation of the trained machine learning system, or serves as a generative model for generating training or test data. Additionally, it can also be used as a method for training the generative model itself.

[0015] Embodiments of the invention will now be described in more detail with reference to the following drawings, in which: [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic flow chart illustrating stable diffusion. [Figure 2] 1 is a schematic flow chart illustrating ControlNet. [Figure 3] 1 is a schematic flow chart illustrating an extractor and a ControlNet. [Figure 4] 1 is a schematic flow chart illustrating an extractor and ControlNet for neural representations. [Figure 5] 1 is a schematic flow chart illustrating feature extraction from stable diffusion. [Figure 6] 1 is a schematic flow chart illustrating PCA. [Figure 7] 1 is a schematic flow chart showing the Extractor and ControlNe using PCA. DETAILED DESCRIPTION OF THE INVENTION

[0017] Recent large-scale text-to-image diffusion models have demonstrated remarkable performance. In particular, stable diffusion is a state-of-the-art open-source visual language generation model trained on billions of text-image pairs. Stable diffusion is a latent diffusion model, a special case of a diffusion model trained in a predetermined latent space instead of the original image space. More specifically, following VQ-GAN, stable diffusion first trains an autoencoder, where images are encoded into a latent space Z and decoded back to reconstruct the given image. In a second stage, a diffusion model is trained in the latent space Z. In addition to text input, recent studies, for example, ControlNet, have attempted to add other conditioning inputs (e.g., label maps, edges) to further improve the controllability of the generation process. However, we found that existing conditioning inputs are insufficient to describe desired scenes with sufficient specificity. In the following, we propose a better conditioning design for fine-tuning diffusion models that better specifies the generation outcome without requiring manual annotation.

[0018] Instead of operating in image space, stable diffusion (SD) operates in the latent space of an autoencoder, as shown in Figure 1. First, an encoder E maps a given image x to a spatial latent code z = E(x), and then z is mapped back to image space by a decoder D. The autoencoder is trained to reconstruct a given image, i.e.,

number

[0019] In the second stage, a diffusion model is trained in this latent space Z. The diffusion model consists of a forward diffusion process and a backward denoising process. The forward pass is a Markov chain that gradually adds Gaussian noise to the clean data. Formally, this is

number

number

number

[0020] The backward noise reduction process is

number

[0021] In effect, μ θ (z t ,t) is z t and the predicted noise ε θ |(z t,t), which is modeled by UNet. The parameters of UNet can be learned by minimizing the L2 norm of the noise prediction at sampled time step t, Formula (1)

number

[0022] For a more mathematical derivation, see DDPM.

[0023] During inference, the latent representation z is calculated from the Gaussian distribution. T can be randomly sampled, and then the trained denoising UNet can be successively used to denoise z from t=T to t=1. t Given a denoised latent state z t-1 The final synthesized image can be obtained by feeding the clean latent state z_0 through the decoder D.

[0024] Recent work (https: / / arxiv.org / abs / 2302.05543) in ControlNet proposes to augment stable diffusion with additional conditions, e.g., label maps. As shown in Figure 2, ControlNet clones a trainable copy of the UNet encoder, and the original stable diffusion is frozen during fine-tuning. The input conditions are fed as inputs to ControlNet after the zero convolutional layers and the trainable encoder, and the features are inserted back into the stable diffusion decoder, where each zero convolutional layer is effectively a 1x1 convolutional layer with both weights and biases initialized as 0. The training objective is simply taken from equation (1) by inserting the condition y, i.e., Formula (2)

number

[0025] We propose a two-part system for feature extraction and term generation. For feature extraction, previous studies have used either manual effort or error-prone neural networks to annotate high-level semantic information (e.g., labels or depth maps) present in a human-interpretable format. In this invention, we use neural representations from a base model to directly capture semantic information. Feature extraction first obtains a neural image representation from an image using a pre-trained base model (e.g., DINO or stable diffusion). That is, Formula (3) F raw =FM(x) where F M represents the frozen basis model, x represents the input image, and F raw represents the extracted raw features.

[0026] For example, previous research (Shir Amir et al., "Deep ViT Features as Dense Visual Descriptors") demonstrated that key values ​​in later layers (e.g., the 9th and 11th layers) of the DINO ViT model (see Figure 4) contain semantic information useful for co-part segmentation and correspondence matching.

[0027] Regarding stable diffusion, as shown in the plug-and-play diffusion feature for text-driven image-to-image translation, spatial features within stable diffusion can effectively capture semantic information, such as layout and object shape. Also, features in various layers can capture details at different granularities. More specifically, as shown in Figure 5, spatial features extracted from the intermediate decoder layer (f_inter) combined with a high-resolution self-attention map (SAres_32) can encode semantic layout information. Meanwhile, if appearance information needs to be preserved, deeper features (e.g., fdeep_16 and fdeep_32) can be added as well. It has been shown that these types of neural image representations can be decomposed into high-level semantic information and low-level appearance details using a simple linear projection. For example, the projection itself can be obtained by using principal component analysis (PCA) and selecting the first n eigenvectors, as shown in Figure 6. Due to these properties of neural representations, a reference image can be used to specify the desired high-level information content (e.g., object class and geometry) while discarding disturbing variations. Alternatively, the extracted features can be projected through a learnable linear layer, or full features can be used to preserve as much detail as possible. Such a projection step removes the information of disturbing variations and captures only the semantic information relevant to the next step. The projected features F_trans are placed into a pixel-aligned condition image, where each pixel is a neural representation obtained from the projection.

[0028] For condition generation, we adapted our previous ControlNet (see Figure 2) to incorporate neural representations as conditioning information, as shown in Figure 3. Using the pairing of neural features extracted for the target data domain with images, we can fine-tune the ControlNet. This trains the condition generator to fill in the removed distracting information and generate more realistic images. Formally, the training goal is to generate the transformed features Ft rans It can be adapted from equation (2) by inserting: Formula (4)

number

[0029] In summary, our invention leverages neural representations extracted from frozen underlying models, which avoids manual annotation and provides richer semantic information, resulting in diverse, high-quality synthetic images that preserve the desired information content.

Claims

1. 1. A computer-implemented method for generating a synthetic image using a conditional diffusion model, comprising: providing, as input to the ControlNet, neural conditioning determined by the underlying model for the synthetic image to be generated; propagating the neural conditioning and latent input representations for the diffusion model through the ControlNet and providing the output of the ControlNet as an additional injection to the diffusion model; propagating the latent input representation through the diffusion model, during which the additional injections from the ControlNet are injected into corresponding layers of the diffusion model; A method comprising:

2. The method of claim 1 , wherein the neural conditioning is determined by propagating a synthetic image to be generated through the base model and selecting multiple intermediate results of the base model as the neural conditioning.

3. The method of claim 2 , wherein a principal component analysis or machine learning system is applied to the plurality of intermediate results to obtain the neural conditioning.

4. The method of claim 1 , wherein the neural conditioning is a pixel-by-pixel neural representation of a reference image.

5. the diffusion model includes a forward diffusion process and a backward denoising process; To train the diffusion model, acquiring a given image; encoding the given image into a latent code using an encoder of an autoencoder; generating a noisy latent code by adding Gaussian noise to a clean latent code according to a fixed dispersion schedule; decoding the latent code back to image space using a decoder of the autoencoder; 5. The method according to claim 1, wherein:

6. A computer-implemented method for using the generated synthetic images of any one of claims 1 to 5 to train an image classifier.

7. The method of claim 6 , wherein the image classifier is used to control an at least partially autonomous robot and / or manufacturing machine and / or access control system.

8. A computer program configured, when said computer program is executed by a processor, to cause said computer to carry out the method according to any one of claims 1 to 7 together with all the steps thereof.

9. A machine-readable storage medium storing the computer program of claim 8.

10. A system configured to perform the method of any one of claims 1 to 7.

Images