Apparatus and method for synthetic image generation

Abstract

A computer-implemented method for generating synthetic images using a conditional diffusion model. The method involves providing neural conditioning determined by a foundation model as input to a ControlNet. The neural conditioning and a latent input representation are then propagated through the ControlNet, and the ControlNet outputs are used as additional injections for the diffusion model. The latent input representation is further propagated through the diffusion model, with the additional injections from the ControlNet being injected into corresponding layers of the diffusion model during propagation.

Description

[0001] The invention relates to a method for generating synthetic images using a conditional diffusion model as well as a method for operating an actuator, a computer program and a machine-readable storage medium, a classifier, a controller, and a training system thereof. State of the art

[0002] Current large-scale text-to-image generative models, such as Stable Diffusion, have demonstrated impressive ability to generate diverse and creative images from a text prompt. However, it is difficult to have fine-grained control over the generation process, such as controlling the image layout and object position. ControlNet builds on Stable Diffusion (SD) and introduces an additional branch—a trainable copy of the UNet encoder in SD—to accommodate additional constraints, such as segmentation label maps and depth. However, ground-truth label maps or depth annotation can be costly. Furthermore, the raw conditional information may lack high-level meaningful semantic information, and the model may ignore the conditional information, resulting in the synthesized image not aligning with the input constraint. Advantages of the invention

[0003] This invention proposes using per-pixel neural representation of an image as conditioning information, i.e., features extracted from a pre-trained foundation model (e.g., DINO, Stable Diffusion). This eliminates the need for expensive annotation (thus reducing annotation costs and providing large amounts of unlabeled data for training), and the extracted features also contain rich semantic information, leading to better synthesis quality and conditional adaptation.

[0004] The present invention eliminates the need for manual annotations to obtain the conditioning information, significantly reducing the cost of creating datasets for fine-tuning and allowing the use of large amounts of unlabeled data for training. Despite the lower cost, the invention can still specify the semantic layout when using neural representations that encode this information. Furthermore, the neural representation captures additional scene information that cannot be described with semantic labels alone (e.g., object orientation). This enables additional application for downstream data creation tasks for the purpose of training neural networks. Because object orientation is preserved, the synthesized data can be used not only to train a segmenter but also to reuse the bounding box annotation from the original image for training a pose estimator. Disclosure of the invention

[0005] In a first aspect according to claim 1, a novel format of conditional information for fine-tuning diffusion models is proposed, whereby the condition can be effectively utilized by the diffusion models, leading to improved synthesis quality and alignment. ControlNet has a trainable copy of the UNet encoder and feed conditions such as semantic label maps as input to the network. After being added to the noisy latent and processed by the stacks of layers, the features are inserted into the original denoised UNet of Stable Diffusion, which is frozen during fine-tuning.

[0006] However, controlling image generation using label maps is costly for fine-tuning and lacks options for specifying additional details such as object orientation and 3D pose. Instead, we propose using the neural representation of an image as per-pixel conditioning information. Fine-tuning with this representation has no annotation cost, as it can be automatically extracted from a corresponding image using pre-trained ground models (e.g., DINO features or Stable Diffusion features). At inference time, the neural representation can be manipulated to obtain desired information such as object orientation and scene layout, while simultaneously randomizing nuisance variables such as object texture and appearance.Such processing includes linear projections along semantically meaningful directions (obtained, for example, by principal component analysis of the features), cutting and pasting of conditioning information, and text-based processing.

[0007] In further aspects of the invention, the diffusion model is used to generate training data images used to train an image classifier.

[0008] In further aspects of the invention, it is provided to use the classifier trained with one of the above-mentioned methods by a method comprising the steps of: - Receiving a sensor signal with data from a sensor, - Determination of an input signal that depends on this sensor signal, and - Feeding the input signal into the classifier to obtain an output signal that characterizes a classification of the input signal.

[0009] This classifier or segmenter, e.g., a neural network, can be equipped with a structure such that it is trainable, e.g., to identify and distinguish pedestrians and / or vehicles and / or traffic signs and / or traffic lights and / or road surfaces and / or human faces and / or medical anomalies in imaging sensor images. Alternatively, the classifier, e.g., a neural network, can be equipped with a structure such that it can be trained to identify spoken commands in audio sensor signals.

[0010] In a further aspect of the invention, a computer-implemented method is proposed for using the classifier trained with the method according to one of the preceding aspects to provide an actuator control signal for controlling an actuator. Determining an actuator control signal as a function of an output signal of the classifier, which can be determined as described in the previous section. It is proposed that the actuator controls at least partially autonomous robots and / or a production machine and / or an access control system.

[0011] In a further aspect of the invention, a control system for operating the actuator is proposed. The control system comprises the classifier adopted according to any one of the preceding aspects of the invention and is configured to operate the actuator in accordance with an output of the classifier.

[0012] This invention can be applied to various types of digital images, including video, radar, LiDAR, ultrasound, motion, and thermal images. The invention can be used to classify the sensor data, detect the presence of objects in the sensor data, or perform semantic segmentation on the sensor data, e.g., with respect to traffic signs, road surfaces, pedestrians, vehicles, and / or object classes that might be displayed in the semantic segmentation task, e.g., trees, sky, etc.

[0013] The upstream core of the invention involves active learning in a test bench. It can be used for various purposes, e.g., selecting suitable data points for training a machine learning system, testing / verifying / validating a machine learning system, or other specified use cases. The invention interacts with a test bench in the following manner, which can be replaced if it is deemed unsuitable.

[0014] With regard to active learning / testing and data curation, the invention can be used to actively select data that a technical system transmits to a back-end computer. This selective process helps reduce data traffic. The information obtained from this data collection can then be used for training a machine learning system, testing / verifying / validating a machine learning system, or other specific use cases.

[0015] This interaction occurs as follows: The invention serves as a method / data for training and functions as an upstream component in the machine learning tool chain. It does not directly improve a machine learning system for the aforementioned applications. Instead, it serves as a method to train such a machine learning system, generate training data for that training, generate test data to ensure the safe operation of the trained machine learning system, or act as a generative model for creating the training or test data. Furthermore, it can be used as a method to train the generative model itself.

[0016] Embodiments of the invention are explained in more detail with reference to the following figures. The figures show: Fig. a schematic flow diagram of stable diffusion; Fig. a schematic flow diagram of ControlNet; Fig. a schematic flow diagram of an extractor and the ControlNet; Fig. a schematic flow diagram of an extractor and the ControlNet for neural representation; Fig. a schematic flow diagram of feature extraction from stable diffusion; Fig. a schematic flow diagram of PCA; Fig. a schematic flow diagram of an extractor and the ControlNet with PCA.

[0017] Recent large-scale text-to-image diffusion models have shown impressive performance. In particular, Stable Diffusion is the state-of-the-art open-source model, trained over billions of text-image pairs. Stable Diffusion is a latent diffusion model, which is a special case of the diffusion model trained in a specific latent space instead of the original image space. More specifically, Stable Diffusion, following VQ-GAN, first trains an autoencoder, where the image is encoded and decoded into the latent space Z to reconstruct the given image. In the second stage, a diffusion model is trained in this latent space Z. In addition to text input, recent works, e.g., ControlNet, attempt to add other conditional inputs (e.g., label maps, edges) to further increase the controllability of the generation process.However, the inventors found existing conditioning inputs insufficient to describe the desired scene with sufficient specificity. The following proposes a better conditional design for fine-tuning diffusion models that better specifies the generation result without manual annotations.

[0018] Instead of working in the image space, Stable Diffusion (SD) works in the latent space of an autoencoder, as in Fig. First, the encoder E maps the given image x into a spatial latent code z = E(x). The autoencoder is trained to reconstruct the given image, d(E(x)) x.

[0019] In the second stage, a diffusion model is trained in this latent space Z. The diffusion model consists of a forward diffusion process and a backward denoising process. The forward pass is a Markov chain to gradually add Gaussian noise to the clean data. Formally, it can be written as: , where {} are fixed variance schedule. The noisy latent space can be calculated in a closed form, ie,=q(zt|zt−1)=N(zt,(1−βt)zt−1,βtI), where:βtzt=(αt)z0+1−αtεε∼N(0,I) z0=E(x) and αt:=∏s=1t(1−βs).

[0020] The reverse denoising process can be parameterized by pθ(zt−1|zt):=N(zt−1;μθ(zt,t),σθ(zt,t))

[0021] Essentially expressed as a linear combination of and predicted noise , which is modeled by a UNet. The parameters of the UNet can be determined by minimizing the L2 norm of the noise prediction at a sampled time step t: µ θ (e.g. t ,t) zt ε θ |(z t ,t) Lnoise=Ez∼E(x),ε∼N(0,I),t[‖ε−εθ(zt,t)‖2]

[0022] For further mathematical derivations, see DDPM.

[0023] At inference time, one can randomly sample from the Gaussian distribution and then sequentially use the trained denosization UNet to obtain the denosized latent image from to . The final synthesized image can be obtained by feeding the clean latent image z_0 through the decoder D. zTzt−1ztt=T t=1

[0024] Recent work by ControlNet (https: / / arxiv.org / abs / 2302.05543) proposes to improve stable diffusion with additional constraints, e.g., label map. As in Fig. As shown, ControlNet clones a trainable copy of the UNet encoder, and the original Stable Diffusion is frozen during fine-tuning. The input condition is fed as input to ControlNet, and after the zero-convolutional layers and the trainable encoder, the features are reinserted into Stable Diffusion's decoder, where the zero-convolutional layers are essentially 1 convolutional layers with weight and bias initialized to zero. The training objective is simply inherited from eq. (1) by setting the condition y: Lnoise=Ez|∼E(x),ε∼N(0,I),t[‖ε−εθ(zt,t,y)‖2]

[0025] A system is proposed that consists of two parts: feature extraction and conditional.

[0026] For feature extraction, previous work uses either manual work or error-prone neural networks to annotate high-level semantic information in a human-interpretable format (e.g., labels or depth maps). The invention uses neural representations of foundation models to directly capture semantic information. In feature extraction, pre-trained foundation models (e.g., DINO or Stable Diffusion) are first used to obtain neural image representations from an image: Fraw=FM(x) where x represents the frozen foundation model, x the input image and the extracted raw features.FMF raw

[0027] For example, previous work (Shir Amir et al in their publication Deep ViT Features as Dense Visual Descriptors) has shown that the key values ​​of the later layers (e.g. 9th, 11th) of the DINO ViT model (see Fig. ) contain semantic information useful for co-segmentation and correspondence matching.

[0028] For Stable Diffusion, as shown in Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation, spatial features in Stable Diffusion can well capture semantic information, such as layout and object shape. And features at different levels can capture different granularities of details. More specifically, as shown in Fig. As shown, spatial features extracted from intermediate decoder layers (f_inter) can be combined with higher-resolution self-recording maps (SAres_32) to encode semantic layout information. If appearance information is to be preserved, deeper features (e.g., fdeep_16 and fdeep_32) can also be added. It has been shown that these types of neural image representation can be decomposed into high-level semantic information and low-level details by using simple linear projections. For example, the projections themselves can be obtained by using principal component analysis (PCA) and selecting the first n eigenvectors, as in Fig. Because of these properties of neural representations, the desired high-level information content (e.g., object class and geometry) can be specified using a reference image, without spurious variations. Alternatively, the extracted features can be projected over learnable linear layers, or the full features can be used to preserve as much detail as possible. This projection step removes information from spurious variations and captures only the relevant semantic information for the next step. The projected features F_trans are arranged in a pixel-oriented conditional image, where each pixel is the neural representation obtained from the projection.

[0029] For conditional generation, we used the previous work ControlNet (see Fig. ) to incorporate the neural representation as conditioning information, as in Fig.One can optimize ControlNet using pairs of extracted neural features and images on a target data domain. This trains the constraint generator to fill in the removed noise so that it can continue to generate realistic images. Formally, the training objective of eq. (2) can be adjusted by inserting the transformed features F trans : Lnoise=Ez∼E(x),ε∼N(0,I),t[‖ε−εθ(zt,t,Ftrans)‖2]

[0030] In summary, our invention leverages neural representation extracted from frozen foundation models, thus avoiding manual annotation and providing richer semantic information. This results in a variety of high-quality synthetic images that retain the desired information content.

Claims

[1] A computer-implemented method for generating synthetic images using a conditional diffusion model, which includes the following steps: Providing neural conditioning for a ControlNet as input, wherein the neural conditioning was determined by a baseline model for the synthetic image to be generated; Propagating neural conditioning and a latent input representation for the diffusion model through the ControlNet and providing outputs of the ControlNet as additional injections for the diffusion model; Propagating the latent input representation through the diffusion model, whereby during propagation the additional injections from the ControlNet are injected into corresponding layers of the diffusion model. [2] The method according to claim 1, wherein the neural conditioning is determined by propagating the synthetic image to be generated through the founding model and selecting a plurality of intermediate results of the founding model as neural conditionings. [3] The method of claim 2, wherein a principal component analysis or a machine learning system is applied to the plurality of intermediate results to obtain the neural conditioning. [4] A method according to any one of the preceding claims, wherein the neural conditioning is a per-pixel neural representation of a reference image. [5] A method according to any one of the preceding claims, wherein the diffusion model comprises a forward diffusion process and a backward denoising process, wherein for training the diffusion model the steps of obtaining a given image, encoding the given image into a latent code with an encoder of an autoencoder, generating a noisy latent code by adding Gaussian noise to the clean latent code according to a fixed variance schedule and decoding the latent code back into image space with a decoder of the autoencoder are applied; [6] A computer-implemented method for using the generated synthetic image according to any one of the preceding claims to train an image classifier. [7] Method according to claim 6, wherein the image classifier is used to control an at least partially autonomous robot and / or a manufacturing machine and / or an access control system. [8] A computer program arranged to cause a computer to carry out the method according to any one of claims 1 to 7 with all its steps when the computer program is executed by a processor. [9] A machine-readable storage medium on which the computer program according to claim 8 is stored. [10] System arranged to carry out the method according to any one of claims 1 to 7.

Images