Apparatus and computer-implemented method for digital image processing
By combining a text-to-image diffusion model with a control network, and utilizing pixel-wise gradient analysis and inverse denoising, the problem of artifact identification and improvement in synthetic digital images is solved, image quality is improved, and the training and testing of anomaly detection systems are supported.
Patent Information
- Application Number
- CN202510949707.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-07-10
- Filing Date
- 2025-07-10
- Publication Date
- 2026-01-16
AI Technical Summary
Existing text-to-image diffusion models struggle to effectively identify and remove artifacts with natural and unnatural appearances when generating synthetic digital images, thus affecting image quality.
By employing a text-to-image diffusion model combined with a control network, and through pixel-by-pixel gradient analysis and reverse denoising, image quality is automatically identified and improved. This includes progressively determining gradient magnitude and metrics such as mean, maximum output, and variance until the quality conditions are met.
It enables automatic identification and improvement of artifacts in synthetic digital images, thereby improving image quality and making it suitable for training and testing anomaly detection systems.
Smart Images
Figure CN121353447A_ABST
Abstract
Description
BACKGROUND
[0001] The present invention relates to an apparatus and a computer-implemented method for digital image processing.
[0002] Stable diffusion and control nets can be used to create a synthetic digital image from text and a given digital image. An automatic text-image alignment metric can be used to confirm the quality of the synthetic digital image.
[0003] High-resolution image synthesis with latent diffusion models (arXiv: 2112.10752) discloses stable diffusion. Adding conditional control to text-to-image diffusion models (arXiv: 2302.05543) discloses ControlNet. DreamFusion: Text-to-3D using 2D diffusion (arXiv: 2209.14988) discloses fractional extraction sampling loss for text-to-3D synthesis. If at first you don’t succeed, try, try again: Automatic text-image alignment metric for fidelity-based text-to-image generation (arXiv: 2305.13308) discloses an automatic text-image alignment metric. SUMMARY
[0004] A computer-implemented method for digital image processing, comprising determining a synthetic digital image with text-to-image diffusion from an input representing a digital image, from a noise sample, and from an embedding representing text, wherein the text-to-image diffusion comprises a forward diffusion process to determine a noise latent from the input and the noise sample, wherein the noise latent is parameterized by a parameter, wherein the text-to-image diffusion comprises a backward denoising process to determine an output representing the synthetic digital image from a linear combination of the noise latent and a predicted noise, wherein the synthetic digital image comprises pixels, wherein the method comprises determining, for at least one pixel, a magnitude of a gradient of the parameter with respect to a difference between the predicted noise for the pixel and the noise sample for the pixel, in particular the difference weighted by a variable weight. The method automatically indicates the magnitude of natural and unnatural appearing artefacts in the synthetic digital image.
[0005] The backward denoising process can comprise determining a stepwise continuous linear combination, wherein a noise latent of a step is a result of a linear combination of a predicted noise of a previous step and the noise latent, wherein the method comprises determining the magnitude of the gradient stepwise and determining a metric from the stepwise determined magnitudes, in particular a mean, an argmax of the output corresponding, an average value, or a variance of the stepwise determined magnitudes. The metric provides a pixel-wise feedback to identify whether a pixel is an artefact based on the determined magnitude for the pixel.
[0006] The method can comprise providing a threshold for the metric and picking the synthetic digital image in case the metric exceeds the threshold. In case the metric exceeds the threshold, this automatically classifies the synthetic digital image as comprising an artifact.
[0007] The method can comprise determining the size pixel by pixel for a plurality of pixels of the synthetic digital image and outputting an error heat map visualizing the metric pixel by pixel. The heat map depicts or explains the location of the artifact in the synthetic digital image.
[0008] The method can comprise determining the metric pixel by pixel for a plurality of pixels of the synthetic digital image, wherein the metric is associated with a pixel of the plurality of pixels for which the respective mean is determined, and wherein the method comprises determining a region of pixels of the synthetic digital image from the metric, in particular a bounding box. This means that a region comprising at least one artifact is identified.
[0009] Determining the region can comprise identifying a region comprising pixels associated with a metric that is greater than a metric associated with a pixel outside the region from the metric.
[0010] Determining the region can comprise determining a mean and a variance of the metric and determining a region comprising pixels associated with a metric that lies within the variance around the mean.
[0011] The method can comprise replacing pixels in the synthetic digital image with random noise, determining another input for the text-to-image diffusion representing the synthetic digital image comprising the random noise in the region, and determining another synthetic digital image with the text-to-image diffusion from the other input, from another noise sample, and from an embedding representing the text. This means that the digital image is improved.
[0012] The method can comprise replacing pixels in the synthetic digital image to determine another synthetic digital image and a metric for the other synthetic digital image until the metric determined for the other synthetic digital image satisfies a condition. This means that the digital image is improved until a desired condition, e.g. quality, is met.
[0013] The method can comprise determining a plurality of synthetic digital images with text-to-image diffusion for different text embeddings representing text describing an anomaly in a real-world technical component, the plurality of synthetic digital images for training or testing an anomaly detection system to identify anomalies in digital images of real-world components.
[0014] A device for digital image processing comprises at least one processor and at least one memory, wherein the at least one memory is configured to store instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the device to perform the method.
[0015] A computer program for digital image processing comprises instructions executable by a computer and, when executed by the computer, cause the computer to perform the method. BRIEF DESCRIPTION OF DRAWINGS
[0016] Further advantageous embodiments can be derived from the following description and the attached drawings. In the drawings:
[0017] Figure 1 A device for digital image processing is schematically depicted,
[0018] Figure 2 An exemplary text-to-image diffusion is schematically depicted,
[0019] Figure 3 A flowchart comprising steps of a method of digital image processing is depicted. DETAILED DESCRIPTION
[0020] The digital image can be a video image, a radar image, a lidar image, an ultrasound image, a motion image, or a thermal image.
[0021] Figure 1 A device 100 for digital image processing is schematically depicted.
[0022] The device 100 comprises at least one processor 102 and at least one memory 104.
[0023] The at least one processor 102 is configured to execute instructions causing the device 100 to perform the method for digital image processing. The at least one memory 104 is configured to store the instructions. The at least one memory 104 can comprise a transitory and / or a non-transitory memory.
[0024] The device 100 can comprise a computer configured to execute instructions. The computer program for digital image processing can comprise instructions.
[0025] Figure 2 An exemplary text-to-image diffusion 200 based on a stable diffusion 202 and a control net 204 is schematically depicted.
[0026] The text-to-image diffusion 200 is for example implemented as an artificial neural network.
[0027] The text-to-image diffusion 200 operates in a latent space Z. The text-to-image diffusion 202 is based on an input zo representing a digital image x in the latent space Z.
[0028] The encoder ε is configured to determine the input z0 representing the digital image x. The encoder ε maps the given digital image x from the image space to the spatial latent code z0 = ε(x). The encoder ε can be implemented as part of an artificial neural network or as a separate artificial neural network.
[0029] Text-to-image diffusion 200 is configured to work based on the input z0, based on the noise sample ∈ t And determine the synthetic digital image based on the embedding y representing the text.
[0030] The text-to-image diffusion 200 includes a forward diffusion process 206 in the stable diffusion 202, to determine the input z0 and noise sample ∈ t Determine the noisy latent image z in the latent space Z t .
[0031] Noise latent image z t (φ) is parameterized by parameter φ.
[0032] Text-to-image diffusion 200 includes an inverse denoising process 208 in stable diffusion 202 to determine the representation of the synthesized digital image. Output in the latent space Z
[0033] The reverse denoising process 208 includes an encoder 210 and a decoder 212, such as a convolutional neural network based on the UNet architecture.
[0034] To account for the text in the text-to-image diffusion 200, the control network 204 includes a trainable copy 210' of at least a portion of the inverse denoising process 208. For example, the control network 204 includes a trainable copy of the encoder 210.
[0035] The control network 204 is configured to determine the input 214 of the inverse denoising process 208.
[0036] The control network 204 is configured, for example, to determine the input 214 in a plurality of consecutive zero-convolutional layers 216. The control network 204 is configured, for example, to determine one input 214 for each consecutive zero-convolutional layer 216.
[0037] A zero convolutional layer is, for example, a 1×1 convolutional layer where the weights and biases are initialized to zero.
[0038] Decoder D is configured to output Determine the synthetic digital image Decoder D can be implemented as part of an artificial neural network or as a separate artificial neural network.
[0039] Based on an example, decoder D will use the spatial latent code from latent space Z. Mapping to synthetic digital images
[0040] For example, the text-to-image diffusion 200 operates in the latent space Z of an autoencoder comprising an encoder ε and a decoder D.
[0041] The encoder ε and the decoder D are trained, for example, with a set of digital images to reconstruct a given image x:
[0042]
[0043] The stable diffusion 202 comprises a plurality of steps t to gradually add noise to the input z0. At step t, a noise sample ∈ t (e.g. Gaussian noise) is sampled and added to the input z0.
[0044] The forward diffusion process 206 comprises, for example, a Markov chain of length T to gradually add noise:
[0045]
[0046] where denotes a fixed variance schedule and I is an identity matrix of appropriate size.
[0047] The noise latent z t is computed, for example, in closed form, for example:
[0048]
[0049] where and I is an identity matrix of appropriate size.
[0050] The backward denoising process 208 uses, for example, another Gaussian distribution
[0051] p θ (z t-1 |z t ):= N(z t-1 ; μ θ (z t , t), σ θ (z t , t))
[0052] where μ θ (z t , t) is a linear combination of z t and a predicted noise ∈ Θ (z t , t). The predicted noise ∈ Θ (z t , t, y) is modeled, for example, by a UNet.
[0053] The output of the stable diffusion 202 is the reverse denoising process 208 in the last step of the prediction.
[0054] The stable diffusion 202 is trained, for example in the latent space Z, to minimize the L2 norm of the noise prediction at a sampling step t:
[0055] L n = E z～ε(x),∈～N(0,I),t [‖∈-∈ Θ (z t ,t)‖ 2 ]
[0056] The control net 204 is trained, for example, to minimize the L2 norm of the prediction noise ∈ Θ (z t ,t,y) at a sampling time step t conditioned on the embedding y of the text:
[0057] L n = E z～ε(x),∈～N(0,I),t [‖∈-∈ Θ (z t ,t,y)‖ 2 ]
[0058] The stable diffusion 202 is frozen during the training of the control net 204.
[0059] Training the neural network using the loss discussed earlier optimizes the neural network such that the neural network can transform a noise sample z T into an output representing a synthetic digital image
[0060] The latent z t (Φ) at a step t is parameterized by the parameters φ.
[0061] In order to edit a given digital image x according to the text represented by the embedding y, the loss
[0062]
[0063] can be minimized to optimize the synthetic digital image The neural network is frozen to optimize the synthetic digital image This means that the latent code z t is directly optimized.
[0064] The loss L(φ) is itself difficult to compute, but, the gradient t at the latent z (Φ) parameterized by the parameters φ at a step t can be computed using the chain rule:
[0065]
[0066] The predicted noise ∈ Θ (z t , t, y) predicted by the backward denoising process 208 (e.g. UNet) in step t can be used to estimate the gradient
[0067] This means that for a given noise sample ∈ θ (z t (Φ), t, y) and a step t the gradient estimate is a scaled difference between the estimated and the real noise ∈
[0068] The digital image x and the synthetic digital image comprise pixels. The gradient is a pixel-wise gradient.
[0069] This means that for a pixel the gradient comprises a difference (∈ θ (z t (Φ), t, y)) between the predicted noise (∈ θ (z t (Φ), t, y)) for this pixel and the noise sample (∈) for this pixel.
[0070] According to one example the difference (∈ θ (z t (Φ), t, y)) is weighted by a weight ω(t). The weight ω(t) is variable, i.e. different weights ω(t) can be used for different steps t.
[0071] The magnitude of the gradient is higher in areas 218 which can contain artifacts. Thus, based on the magnitude of the gradient the areas 218 are identifiable.
[0072] Figure 3 A flowchart depicting steps of a digital image processing method is shown.
[0073] The method comprises a step 302.
[0074] The step 302 comprises providing a digital image x, a noise sample ∈ and an embedding y representing text.
[0075] The method comprises a step 304.
[0076] The step 304 comprises determining a synthetic digital image
[0077] Step 302 or step 304 can comprise providing an input z0= ε(x).
[0078] The text-to-image diffusion comprises a forward diffusion process 202 to determine a corresponding noisy latent image z t from the input z0and a noise sample ε. t The noisy latent image z θ (z t-1 | z t ) := N(z t-1 ; μ θ (z t , t), σ θ (z t , t))
[0079] The text-to-image diffusion 200 comprises a backward denoising process 204 to determine an output representing a synthetic digital image from the corresponding prediction.
[0080] p θ (z t-1 | z t ) := N(z t-1 ; μ θ (z t , t), σ θ (z t , t))
[0081] This means that the output is determined from a linear combination of the noisy latent image z t and the predicted noise ε θ (z t , t, y).
[0082] The synthetic digital image comprises pixels.
[0083] The method comprises a step 306.
[0084] Step 306 comprises determining, for at least one pixel of the synthetic digital image , a gradient size of a parameter φ of a difference ε θ (z t (Φ), t, y) - ε between a predicted noise ε θ (z t (Φ), t, y) for the pixel and a noise sample ε for the pixel.
[0085] The difference can be a difference ε θ (z t (Φ), t, y) - ε weighted by a variable weight ω(t).
[0086] For example, the pixel-wise gradient
[0087]
[0088] is determined, where E is an expected value. The larger the size of the gradient, the worse the quality of the estimated pixel is represented.
[0089] The reverse denoising process 208 comprises determining a stepwise successive linear combination, wherein the noise latent image z t-1 is a prediction noise ∈ θ of the previous step t t , y) and the noise latent image z t .
[0090] The step 306 can comprise determining the size of the gradient stepwise and determining a mean of the stepwise determined sizes.
[0091] The mean is one example of a measure. The measure can be a result of performing an argmax operation on the gradient sizes of the pixels. The measure can be a mean of the pixel gradient sizes. The measure can be a variance of the pixel gradient sizes.
[0092] The step 306 can comprise determining a pixel-wise size of a plurality of pixels of the synthetic digital image .
[0093] The step 306 can comprise determining a measure for a plurality of pixels of the synthetic digital image pixel-wise. The measure is for example associated with the pixels of the plurality of pixels for which the respective measure is determined.
[0094] The method can comprise a step 308.
[0095] The step 308 can comprise providing a threshold value for the measure and picking out the synthetic digital image in case the measure exceeds the threshold value. The threshold value can be a measure value which indicates that the quality of the synthetic digital image is too low for picking out a synthetic digital image of poor quality.
[0096] The step 308 can comprise outputting an error heat map which visualizes the measure pixel-wise.
[0097] The step 308 can comprise determining a pixel region 218, in particular a bounding box, of the synthetic digital image from the measure.
[0098] Determining the region can comprise identifying a region comprising pixels associated with a measure which is greater than a measure associated with a pixel outside the region from the measure.
[0099] Determining the region can comprise determining a mean and a variance of the measure and determining a region comprising pixels associated with a measure which lies within the variance around the mean.
[0100] The method can comprise a step 310.
[0101] The step 310 comprises replacing a pixel in the synthetic digital image with a random noise. The step 310 for example comprises replacing the synthetic digital image pixels in the synthetic digital image to determine another synthetic digital image. Step 310 comprises replacing the pixels in the region in the synthetic digital image with pixels in the other synthetic digital image.
[0102] The method can comprise step 312.
[0103] Step 312 comprises determining another input for the text-to-image diffusion, the input representing a synthetic digital image comprising random noise in the region.
[0104] After step 312, the method can continue to step 304 for determining another synthetic digital image with the text-to-image diffusion 200 from the other input, from another noise sample, and from the embedding y representing the text.
[0105] The method for example comprises replacing the pixels in the synthetic digital image determined with the text-to-image diffusion 200 to determine another synthetic digital image, and repeating the determining of the measure of the synthetic digital image until the measure determined for the synthetic digital image satisfies a condition.
[0106] The condition can be that the measure value is smaller than a threshold value, the threshold value indicating that the quality of the synthetic digital image is sufficiently high.
[0107] The text-to-image diffusion 200 can be trained and used for the purpose of anomaly detection in real images.
[0108] The digital image x can be a digital image of a real-world technical component. The text can be a description of an anomaly in the real-world technical component that should be depicted in the synthetic digital image The text-to-image diffusion model 200 can be trained on a restricted image domain, the restricted image domain comprising digital images of real-world technical components from the domain.
[0109] The synthetic digital image is for example determined for the purpose of anomaly detection. For example, the size of the error heat map or the gradients in the region indicates that the synthetic digital image is not usable for training or testing a training set of an anomaly detection system, such as a machine learning system, with the synthetic digital image for anomaly detection. The region or the error heat map is determined for the purpose of picking out the synthetic digital image.
Claims
1. A computer-implemented method for digital image processing, characterized in that, determining (304) a synthetic digital image with a text-to-image diffusion (200) from an input (z0) representing a digital image (x), from a noise sample (∈ t ) and from an embedding (y) representing text, wherein the text-to-image diffusion (200) comprises a forward diffusion process (206) to determine a noisy latent image (z t ) from the input and the noise sample, wherein the noisy latent image (z t ) is parameterized by a parameter (Φ), wherein the text-to-image diffusion (200) comprises a backward denoising process (208) to determine an output representing a synthetic digital image t (z0) from a linear combination of the noisy latent image (z θ ) and the predicted noise (∈ θ ), wherein the synthetic digital image comprises pixels, wherein the method comprises determining (306) for at least one pixel a gradient magnitude of a parameter (Φ) of a difference between a predicted noise (∈ t ) for the pixel and a noise sample (∈ t-1 ) for the pixel. In particular the difference is weighted by a variable weight.
2. The method of claim 1, wherein, The inverse denoising process comprises determining a stepwise successive linear combination, wherein a noise latent image (z t-1 ) of a step (t) is a result of a linear combination of a predicted noise (e θ ) of a preceding step (t) and a noise latent image (z t ), wherein the method comprises determining (306) a size of the gradient stepwise and determining a measure from the stepwise determined size, in particular a mean, an argmax, a mean value or a variance of the stepwise determined size.
3. The method of claim 2, wherein, The method includes providing (308) a threshold for the metric, and picking out the synthetic digital image if the metric exceeds the threshold.
4. The method according to one of claims 2 or 3, characterized in that, The method includes determining (306) the size pixel by pixel for a plurality of pixels of the synthetic digital image, and outputting (308) an error heat map visualizing the metric pixel by pixel.
5. The method according to one of claims 2 to 4, characterized in that, The method comprises determining (306), pixel by pixel, for a plurality of pixels of the synthetic digital image a measure, wherein the measure is associated with a pixel of the plurality of pixels for which a respective measure is determined, and wherein the method comprises determining (308), from the measures, a region of pixels (218), in particular a bounding box, of the synthetic digital image.
6. The method of claim 5, wherein, Determining (308) the region (218) includes identifying a region comprising pixels that are associated with a greater metric than a metric associated with pixels outside the region, according to the metric.
7. The method of claim 5, wherein, Determining (308) the region (218) includes determining a mean and a variance of the metric, and determining a region comprising pixels that are associated with a metric within the variance that is located near the mean.
8. The method according to one of claims 5 to 7, characterized in that, The method comprises replacing (310) pixels in the synthetic digital image with random noise determining (312) another input to the text-to-image diffusion representing the synthetic digital image including the random noise in the region, and determining (304) another synthetic digital image having the text-to-image diffusion from the other input, from another noise sample, and from an embedding (y) representing the text.
9. The method of claim 8, wherein, The method includes replacing (310) the synthetic digital image. The pixels in the image are used to determine another synthetic digital image and the measure of the other synthetic digital image until the measure determined for the other synthetic digital image satisfies the condition.
10. The method according to any of the preceding claims, characterized in that, For different text embeddings representing text describing anomalies in real-world technical components, the text is diffused (200) into images to determine a plurality of synthetic digital images for training or testing an anomaly detection system to identify anomalies in digital images of real-world components.
11. An apparatus (100) for digital image processing, characterized in that The device (100) comprises at least one processor (102) and at least one memory (104), wherein the at least one memory (104) is configured to store instructions executable by the at least one processor (102), and the instructions, when executed by the at least one processor (102), cause the device (100) to perform the method according to one of claims 1 to 10.
12. A computer program for digital image processing, characterized in that, The computer program comprises instructions executable by a computer, and the instructions, when executed by the computer, cause the computer to perform the method according to one of claims 1 to 10.