Apparatus and computer-implemented method for processing digital images
The method addresses artifact removal in composite digital images by determining gradient magnitudes and iteratively refining images, enhancing image quality through a backward denoising process.
Patent Information
- Application Number
- JP2025115968
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-07-10
- Filing Date
- 2025-07-09
- Publication Date
- 2026-01-23
AI Technical Summary
Existing methods for creating composite digital images from text and images struggle with identifying and removing unnatural artifacts, leading to low-quality outputs.
A computer-implemented method using text-to-image diffusion that determines gradient magnitudes to identify and automatically remove unnatural artifacts in composite digital images, employing a backward denoising process and thresholding to improve image quality.
The method effectively identifies and removes artifacts, resulting in improved natural-looking composite digital images by iteratively refining the image until desired quality metrics are met.
Smart Images
Figure 2026012138000069 
Figure 2026012138000070 
Figure 2026012138000071
Abstract
Description
[Technical Field]
[0001] background The present invention relates to an apparatus and a computer-implemented method for processing digital images. [Background technology]
[0002] Stable Diffusion and ControlNet can be used to create a composite digital image from text and a given digital image, and automated text-to-image alignment metrics can be used to verify the quality of the composite digital image.
[0003] "High-Resolution Image Synthesis with Latent Diffusion Models" (arXiv:2112.10752) describes Stable Diffusion. "Adding Conditional Control to Text-to-Image Diffusion Models" (arXiv:2302.05543) describes ControlNet. "DreamFusion: Text-to-3D using 2D Diffusion" (arXiv:2209.14988) describes a score distillation sampling loss for text-to-3D synthesis. "If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection" (arXiv:2305.13308) describes an automated text-to-image alignment metric. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] “High-Resolution Image Synthesis with Latent Diffusion Models”(arXiv:2112.10752) [Non-patent document 2] “Adding Conditional Control to Text-to-Image Diffusion Models”(arXiv:2302.05543) [Non-patent document 3] “DreamFusion: Text-to-3D using 2D Diffusion”(arXiv:2209.14988) [Non-patent document 4] “If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection”(arXiv:2305.13308) Summary of the Invention [Means for solving the problem]
[0005] Disclosure of the Invention A computer-implemented method for processing a digital image includes determining a composite digital image using text-to-image diffusion depending on an input representing the digital image, depending on a noise sample, and depending on an embedding representing the text, wherein the text-to-image diffusion includes a forward diffusion process for determining a noisy latent representation depending on the input and the noise sample, the noisy latent representation being parameterized by a parameter, and the text-to-image diffusion includes a backward denoising process depending on a linear combination of the noisy latent representation and a predicted noise for determining an output representing the composite digital image, the composite digital image including a plurality of pixels, and the method includes determining, for at least one pixel, a gradient magnitude with respect to a parameter of the difference between the predicted noise for that pixel and the noise sample for that pixel, particularly the difference weighted by a variable weight. The method automatically indicates natural-looking and unnatural-looking artifacts in the composite digital image by magnitude.
[0006] The backward denoising process may include determining successive linear combinations at each step, where the noisy latent representation at a step is the result of a linear combination of the noisy latent representation at a previous step and the predicted noise, and the method includes determining a gradient magnitude at each step and determining a metric dependent on the magnitude determined at each step, in particular a representative value, argmax, mean, or variance of the magnitude determined at each step. The metric provides feedback for each pixel and identifies whether the pixel is an artifact based on the magnitude determined for that pixel.
[0007] The method may include thresholding the metric and screening the composite digital image if the metric exceeds the threshold, whereby the composite digital image is automatically screened as containing artifacts if the metric exceeds the threshold.
[0008] The method may include determining a magnitude for each pixel of a plurality of pixels of the composite digital image and outputting an error heatmap that visualizes the metric for each pixel, the heatmap depicting or describing the location of the artifact in the composite digital image.
[0009] The method may comprise determining a metric for each of a plurality of pixels of the composite digital image, the metric being associated with a pixel of the plurality of pixels for which a respective representative value is determined, and the method comprises determining a region, in particular a bounding box, of pixels of the composite digital image in dependence on the metric, which means that a region containing at least one artifact is identified.
[0010] Determining the region may involve identifying a region that includes pixels associated with a metric that is greater than the metric associated with pixels outside the region, depending on the metric.
[0011] Determining the region may include determining a mean and variance of the metric, and determining a region that includes pixels associated with the metric that are within the variance around the mean.
[0012] The method may include replacing pixels in the composite digital image by random noise, determining another input representing the composite digital image with the random noise in the region for text-to-image diffusion, and determining another composite digital image using text-to-image diffusion depending on the another input, depending on the another noise sample and depending on the embedding representing the text, which means that the digital image is improved.
[0013] The method may include determining another composite digital image by replacing pixels in the composite digital image and determining metrics for the another composite digital image until the metrics determined for the another composite digital image satisfy a condition, which means that the digital image is improved until a desired condition, e.g., quality, is met.
[0014] The method may include determining a plurality of synthetic digital images for training or testing an anomaly detection system to recognize anomalies in digital images of the real-world component using text-to-image diffusion against a plurality of different text embeddings representing text that describes anomalies in the real-world technical component.
[0015] An apparatus for processing digital images comprises at least one processor and at least one memory, the at least one memory configured to store instructions that are executable by the at least one processor and that, when executed by the at least one processor, cause the apparatus to perform the method described above.
[0016] A computer program for processing digital images comprises instructions that are executable by a computer and that, when executed by the computer, cause the computer to carry out the method described above.
[0017] Further advantageous embodiments will become apparent from the following description and drawings. [Brief explanation of the drawings]
[0018] [Figure 1] 1 shows a schematic diagram of an apparatus for processing digital images; [Figure 2] FIG. 1 is a schematic diagram illustrating exemplary text-to-image diffusion. [Figure 3] 1 is a flowchart including steps of a method for processing a digital image. DETAILED DESCRIPTION OF THE INVENTION
[0019] The digital image may be a video image, a radar image, a LiDAR image, an ultrasound image, a motion image, or a thermal image.
[0020] FIG. 1 shows a schematic diagram of an apparatus 100 for processing digital images.
[0021] The device 100 includes at least one processor 102 and at least one memory 104 .
[0022] The at least one processor 102 is configured to execute instructions that cause the device 100 to perform a method for processing digital images. The at least one memory 104 is configured to store instructions. The at least one memory 104 may include temporary and / or non-temporary memory.
[0023] The device 100 may include a computer configured to execute instructions. A computer program for processing digital images may include the instructions.
[0024] FIG. 2 shows a schematic diagram of an exemplary text to image diffusion 200 based on Stable Diffusion 202 and ControlNet 204 .
[0025] The text-to-image diffusion 200 is implemented, for example, as an artificial neural network.
[0026] Text-to-image diffusion 200 operates in the latent space Z. Text-to-image diffusion 202 is based on an input z0 in the latent space Z, which represents a digital image x.
[0027] The encoder ε is configured to determine an input z0 representing a digital image x. The encoder ε projects a given digital image x from image space into a spatial latent code z0 = ε(x). The encoder ε may be implemented as part of an artificial neural network or as a separate artificial neural network.
[0028] The text-to-image diffusion 200 is based on the noise samples z0.
number
number
[0029] Text-to-image diffusion 200 uses input z0 and noise samples in Stable Diffusion 202.
number
[0030] Noisy latent representation z t (Φ) is parameterized by the parameter Φ.
[0031] Text-to-image diffusion 200 involves the generation of a synthetic digital image in a latent space Z in Stable Diffusion 202.
number
number
[0032] The backward denoising process 208 includes an encoder 210 and a decoder 212, for example a convolutional neural network according to the UNet architecture.
[0033] To account for text in text-to-image diffusion 200, ControlNet 204 includes a trainable copy 210′ of at least one portion of backward denoising process 208. For example, ControlNet 204 includes a trainable copy of encoder 210.
[0034] The ControlNet 204 is configured to determine the inputs 214 for the backward noise reduction process 208 .
[0035] The ControlNet 204 is configured, for example, to determine inputs 214 at multiple successive zero-convolutional layers 216. The ControlNet 204 is configured, for example, to determine one input 214 for each successive zero-convolutional layer 216.
[0036] A zero convolutional layer is, for example, a 1x1 convolutional layer in which both the weights and bias are initialized as 0.
[0037] Decoder D outputs
number
number
[0038] According to one example, the decoder D generates a spatial latent code from the latent space Z.
number
number
[0039] For example, text-to-image diffusion 200 operates in the latent space Z of an autoencoder that includes an encoder ε and a decoder D.
[0040] For example, the encoder ε and decoder D are trained with a set of digital images to reconstruct a given image x.
number
[0041] Stable Diffusion 202 involves multiple steps t to gradually add noise to input z0. At step t, the noise sample
number
[0042] The forward diffusion process 206 includes, for example, a Markov chain of length T to gradually add noise.
number
number
[0043] Noisy latent representation z t is, for example, in closed form, e.g.,
number
number
[0044] The backward noise reduction process 208 may, for example, use another Gaussian distribution
number
number
number
[0045] Stable Diffusion202 output
number
number
[0046] Stable Diffusion 202 is trained in the latent space Z, for example, to minimize the L2 norm of the noise prediction at sampled step t.
number
[0047] The ControlNet 204 calculates, for example, the predicted noise at sampled time step t, conditioned on the text embedding y.
number
number
[0048] Stable Diffusion 202 is frozen during the training of ControlNet 204.
[0049] By training a neural network using the loss mentioned above, we can ensure that the neural network can accurately predict the noise samples z T A, composite digital image
number
number
[0050] Latent representation z at step t t (Φ) is parameterized by the parameter Φ.
[0051] To edit a given digital image x depending on the text represented by the embedding y,
number
number
number
[0052] The loss L(Φ) itself is difficult to compute, but it is the latent representation z parameterized by the parameters Φ at step t. t Gradient at (Φ)
number
number
[0053] This means that for a given noise sample
number
number
[0054] Digital image x and composite digital image
number
[0055] This means that the gradient ∇ Φ L(Φ) is the predicted noise for a pixel.
number
number
number
[0056] According to one example, the difference
number
[0057] Gradient ∇ Φ The magnitude of L(Φ) is larger in the region 218 that is more likely to contain artifacts. Therefore, this region 218 is expressed as the gradient ∇ Φ It is possible to distinguish based on the magnitude of L(Φ).
[0058] FIG. 3 is a flow chart including steps of a method for processing a digital image.
[0059] The method includes step 302 .
[0060] Step 302 involves computing a digital image x and noise samples
number
[0061] The method includes step 304 .
[0062] Step 304 uses text-to-image diffusion 200 to generate noise samples z0 = ε(x) depending on the input z0 = ε(x) representing the digital image x.
number
number
[0063] Step 302 or step 304 may include providing an input z0=ε(x).
[0064] Text-to-image diffusion is done by using input z0 and noise samples
number
[0065] Text-to-image diffusion 200 predicts
number
number
number
[0066] This means that the output
number
number
[0067] The composite digital image comprises a plurality of pixels.
[0068] The method includes step 306 .
[0069] Step 306 generates a composite digital image
number
number
number
number
[0070] This difference is weighted by a variable weight ω(t).
number
[0071] For example, per-pixel gradients
number
[0072] The backward denoising process 208 involves determining successive linear combinations at each step to find the noisy latent representation z t-1 is the noisy latent representation z t and the predicted noise
number
[0073] Step 306 may include determining the magnitude of the gradient for each step and determining a representative value of the magnitude determined for each step.
[0074] The median value is one example of a metric. The metric may be the result of an argmax operation performed on the gradient magnitude of a pixel. The metric may be the mean value of the gradient magnitude of a pixel. The metric may be the variance in the gradient magnitude of a pixel.
[0075] Step 306 generates a composite digital image
number
[0076] Step 306 may include determining a metric for each of a plurality of pixels of the composite digital image, the metric being associated with, for example, the pixel of the plurality of pixels for which the respective metric is determined.
[0077] The method may include step 308 .
[0078] Step 308 may include setting a threshold for the metric and filtering out the composite digital image if the metric exceeds the threshold, which may be a value of the metric that indicates that the quality of the composite digital image is too low, in order to filter out low-quality composite digital images.
[0079] Step 308 may include outputting an error heatmap that visualizes the metric pixel by pixel.
[0080] Step 308 generates a composite digital image based on the metric.
number
[0081] Determining the region may involve identifying a region that includes pixels associated with a metric that is greater than the metric associated with pixels outside the region, depending on the metric.
[0082] Determining the region may include determining a mean and variance of the metric, and determining a region that includes pixels associated with the metric that are within the variance around the mean.
[0083] The method may include step 310 .
[0084] Step 310 generates a composite digital image.
number
number
number
[0085] The method may include step 312 .
[0086] Step 312 involves determining another input representing a synthetic digital image containing random noise in the region for text-to-image diffusion.
[0087] After step 312, the method may continue with step 304 of determining another composite digital image using text-to-image diffusion 200 depending on another input, depending on another noise sample, and depending on the embedding y representing the text.
[0088] The method includes, for example, determining another composite digital image by replacing pixels in the composite digital image determined using text-to-image diffusion 200, and iteratively determining metrics for the composite digital image until the metrics determined for the composite digital image satisfy a condition.
[0089] The condition may be that the value of the metric is below a threshold that indicates that the quality of the composite digital image is sufficiently high.
[0090] Text-to-image diffusion 200 may be trained and used to detect anomalies in real-world images.
[0091] The digital image x may be a digital image of a real-world technical component. The text may be a composite digital image.
number
[0092] The composite digital image may be determined, for example, for the purpose of detecting anomalies, e.g., regions or error heatmaps may be determined to screen a composite digital image when the magnitude of the gradients in the error heatmap or regions indicates that the composite digital image is unusable for use against a training set for training or testing an anomaly detection system, e.g., a machine learning system, using the composite digital image to detect anomalies.
Claims
1. 1. A computer-implemented method for processing a digital image, comprising: The method comprises: Text-to-image diffusion (200) is used to generate an input (z) representing a digital image (x). ０ ) depending on the noise sample [Equation 1] and determining (304) a composite digital image in dependence on an embedding (y) representing the text; The text-to-image diffusion (200) generates a noisy latent representation (z ｔ ), and a forward diffusion process (206) for determining the noisy latent representation (z ｔ ) is parameterized by a parameter (Φ), The text-to-image diffusion (200) is performed by extracting the noisy latent representation (z ｔ ) and the predicted noise [Equation 2] and the composite digital image [Equation 3] a backward denoising process (208) for determining an output representing the composite digital image, the composite digital image comprising a plurality of pixels; The method comprises: For at least one pixel, the predicted noise for that pixel [Equation 4] and the noise sample for that pixel [Equation 5] In particular, the gradient (∇) with respect to the parameter (Φ) of the weighted difference between Φ determining (306) the magnitude of L(Φ); A method characterized by:
2. the backward denoising process includes determining successive linear combinations stepwise; The noisy latent representation (z ｔ－１ ) is the noisy latent representation (z ｔ ) and the predicted noise [Equation 6] is the result of a linear combination with The method comprises: determining (306) the magnitude of the gradient stepwise; determining a metric dependent on the magnitudes determined at each step, in particular a mean value, argmax, a mean or a variance of the magnitudes determined at each step; The method of claim 1 , comprising:
3. The method comprises: establishing a threshold for the metric (308); culling the composite digital image if the metric exceeds the threshold; The method of claim 2 , comprising:
4. The method comprises: determining (306) the magnitude for each pixel of the composite digital image; outputting an error heatmap that visualizes the metric pixel by pixel (308); The method of claim 2 or 3, comprising:
5. The method comprises: said composite digital image [Equation 7] determining (306) the metric for each pixel for a plurality of pixels of the metric is associated with a pixel of the plurality of pixels for which the respective metric is determined; The method comprises:
5. The method according to any one of claims 2 to 4, comprising determining (308) a region (218), in particular a bounding box, of pixels of the composite digital image depending on said metric.
6. 6. The method of claim 5, wherein determining (308) the region (218) comprises identifying a region that includes pixels associated with a metric that is greater than a metric associated with pixels outside the region, depending on the metric.
7. Determining (308) the region (218) comprises: determining the mean and variance of said metric; determining a region containing pixels associated with a metric that is within the variance about the mean; The method of claim 5 , comprising:
8. The method comprises: said composite digital image [Equation 8] replacing pixels in with random noise (310); determining (312) another input representing the synthetic digital image including the random noise within the region for the text-to-image diffusion; determining (304) a further composite digital photograph using the text-to-image diffusion in dependence on the further input, in dependence on the further noise samples, and in dependence on the embedding (y) representing the text; 8. The method of claim 5, comprising:
9. The method comprises: said composite digital image [Equation 9] determining another composite digital image by replacing pixels in the image and determining metrics for the another composite digital image until the metrics determined for the another composite digital image satisfy a condition (310). The method of claim 8.
10. The method comprises:
10. The method of claim 1, comprising determining a plurality of synthetic digital images for training or testing an anomaly detection system to recognize anomalies in digital images of real-world components using the text-to-image diffusion (200) for a plurality of different text embeddings representing text describing anomalies in real-world technical components.
11. An apparatus (100) for processing digital images, comprising: The device (100) comprises: at least one processor (102); At least one memory (104); Equipped with the at least one memory (104) configured to store instructions; The instructions are executable by the at least one processor (102) and, when executed by the at least one processor (102), cause the apparatus (100) to perform the method of any one of claims 1 to 10.
1. An apparatus (100) comprising:
12. 1. A computer program for processing digital images, comprising: The computer program includes instructions: The instructions are computer executable and, when executed by the computer, cause the computer to perform the method of any one of claims 1 to 10. A computer program characterized by: