Image processing method for accelerating a denoising process and electronic device for performing the method

Abstract

An image processing method includes: providing a noisy image to a diffusion generative model; and obtaining a noise-reduced restored image by performing a denoising process on the image using the diffusion generative model, wherein the denoising process includes multiple processing steps, and wherein obtaining the restored image includes: in a first processing step, obtaining a feature map using the diffusion generative model and storing the feature map in a memory; and determining that a parameter value is greater than a threshold based on a transformation associated with a second processing step; in a second processing step, transforming feature values ​​included in the feature map stored in the memory based on transformation parameter values ​​associated with the second processing step; and in a third processing step, providing the feature map including the transformed feature values ​​to the diffusion generative model.

Description

Cross-references to related applications

[0001] This application is based on and claims priority to Korean Patent Application No. 10-2025-0003800 filed with the Korean Intellectual Property Office on January 10, 2025, and Korean Patent Application No. 10-2025-0035427 filed with the Korean Intellectual Property Office on March 19, 2025, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure relates to a denoising technique for reducing noise in an image, and an image processing technique using a generative model. Background Technology

[0003] Since the advent of artificial intelligence (AI) modeling technology, the development and research of AI models have progressed rapidly. Among various AI models, generative AI models can be used to generate new data based on given input. Some examples of generative AI models can include generative pre-trained transformers (GPTs) and generative adversarial networks (GANs), where a GPT can refer to a model that generates new text based on given text, and a GAN can refer to a model where two networks compete against each other to generate increasingly realistic images.

[0004] A diffusion model can be defined as a model that generates target data while gradually removing noise from the original data. A diffusion model can be learned or trained using a process that involves gradually adding noise to the original data to generate complex random states, and then gradually removing the noise while restoring the original data. Diffusion models can be used for image processing tasks, such as image denoising to reduce noise in images, generating images from text input, or performing image inpainting to fill in or restore missing parts of an image. Summary of the Invention

[0005] According to one aspect of this disclosure, an image processing method performed by an electronic device includes: providing a noisy image to a diffusion generative model; and obtaining a noise-reduced restored image by performing a denoising process on the image using the diffusion generative model, wherein the denoising process includes a plurality of processing steps, including a first processing step, a second processing step following the first processing step, and a third processing step following the second processing step, and wherein obtaining the restored image includes: in the first processing step, using the diffusion generative model to obtain a feature map corresponding to the input of the diffusion generative model, storing the feature map in a memory, and determining that a parameter value is greater than a threshold based on a transformation associated with the second processing step; in the second processing step, transforming the feature values ​​included in the feature map stored in the memory based on the transformation parameter values ​​associated with the second processing step; and in the third processing step, providing the feature map including the transformed feature values ​​to the diffusion generative model.

[0006] A restored image can be obtained by performing only some of the processing steps selected from the multiple processing steps included in the denoising process.

[0007] The number of times a processing step is executed in these processing steps can correspond to the target number.

[0008] Transforming the eigenvalues ​​of a feature map can include performing linear or nonlinear transformations on the eigenvalues ​​of the feature map based on transformation parameter values.

[0009] The diffusion generative model may include a transformer model, which may include an attention block and a feedforward block, and the attention block may be configured to output a first feature map and the feedforward block may be configured to output a second feature map.

[0010] Storing feature maps in memory can include storing a first feature map and a second feature map in memory.

[0011] In each of the multiple processing steps, a first transformation determination parameter value can be determined for the attention block, and a second transformation determination parameter value can be determined for the feedforward block. Obtaining the restored image may include: in the second processing step, determining whether to transform the first feature map obtained in the first processing step based on the first transformation determination parameter value; and in the second processing step, determining whether to transform the second feature map obtained in the first processing step based on the second transformation determination parameter value.

[0012] Acquiring the restored image may include: determining that parameter values ​​are less than or equal to a threshold based on a transformation associated with the second processing step; and in the third processing step, providing the feature map stored in memory to the diffusion generative model.

[0013] According to one aspect of this disclosure, an electronic device includes: a processor; and a memory configured to store instructions, wherein, when executed by the processor, the instructions cause the electronic device to: provide an image including noise to a diffusion generative model, and perform a denoising process on the image using the diffusion generative model to obtain a noise-reduced restored image, wherein the denoising process includes a plurality of processing steps, the plurality of processing steps including a first processing step, a second processing step following the first processing step, and a third processing step following the second processing step, and wherein, when executed by the processor, the instructions cause the electronic device to: in the first processing step, use the diffusion generative model to obtain a feature map corresponding to the input of the diffusion generative model, store the feature map in the memory, and determine that a parameter value is greater than a threshold based on a transformation associated with the second processing step; in the second processing step, transform the feature values ​​included in the feature map stored in the memory based on the transformation parameter values ​​associated with the second processing step; and in the third processing step, provide the feature map including the transformed feature values ​​to the diffusion generative model.

[0014] Instructions, when executed by the processor, can also enable electronic devices to:

[0015] The restored image is obtained by performing only some of the processing steps selected from the multiple processing steps included in the denoising process.

[0016] The number of times a processing step is executed in these processing steps can correspond to the target number.

[0017] When executed by the processor, the instructions can also enable the electronic device to perform linear or nonlinear transformations on the feature values ​​of the feature map based on the transformation parameter values.

[0018] The diffusion generative model may include a transformer model, which may include an attention block and a feedforward block, and the attention block may be configured to output a first feature map and the feedforward block may be configured to output a second feature map.

[0019] Instructions, when executed by the processor, can also enable electronic devices to:

[0020] The first feature map and the second feature map are stored in memory.

[0021] In each of the multiple processing steps, a first transformation determination parameter value can be determined for the attention block, and a second transformation determination parameter value can be determined for the feedforward block. When the instruction is executed by the processor, the electronic device can also: in the second processing step, determine whether to transform the first feature map acquired in the first processing step based on the first transformation determination parameter value, and in the second processing step, determine whether to transform the second feature map acquired in the first processing step based on the second transformation determination parameter value.

[0022] When the instructions are executed by the processor, the electronic device may also: determine that the parameter value is less than or equal to a threshold based on the transformation associated with the second processing step, and in the third processing step, provide the feature map stored in memory to the diffusion generative model.

[0023] According to one aspect of this disclosure, an inference accelerator system includes: a memory configured to store model information for implementing a diffusion generative model; a cache memory; and a processor configured to perform a denoising process for reducing noise included in an input image using the diffusion generative model, wherein the denoising process includes a plurality of processing steps, the plurality of processing steps including a first processing step, a second processing step following the first processing step, and a third processing step following the second processing step, and wherein the processor is further configured to: in the first processing step, obtain a feature map corresponding to the input of the diffusion generative model using the diffusion generative model, store the feature map in the cache memory, and determine that a parameter value is greater than a threshold based on a transformation determined for the second processing step; in the second processing step, transform the feature values ​​of the feature map stored in the cache memory based on the transformation parameter values ​​associated with the second processing step; and in the third processing step, provide the feature map including the transformed feature values ​​to the diffusion generative model.

[0024] Diffusion generative models can include transformer models, and transformer models can include attention blocks and feedforward blocks.

[0025] The processor can also be configured to perform only some of the multiple processing steps included in the denoising process.

[0026] The processor can also be configured to perform a linear transformation on the feature values ​​of the feature map based on the transformation parameter values, and the linearly transformed feature values ​​can be stored in a cache memory. Attached Figure Description

[0027] The above and other aspects, features, and advantages of certain embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0028] Figure 1 This is a diagram illustrating the configuration of an electronic device performing a noise reduction process according to an embodiment;

[0029] Figure 2 This is a diagram illustrating the denoising process performed using a diffusion-generative model according to an embodiment;

[0030] Figure 3 This is a diagram illustrating the configuration of an inference accelerator for accelerating the denoising process according to an embodiment;

[0031] Figure 4 This is a diagram illustrating the learning process of the denoising process according to an embodiment;

[0032] Figure 5 This is a diagram illustrating a feature map generated from a diffusion generative model by a cache according to an embodiment;

[0033] Figure 6 This is a diagram illustrating the transformation processing performed on the cached feature map according to an embodiment;

[0034] Figure 7 This diagram illustrates the transformation process performed on the feature map based on the feature map generated from the diffusion generative model and the cached feature map according to an embodiment;

[0035] Figure 8 This diagram illustrates the caching and transformation of a feature map according to processing steps, based on an embodiment.

[0036] Figure 9 and Figure 10 This is a flowchart illustrating the operation of an image processing method for performing a denoising process according to an embodiment; and

[0037] Figure 11 This is a block diagram illustrating the configuration of an inference accelerator system performing a denoising process according to an embodiment. Detailed Implementation

[0038] The detailed structural or functional descriptions below are provided as examples only, and various changes and modifications can be made to the described embodiments without departing from the scope of this disclosure. Therefore, the embodiments of this disclosure should not be construed as limited to the specific embodiments described herein, but should be understood to include all changes, equivalents, and substitutions within the spirit and technical scope of this disclosure.

[0039] While terms such as "first" or "second" may be used to describe various components, these components are not limited to these terms. These terms should only be understood as being used to distinguish one component from another. For example, the first component can be referred to as the second component, and similarly, the second component can be referred to as the first component.

[0040] It will be understood that when a component is referred to as being "connected to" or "coupled to" another component, the component may be directly connected to or coupled to that other component, or there may be other intermediate components.

[0041] The singular forms used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising / including” and / or “containing / comprises” as used herein mean the presence of the said feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0042] When expressions such as "at least one of..." follow a list of components, they modify the entire list of components rather than individual components within the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0043] The term "module" as used in conjunction with embodiments of this disclosure may include a unit implemented as hardware, software, or firmware, and may be used interchangeably with other terms such as "logic," "logic block," "component," or "circuit." A module may be a block of code that performs a predetermined function or task and may be configured to form a larger program or software system through interaction with other modules. Alternatively, a module may refer to a hardware component or device capable of performing a function independently, and such modules may be combined with other hardware to form an entire system. A module may be a single integrated component adapted to perform one or more functions, or its smallest unit or portion thereof. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0044] At least one operation described in the embodiments of this disclosure can be executed simultaneously or in parallel with other operations, and the order of operations can be changed. Additionally, at least one operation can be omitted, or other operations can be performed.

[0045] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms such as those defined in common dictionaries shall be interpreted as having a meaning consistent with their meaning in the context of the relevant art and shall not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0046] In the following description, embodiments are described in detail with reference to the accompanying drawings. When describing embodiments with reference to the accompanying drawings, the same reference numerals refer to the same elements, and any repetitive descriptions associated with them will be omitted.

[0047] Figure 1 This is a diagram illustrating the configuration of an electronic device performing a noise reduction process according to an embodiment.

[0048] refer to Figure 1 Electronic device 100 can perform a denoising process to reduce noise in a given image. When an image 130 containing noise is provided, electronic device 100 can reduce noise by using a diffusion generative model (e.g., Figure 3 A diffusion generative model 300 performs a denoising process on image 130 to generate a restored image 140 with reduced noise compared to image 130. The diffusion generative model can be used when performing the denoising process, and it can also be used when generating images based on text input or when filling or restoring missing parts of an image.

[0049] Diffusion generative models can be, or can include, deep learning-based generative models that progressively generate images. A diffusion generative model can be a model trained on a forward diffusion process that progressively adds noise to the original image to generate a noisy image, and a backward diffusion process that progressively removes noise from the noisy image to recover the original image. A diffusion generative model can be, for example, a transformer-based diffusion generative model. A transformer-based diffusion generative model can learn the entire pattern through self-attention and provide high-quality image generation results. A transformer-based diffusion generative model can be, but is not limited to: a diffusion transformer (DiT) that performs diffusion via a transformer; or a model that combines DiT with a deterministic diffusion implicit model (DDIM), which can be defined as DiT-XL DDIM. Diffusion generative models can also be implemented as generative models including an encoder, a denoising model (or a denoising network), and a decoder. Reference Figure 2 Examples of diffusion generative models are described in more detail.

[0050] Electronic device 100 can use a diffusion generative model to accelerate the inference process. The inference process can be the process of inferring a noise-reduced restored image 140 from a noisy image 130. During the inference process, a denoising process can be performed on the noisy image 130 to generate the noise-reduced restored image 140. Electronic device 100 can reuse or transform (e.g., linear or nonlinear transformations) the feature maps generated during the inference process. To reuse the feature maps, caching techniques can be used to store the feature maps generated in previous steps in memory 120, and the feature maps stored in memory 120 can be loaded in the next step. By reusing or transforming the feature maps, the computational cost of computing the feature maps can be reduced. The feature maps can be intermediate representations of the input data of the diffusion generative model and can include feature values, where the feature values ​​include vector values ​​output from each block or layer of the diffusion generative model. Within a block of the diffusion generative model, feature maps output from blocks in previous steps can be passed to blocks in the next step to generate increasingly higher-dimensional feature maps. Furthermore, electronic device 100 can accelerate the inference process of the diffusion generative model while maintaining the high quality of the restored image 140 by optimizing the processing steps of the denoising process during the inference process of the diffusion generative model. Optimizing the processing steps may include reducing the amount of computation required during the inference process by performing only selected portions of the processing steps included in the denoising process instead of performing all the processing steps included in the denoising process. As the amount of computation used in the inference process is reduced, the power consumption of electronic device 100 can be reduced, thereby improving power efficiency and increasing the speed of the inference process using the diffusion generative model. The term "processing step" as used herein may correspond to a "processing stage" or "time step" indicating a processing step.

[0051] The methods described above for accelerating the inference process can be used in conjunction with other inference acceleration techniques, such as quantization and / or pruning. Quantization can refer to techniques that improve computation speed and reduce memory usage by reducing the data representation scheme or the number of bits used to represent the data. Pruning can refer to techniques that reduce computation by setting small weights to zero ("0") in an artificial intelligence (AI) model, such as a transformer, or by reducing computation and improving inference speed by not using specific components of the AI ​​model (e.g., layers, neurons, blocks).

[0052] Electronic device 100 may include processor 110 and memory 120. Electronic device 100 may include other components besides these. For example, electronic device 100 may also include at least one of display circuitry for outputting a noisy image 130 or restoring an image 140 and communication circuitry for communicating with other devices.

[0053] Processor 110 can execute programs or software to control other components (e.g., hardware or software components) of electronic device 100 connected to processor 110, and can perform various data processing or operations. As at least part of the data processing or operation, processor 110 can process instructions and data stored in memory 120 and store the processed result data in memory 120. Processor 110 can perform the operations of electronic device 100 described herein or algorithms corresponding to the operations. Processor 110 can be a hardware-implemented data processing device including circuitry with a physical structure to perform the desired operation. For example, the desired operation can be implemented by code or instructions included in a program.

[0054] Processor 110 may include at least one of a main processor (e.g., a central processing unit (CPU) or application processor) and an auxiliary processor (e.g., a graphics processing unit (GPU), neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor) that may operate independently of or in conjunction with the main processor. Processor 110 may be implemented as a system-on-a-chip (SoC) or integrated circuit (IC) configured to perform processing. The auxiliary processor may be implemented separately from the main processor or may be implemented as part of the main processor. Processor 110 may include one or more processors, and the operation of the electronic device 100 described herein may be performed by one processor or by a combination of multiple processors.

[0055] The processor described herein may include processing circuitry or multiple processors. For example, the term "processor" may refer to or include various processing circuitry comprising at least one processor, wherein the at least one processor may be configured to perform the various functions described herein individually and / or collectively in a distributed manner. When a processor is described herein as performing multiple operations (or functions), this includes, but is not limited to, the following situations: a single processor performing all operations; and a processor performing some of the multiple operations while another processor performs the others.

[0056] Memory 120 may store data used by at least one component of electronic device 100 (e.g., processor 110). Data may include, for example, software input and output data, associated instructions, feature maps obtained from a diffusion generative model, and data related to the diffusion generative model (e.g., transform determination parameter values, transform parameter values). Memory 120 may store instructions executable by processor 110. Memory 120 may include one or more memories, and instructions for controlling processor 110 to perform operations of electronic device 100 as described herein may be stored in one memory or may be partitioned and stored in multiple memories. Memory 120 may include at least one of volatile memory and non-volatile memory. Memory 120 may include main memory, such as dynamic random access memory (DRAM) and cache memory. When executed by processor 110, instructions may cause electronic device 100 to perform various operations of electronic device 100 as described herein.

[0057] A noisy image 130 may be provided during the inference process. Electronic device 100 may provide the noisy image 130 to a diffusion generative model. Providing image 130 to the diffusion generative model may include inputting image 130 into the diffusion generative model. Image 130 may correspond to an input image of electronic device 100 or the diffusion generative model. Electronic device 100 may obtain a restored image 140 with reduced noise compared to image 130 (or compared to image 130) by performing a denoising process to reduce the noise in image 130 provided to the diffusion generative model. The denoising process may include inputting the output of the diffusion generative model into the diffusion generative model and obtaining the output again from the diffusion generative model (e.g., obtaining a new output from the diffusion generative model). The above process may be performed iteratively during the denoising process. The denoising process may include multiple processing steps. Each processing step may correspond to the process of obtaining the output using the diffusion generative model. As the processing steps are performed, the noise in image 130 may be gradually reduced, and image 130 may gradually become the restored image 140.

[0058] In an embodiment, electronic device 100 can perform optimized denoising process steps to accelerate the inference process using a diffusion generative model. Electronic device 100 can acquire the restored image 140 by performing some (e.g., only some) processing steps selected from all processing steps included in the denoising process. The number of times the selected processing steps are performed can correspond to a target number, such as a target number of times the processing steps are performed. The target number of times the processing steps are performed can indicate the desired number of times the processing steps are performed. The target number of times the processing steps are performed can be a preset or predetermined value, but is not limited thereto. For example, when there are a total of 100 processing steps in the denoising process using a diffusion generative model and the target number of times the processing steps are performed is 40, 40 processing steps selected from these 100 processing steps can be performed. The processing steps performed during the acquisition of the restored image 140 can be pre-selected during the learning process of the denoising process, which determines which processing steps will be performed (or should be performed) based on the target number of times the processing steps are performed among all processing steps. During inference, the processing steps selected during the learning process can be performed for the denoising process of image 130.

[0059] A subset of processing steps can be selected for execution during the denoising process based on the priority value of each processing step among all processing steps determined during the learning process of the denoising process. During the learning process, a priority value can be set (or determined) for each processing step, indicating an index of the likelihood of executing each processing step for the denoising process. When the priority value of a processing step is high, the likelihood (or probability) of executing that processing step during the acquisition of the restored image 140 can be higher, while when the priority value is low, the likelihood of executing that processing step can be lower. The electronic device 100 can select a number of processing steps corresponding to a target number (e.g., a target number of times the processing steps can be executed) in descending order of the priority values ​​set for each processing step among all processing steps.

[0060] As described above, the electronic device 100 can reuse or modify feature maps generated during inference. The electronic device 100 can obtain feature maps corresponding to the inputs of the diffusion generative model by using a diffusion generative model in a first processing step within the processing steps. The diffusion generative model can include a transformer model (e.g., a transformer) with attention blocks and feedforward blocks. The attention block and feedforward block can each output feature maps. For example, the attention block can output a first feature map, and the feedforward block can output a second feature map. The attention block can perform operations based on a self-attention mechanism. For example, the attention block can transform its input data into a query Q, a key K, and a value V, and compute importance by the dot product of the query Q and the key K. The attention block can normalize the weights using softmax and multiply the value V by the weights to generate output data. The feedforward block can transform its input using fully connected layers and add non-linearity using an activation function (e.g., a rectified linear unit (ReLU)). The transformer model can also include a residual block located between the attention block and the feedforward block. Residual blocks can add the output of a previous block (e.g., an attention block) to its input, and then provide the sum to the next block (e.g., a feedforward block) based on residual connections. Diffusion generative models can generate noise-reduced feature maps by using cross-attention techniques to input a feature map into a query Q. Diffusion generative models can compute the correlation between the query Q and the key K, which comprises a vector matrix corresponding to the input image in the form of a probability distribution. Diffusion generative models can then add the correlation between the query and the values ​​to the feature map using cross-attention techniques, applying the correlation in the form of a probability distribution and the dot product between the values, to generate a noise-reduced image corresponding to the feature map.

[0061] Electronic device 100 may store the feature maps acquired in the first processing step in memory. Electronic device 100 may store the feature maps acquired from each of the attention blocks and / or feedforward blocks of the diffusion generative model in memory 120 (e.g., the first feature map and the second feature map may be stored in memory 120). In a second processing step following the first processing step, electronic device 100 may compare the transform determination parameter value determined for (or associated with) the second processing step with a threshold. The threshold may be a defined value or a selected value. The threshold may be changed by selection. The transform determination parameter value determined for the processing step may be, for example, a constant value indicating an index of the probability that a transformation process of the feature map will be performed in the corresponding processing step. The larger the transform determination parameter value, the higher the probability (or likelihood) that a transformation process of the feature map will be performed in the corresponding processing step, and the smaller the transform determination parameter value, the lower the probability that a transformation process of the feature map will be performed in the corresponding processing step. The transform determination parameter value for each processing step may be determined during the learning process of the denoising process using the diffusion generative model. Transformation determination parameter values ​​can be defined or determined for each processing step and for each block of the output feature map in the diffusion generative model. For each processing step, transformation determination parameter values ​​can be determined for each of the attention block and feedforward block in the diffusion generative model. For example, a first transformation determination parameter value can be determined for the attention block, and a second transformation determination parameter value can be determined for the feedforward block. Electronic device 100 can determine whether to transform the feature map (e.g., the first feature map) of the attention block acquired in the first processing step based on the transformation determination parameter value determined for the attention block (e.g., the first transformation determination parameter value). Additionally, electronic device 100 can determine whether to transform the feature map (e.g., the second feature map) of the feedforward block acquired in the first processing step based on the transformation determination parameter value determined for the feedforward block (e.g., the second transformation determination parameter value).

[0062] In an embodiment, based on a transformation parameter value determined for a second processing step (e.g., associated with the second processing step) that is greater than a threshold, electronic device 100 can transform (e.g., linear or nonlinearly) the feature values ​​of a feature map stored in memory 120 based on the transformation parameter value determined for the second processing step (e.g., associated with the second processing step). The transformation parameter values ​​for each processing step can be determined during the learning process of a denoising process using a diffusion generative model. The transformation parameter values ​​can include, for example, coefficients and constant terms included in an equation that linearly transforms the feature values ​​of the feature map. The transformation formula for linearly transforming input "x" to output "y" is... In this case, the transformation parameter values ​​can be coefficient "a" and constant term "b". The coefficient "a" can be called the slope, and the constant term "b" can be called the deviation. Electronic device 100 can obtain the linearly transformed feature value "y" by inputting the feature value of the feature map as input "x" into the above linear transformation equation.

[0063] For each processing step and for each block (e.g., attention block, feedforward block) of the output feature map in the diffusion generative model, coefficients and constant terms can be defined. In the third processing step following the second processing step, the electronic device 100 can provide a feature map including the transformed feature values ​​to the diffusion generative model. Based on the transformation determination parameter value determined for the second processing step being less than or equal to a threshold, the electronic device 100 can provide the feature map stored in memory 120 to the diffusion generative model in the third processing step. Based on the transformation determination parameter value for the attention block of the diffusion generative model being greater than the threshold, and the transformation determination parameter value for the feedforward block being less than or equal to the threshold, the electronic device 100 can transform the feature map of the attention block stored in memory 120 and provide the transformed feature map as input to the attention block in the next processing step, and can also provide the feature map of the feedforward block stored in memory 120 as input to the feedforward block in the next processing step without undergoing a transformation process. In the third processing step, a process of calculating, generating, and inferring feature maps in the attention block and feedforward block of the diffusion generative model and storing the generated feature maps may be performed, or a process of reusing and / or transforming feature maps may be performed based on parameter values ​​and transformation parameter values ​​determined for the transformation in the third processing step. The first, second, and third processing steps may be included in a subset of processing steps selected based on optimization of the processing steps in the denoising process.

[0064] As described above, the electronic device 100 can reuse the feature map stored in the memory 120 as input to at least one of the attention block and feedforward block in a partial processing step of the denoising process, or it can transform the feature map stored in the memory 120 and then use the transformed feature map as input to at least one of the attention block and feedforward block. Transforming the feature map may, for example, include transforming the eigenvalues ​​of the feature map by applying a defined linear or nonlinear transformation equation to the eigenvalues. By transforming and using the feature map, the performance of inference for diffusion generative models can be accelerated with relatively less computation.

[0065] Electronic device 100 may be included in and operate in devices such as mobile devices, such as at least one of smartphones, cameras, or tablets, smart TVs (TVs), augmented reality (AR) / virtual reality (VR) devices, closed-circuit television (CCTV) devices, medical imaging devices, servers, semiconductor measurement devices, personal computers (PCs), data centers, and AI accelerators. Image processing methods performed by electronic device 100 can be applied to two-dimensional (2D) / three-dimensional (3D) graphics engines, applications, autonomous driving, AI-based image generation, and neural codecs, but embodiments are not limited thereto. The processor 110 of electronic device 100 may be implemented as at least one of a SoC (System-on-a-Chip) and intellectual property (IP) components within the SoC (e.g., at least one of a GPU, NPU, video processor, and display processor) operating in various devices in the vision or multimodal domain.

[0066] Figure 2 This is a diagram illustrating the denoising process performed using a diffusion-generative model according to an embodiment.

[0067] refer to Figure 2 The diffusion generative model may include a neural network model configured to generate a restored image through the diffusion of noise. The diffusion generative model may include, for example, a class / text encoder, a variational autoencoder, and a denoising model. The diffusion generative model may include a transformer model used as a denoising model. Through iterative operations of the transformer model, a desired restored image can be generated from an image containing noise. The variational autoencoder may perform a process of transforming the output of the denoising model to a low-dimensional space (e.g., a latent space) and then restoring the transformed output. The electronic device 100 may iteratively execute the processing steps of the denoising process by reducing noise through the diffusion generative model during the inference process.

[0068] Diffusion-generative models can be models trained based on forward and backward diffusion processes. According to x0, ..., x... t-1 x t x t+1 ... x T The process, proceeding sequentially from time point t=0 to time point t=T, can be described as a forward diffusion process that generates a noisy image 230 via an intermediate image 220 by gradually adding noise to the original image 210. According to x... T ... x t+1 x t x t-1The process of sequentially moving from time point t=T to time point t=0 can be described as a backward diffusion process that recovers the original image 210 (corresponding to the recovered image) from the intermediate image 220 by gradually removing noise from the noisy image 230. For example, a forward diffusion process can be a process of gradually adding noise values ​​following a fixed normal distribution (e.g., Gaussian distribution) to the eigenvalues ​​of a feature map, and a backward diffusion process can be a process of gradually subtracting noise values ​​generated by the learned normal distribution from the eigenvalues ​​of the feature map. A denoising model of the diffusion generative model can be trained to predict the noise applied to the original image 210.

[0069] Electronic device 100 can recover a nearly noise-free or noise-free original image 210 from a noisy image 230 by iteratively performing a backdiffusion process of a diffusion generative model a predetermined number of times. Here, each backdiffusion process may correspond to the processing steps described above. The iteratively performed backdiffusion processes may be included in the denoising process. The iteration count of the performed backdiffusion processes may be determined during the learning process of the denoising process, or it may be determined empirically.

[0070] Electronic device 100 can execute only some of the backdiffusion processes selected during the learning process of the denoising process. For example, when the backdiffusion process (e.g., according to x) T ... x t+1 x t x t-1 The total number of processing steps (in the order of x, ..., x0) is one thousand, and when it is determined during the learning process that one hundred processing steps need to be performed, the electronic device 100 can perform only one hundred of the one thousand processing steps. As the diffusion generative model used for image generation becomes larger and requires a large number of processing steps, the need to improve the speed of the inference process for image generation and reduce the computational cost may increase. The electronic device 100 can reduce the computational cost required for image restoration and accelerate the image restoration process by performing an optimized denoising process, as described above, which only performs a selected number of processing steps from all the processing steps. For example, the electronic device 100 can omit the transformer inference operation in some processing steps and instead use caching techniques to store the feature maps computed in previous processing steps in memory (e.g., ...). Figure 1In the next processing step, the electronic device 100 can determine whether to use the feature map stored in the memory 120 as input to the attention block or feedforward block of the transformer without modification, or to transform the feature map stored in the memory (e.g., a linear or nonlinear transformation) and use the transformed feature map as input to the attention block or feedforward block of the transformer. It is known that the feature maps generated from the attention block and feedforward block of the transformer model have high similarity between adjacent processing steps. Therefore, instead of performing the transformer inference operation every time in each processing step, the electronic device 100 can reduce the computational cost of the transformer model (which may account for a large part of the computation of the denoising model) by using caching techniques to store and reuse the feature maps generated in previous processing steps, thereby accelerating the image restoration process.

[0071] Figure 3 This is a diagram illustrating the configuration of an inference accelerator for accelerating the denoising process according to an embodiment.

[0072] refer to Figure 3 An inference accelerator configured to accelerate the denoising process using a diffusion-generative model 300 may include a processing step optimization module 310, a caching module 320, and a transformation module 330. The inference accelerator can reduce computational load, power consumption, and heat generation by accelerating the denoising process. The operation of each of the processing step optimization module 310, caching module 320, and transformation module 330 of the inference accelerator can be determined by… Figure 1 The processor 110 executes the process. The diffusion generative model 300 can be a pre-trained diffusion generative model and may include a transformer model.

[0073] The processing step optimization module 310 can select processing steps to be performed during the inference process using the diffusion generative model 300 to reduce image noise. The processing step optimization module 310 can execute some (e.g., only some) of the processing steps selected from all processing steps defined (e.g., included) in the denoising process. The number of times the selected processing steps are executed can correspond to a target number (e.g., a target number of times the processing steps are executed). The target number can be a predetermined value or a variable value. When the target number is a value predetermined during the learning process of the denoising process and the target number is S, the processing step optimization module 310 can determine to execute the denoising process by selecting only S processing steps determined during the learning process from all M processing steps included in the denoising process, where S is less than M. During the learning process, S specific processing steps from all M processing steps can also be determined. The processing step optimization module 310 can accelerate the denoising process while maintaining high performance by executing some processing steps optimized for the denoising process.

[0074] In this embodiment, the inference accelerator can be used in an on-device state, where it can be adapted to change the target number of times a processing step is executed based on a given instruction (e.g., an input prompt) and / or the execution environment. For example, the target number can be changed based on the input prompt, and can be set to a relatively small value when available computing resources are insufficient. The target number can be changed in real time.

[0075] In an embodiment, during the learning process of the denoising process, a priority value can be determined for each processing step (e.g., each of the M processing steps). When the target number of times a processing step is executed is selected as a variable value V (which is less than the total number of processing steps M), the processing step optimization module 310 can select a number of processing steps corresponding to the target number V in descending order of the priority values ​​set for each of the M processing steps. When there are processing steps necessary for the denoising process, the processing step optimization module 310 can select the remaining processing steps other than the necessary processing steps from all processing steps based on the priority value. When the target number is V and the number of necessary processing steps is R, the processing step optimization module 310 can select a number of processing steps equal to the number VR from all the M processing steps based on the priority value. In this case, the remaining processing steps can be selected in descending order of priority value, but the embodiment is not limited to this.

[0076] The caching module 320 can cache feature maps generated by the diffusion generative model 300 in certain processing steps selected by the processing step optimization module 310. For example, the caching module 320 can perform an inference process using the diffusion generative model 300 for every k (where k is a natural number greater than or equal to two (“2”) processing steps selected for the denoising process, and cache the feature maps generated by each of the attention blocks and feedforward blocks of the transformer model included in the diffusion generative model 300. The caching may include storing the feature maps generated by the attention blocks and feedforward blocks in memory (e.g., ...). Figure 1 The cached feature maps are stored in memory 120. These cached feature maps can be reused in the next processing step.

[0077] When reusing feature maps, transformation module 330 can determine whether to use the stored feature map as is (e.g., the unmodified feature map) or to use the stored feature map after transformation (e.g., the transformed feature map). For example, if the transformation determination parameter value determined for a processing step is greater than a threshold, transformation module 330 can transform the feature values ​​of the feature map stored in memory 120 based on the transformation parameter value determined for the processing step, and use the transformed feature values ​​as input for the diffusion generative model 300 in the next processing step. The transformation can include linear or nonlinear transformations, but the type of transformation is not limited to these. The transformation determination parameter value and transformation parameter value for each processing step can be determined during the learning process of the denoising process using the diffusion generative model 300. If the transformation determination parameter value determined for a processing step is less than or equal to a threshold, transformation module 330 can use the feature map stored in memory as is (untransformed) as input for the diffusion generative model in the next processing step.

[0078] Through the operations of the aforementioned caching module 320 and transformation module 330, the computation of feature maps performed in the diffusion generative model 300 can be reduced, thereby accelerating the denoising process.

[0079] Figure 4 This is a diagram illustrating the learning process of the denoising process according to an embodiment.

[0080] refer to Figure 4 During the learning process of denoising, one can choose to use a diffusion generative model (e.g., Figure 3 The processing steps performed during the inference process of the diffusion generative model 300 can determine the parameter values ​​required for inference (e.g., priority values, transformation parameter values, and transformation parameter values ​​for each processing step). The learning process can be performed by the electronic devices described herein (e.g., Figure 1 Electronic device 100), processing step optimization module (e.g., Figure 3 The learning process is executed in at least one of the following devices: the processing step optimization module 310 and other devices (e.g., a server). The learning process can be executed based on a learning algorithm. The learning algorithm can include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0081] During the learning process of the denoising process, the optimal processing step to be executed during inference can be selected from all the processing steps of the denoising process. For example, during the learning process, the number of times the denoising processing step should be executed and how to set the interval between processing steps can be determined. The number of selected processing steps can be less than the total number of processing steps.

[0082] The process of selecting the optimal processing step to be performed during the inference process can be the process of selecting the processing steps to be used for denoising during the inference process. When noise is generated in M ​​steps during learning using a diffusion generative model, and during inference, the goal can be to select S processing steps (corresponding to the target number of times the processing steps are performed) to perform the denoising process, a number N satisfying the relation M ≥ N > S can be selected. According to an embodiment, N can be a predetermined or selected value satisfying this relation. For example, N can be adjusted according to the quality of the restored image, depending on the learning environment or the number of processing steps. For example, when M is one thousand and S is three hundred, N can be selected as a number greater than three hundred and less than or equal to one thousand. During the learning process, the processing step adjuster 440 can be trained based on the result of comparing the inference result through N processing steps with the inference result through S processing steps. The processing step adjuster 440 can use the denoising model 420 of the diffusion generative model to learn the priority value and / or loss weight for each processing step in the denoising process. During the learning process, a total of 2N parameters, including N priority values ​​and N weights, can be learned. The processing step adjuster 440 can also be referred to as the processing step router. The denoising model 420 can be pre-trained.

[0083] The processing step adjuster 440 can output a priority value for each processing step included in all processing steps (e.g., all M processing steps). The priority value can be an index of the probability that each processing step will be executed in the denoising process. When the priority value of a processing step is high, the probability of executing that processing step during the acquisition of the restored image is higher, while when the priority value is low, the probability of executing that processing step is lower.

[0084] In the first process 400, denoising processing using the denoising model 420 can be iteratively performed on a given training image 410. The training image 410 can be an image containing noise (e.g., a noisy image). By performing denoising processing using the denoising model 420 for N processing steps of the denoising process, a first-first output x can be obtained. N Performing denoising by applying N processing steps may include: passing the noisy training image 410 through the denoising model 420 N times, and then using an autoencoder 430 to process the first-to-first output x. N Decoding is then performed. The training image 410 can be input into the denoising model 420 to obtain output x1, and output x1 can be input again into the denoising model 420 to obtain output x2. By iterating this process, the first-to-first output x2 can be obtained. N The first-to-first output x can be... NThe input is to the autoencoder 430 included in the diffusion generative model, and the first and second outputs S can be output from the autoencoder 430. N The automatic encoder 430 can output x from the first to the first output. N Decode to generate the first and second outputs S N The autoencoder 430 can be a variational autoencoder. The autoencoder 430 can be pre-trained. The first-to-first output x is determined through the first process 400. N and the first-second output S N It can be considered a reference value.

[0085] In the second process 405, when S is the target number of times the processing steps are executed (i.e., the expected number of times the processing steps are to be executed), the processing step adjuster 440 can select S steps from the N processing steps executed in the first process 400, and the processing step adjuster 440 can be trained during the learning process. When the processing step adjuster 440 selects S processing steps according to the priority value of each processing step, denoising processing by the denoising model 420 can be performed only on the selected processing steps to generate the second-first output x. SN Performing denoising by applying S (the target number of processing steps) processing steps may include: passing the noisy training image 410 through the denoising model 420 S times, and then passing the second-first output x through the autoencoder 430. SN Decoding is performed. When S processing steps are selected based on the priority of each processing step, the second-first output x can be generated by passing the noisy training image 410 through the denoising model 420 only for the selected processing steps. SN In the above process, the priority value of each processing step determined by the processing step adjuster 440 can be applied to the output of the denoising model 420 and input into the denoising model 420 in the next processing step. The second-first output x can be... SN Input is given to the auto encoder 430, and a second-to-second output S can be output from the auto encoder 430. S The automatic encoder 430 can output x from the second to the first output. SN Decode to generate the second-second output S S .

[0086] During the learning process of the denoising process, the processing step adjuster 440 can be trained to make the second-second output S S With the first-second output S NThe processing steps become identical or similar. Based on the learning of the processing step adjuster 440, the optimal processing step corresponding to the target number of times the processing step is executed can be selected, and the priority value of each processing step can be determined. The processing step adjuster 440 can assign priority values ​​p to N processing steps and assign a loss weight a to each processing step when calculating the loss. The priority value p of a processing step can be the probability or likelihood of that processing step being selected during inference. In an embodiment, a processing step with a priority value greater than or equal to a threshold or a processing step designated as necessary to be executed during inference (e.g., a necessary processing step) can be executed during the inference process. Based on the priority value of a processing step being less than the threshold, the processing step may not be executed during the inference process. During the learning process, a pass-through estimator (STE) technique can be used, which can be used to approximate the differentiation of a nonlinear function. The STE technique can include forward pass-throughs that apply discrete functions (e.g., quantized to zero ("0") or one ("1")) to generate an output, and backward pass-throughs that treat the derivative in the discrete transformation as one ("1") so that the normal gradient can be used during the backward pass-through process. A forward pass can be performed based on the priority value of the processing step being greater than or equal to a threshold, and a full backward pass can be performed during the backward pass. Here, the threshold can be the priority value of the processing step with the S-th highest priority value among N processing steps. When it is necessary to include R processing steps (where R is a natural number greater than or equal to one ("1")), the threshold can be specified as the priority value of the processing step with the SR-th highest priority value among N processing steps.

[0087] In this embodiment, the loss used to train the processing step adjuster 440 is... f It can be based on the representation of the first - first output x N With the second - first output x SN Loss of the difference x S represents the first and second outputs S. N With the second-second output S S Loss of the difference s For example, the loss used to train the processing step adjuster 440. f It can be determined by the following equation 1.

[0088] [Equation 1]

[0089]

[0090] Here, a x This can be represented as applied to loss. x The loss weights, and a s This can be represented as applied to loss. s The loss weights. When calculating the loss...x When this happens, the loss weight a can be used. x Scaling the loss magnitude adjusts how much the loss reflects each processing step, and the loss weights a can be adjusted during the learning process. x Let's learn together. We can train the processing step adjuster 440 to adjust the loss. f Minimizing the loss, and as a result of training, allows us to determine the priority for each processing step, and select a number of processing steps corresponding to the number of targets. Gradient descent can be used to find the loss that minimizes the target. f Minimize processing steps and priority values.

[0091] After the learning process is complete, the number and time interval of processing steps to be executed during inference can be determined based on the priority value of each processing step. For example, the top S processing steps (corresponding to the target number of times the processing steps are executed) can be selected according to the priority value determined for each processing step, and the denoising process can be performed based on the selected S processing steps. As another example, during the learning process, instead of the top S processing steps based on priority values, the learning process can be performed in such a way that processing steps with a weighted average (or weighted average) of less than N processing steps are not executed, and the inference process is performed using processing steps with a weight greater than the average. Because each processing step performing the denoising process may require operation using a denoising model 420 (e.g., a transformer model with large parameters), the inference speed may slow down as the number of processing steps increases. In addition, the denoising performance may vary depending on which processing step is executed among all the processing steps. According to an embodiment, denoising can be performed during the inference process using only some of the best selected processing steps instead of executing all processing steps, thereby reducing the computational load without degrading the quality of the restored image compared to executing all processing steps.

[0092] Figure 5 This is a diagram illustrating a feature map generated from a diffusion generative model according to an embodiment.

[0093] refer to Figure 5 The diffusion generative model used in the denoising process (e.g., Figure 3 The diffusion generative model (300) may include a transformer model. The transformer model can perform inference on the input and provide an output. The transformer model can determine the output of the current processing step based on the input and output of previous processing steps. The denoising process may correspond to the inference process of the diffusion generative model.

[0094] The transformer model may include an attention block 510, a residual block 520, and a feedforward block 530. The residual block 520 may be located between the attention block 510 and the feedforward block 530. The attention block 510 may correspond to multi-head self-attention, which performs self-attention operations in parallel. The residual block 520 may perform at least one of a residual connection operation that adds the input and output together and a layer normalization operation that performs normalization using the mean and variance. The feedforward block 530 may correspond to a feedforward neural network.

[0095] The cache used to store the attention block feature map 515 (e.g., the first feature map) and the feedforward block feature map 535 (e.g., the second feature map) described below can be provided by an electronic device (e.g., Figure 1 Electronic device 100) or cache module (e.g., Figure 3 The cache module 320) is executed.

[0096] Attention block feature map 515 can be generated in attention block 510 of the diffusion generative model at a specific processing step in the inference process. Attention block feature map 515 can be a feature map generated by attention block 510. Attention block feature map 515 can be stored in memory 120. Attention block feature map 515 can be passed to residual block 520. In residual block 520, at least one of residual connection operation and layer normalization operation can be performed on attention block feature map 515. The output of residual block 520 can be passed to feedforward block 530. Feedforward block 530 can perform feedforward operation to generate feedforward block feature map 535. Feedforward block feature map 535 can be a feature map generated by feedforward block 530. Feedforward block feature map 535 can be stored in memory 120. According to an embodiment, attention block feature map 515 generated in attention block 510 and feedforward block feature map 535 generated in feedforward block 530 can be stored separately in different memory regions.

[0097] Figure 6 This is a diagram illustrating the transformation processing performed on the cached feature map according to an embodiment.

[0098] refer to Figure 6 The feature maps cached in a previous processing step (which may be referred to as the "first processing step") (e.g., attention block feature map 515 and / or feedforward block feature map 535) can be used as input to the diffusion generative model in the next processing step (which may be referred to as the "second processing step") as is (e.g., without modification or transformation), or they can be transformed and used as input to the diffusion generative model. The caching processing operations described below can be performed by electronic devices (e.g., Figure 1 Electronic device 100) or conversion module or transformation module (e.g., Figure 3 The transformation module 330) is executed.

[0099] In the second processing step, the attention block feature map 515 stored in memory 120 can be loaded, and based on the transformation determination parameter values ​​determined for the attention block in the second processing step (e.g., associated with the attention block), it can be determined whether the attention block feature map 515 should be transformed (e.g., a linear or nonlinear transformation) or used as is (e.g., unmodified or untransformed). For example, if the transformation determination parameter value determined for the attention block is greater than a threshold, a transformed attention block feature map 610 can be generated by applying the feature values ​​of the attention block feature map 515 to a defined linear transformation equation. When performing a linear transformation, a linear transformation equation defined by the transformation parameter values ​​defined in the attention block in the second processing step can be applied. The transformation parameter values ​​can be coefficients and constant terms of the linear transformation equation. The transformed attention block feature map 610 can be input into the attention block in the second processing step. If the transformation determination parameter value determined for the attention block in the second processing step is less than or equal to a threshold, the attention block feature map 515 can be input into the attention block as is without transformation. Not transforming the attention block feature map 515 can include applying an identity function to the attention block feature map 515. Figure 6 In the example shown, attention block feature map 515 can be transformed, and a transformed attention block feature map 610 can be generated.

[0100] The transformed attention block feature map 610 can be passed to the residual block 520. In the residual block 520, residual join operations and / or layer normalization operations can be performed on the transformed attention block feature map 610. The output of the residual block 520 can be passed to the feedforward block. In the second processing step, the feedforward block feature map 535 stored in the memory 120 can be loaded, and based on the transformation determination parameter values ​​determined for the feedforward block in the second processing step, it can be determined whether to transform the feedforward block feature map 535 (e.g., a linear or nonlinear transformation) or to use the feedforward block feature map 535 as is (e.g., unmodified or untransformed). Based on the transformation determination parameter values ​​determined for the feedforward block being greater than a threshold, the transformed feedforward block feature map 620 can be generated by applying the feature values ​​of the feedforward block feature map 535 to a defined linear transformation equation. When performing a linear transformation, the linear transformation equation defined by the transformation parameter values ​​defined in the feedforward block in the second processing step can be applied. The transformed feedforward block feature map 620 can be input into the feedforward block in the second processing step. Based on the transformation determination parameter value determined for the feedforward block in the second processing step being less than or equal to a threshold, the feedforward block feature map 535 can be input into the feedforward block as is without transformation. Not transforming the feedforward block feature map 535 may include applying an identity function to the feedforward block feature map 535.

[0101] The transformed attention block feature map 610 and the transformed feedforward block feature map 620 can be used as inputs to the attention block and feedforward block, respectively, in the third processing step, which can be the next processing step after the second processing step (e.g., after the second processing step or following the second processing step).

[0102] Figure 7 This diagram illustrates the transformation process performed on the feature map based on the feature map generated from the diffusion generative model and the cached feature map, according to an embodiment.

[0103] Figure 6 The example shown selectively uses previously cached feature maps and transformed feature maps based on the parameter values ​​determined by the defined transformation, while the following description... Figure 7 The example shown can generate a transformed feature map (e.g., transformed attention block feature map 730 and transformed feedforward block feature map 760) by combining the feature map generated from the diffusion generative model in the current processing step with a previously cached feature map. The noise reduction performance can be further improved by generating the transformed feature map by combining the feature map generated from the diffusion generative model with a previously cached feature map. The processing operations described below can be performed by electronic devices (e.g., Figure 1 Electronic device 100) or conversion module or transformation module (e.g., Figure 3 The transformation module 330) is executed.

[0104] refer to Figure 7 In a previous processing step (which may be referred to as the "first processing step"), a first attention block feature map 710 can be generated and cached in attention block 510 through the inference process of the diffusion generative model, and a first feedforward block feature map 740 can be generated and cached in feedforward block 530. In the next processing step (which may be referred to as the "second processing step"), the first attention block feature map 710 cached in the first processing step can be loaded from memory 120. In the second processing step, a second attention block feature map 715 can be generated through operations on attention block 510. The first attention block feature map 710 and the second attention block feature map 715 can be combined using combination operation 720 to generate a transformed attention block feature map 730. The combination operation 720 may include at least one of the following operations: an average operation of calculating the average of the feature values ​​of the first attention block feature map 710 and the feature values ​​of the second attention block feature map 715; a weighted average operation of calculating the average by applying weights to the feature values; a summation operation of adding the feature values; a maximum value operation of selecting the larger feature value among the feature values; and a minimum value operation of selecting the smaller feature value among the feature values, but the embodiments are not limited thereto.

[0105] The transformed attention block feature map 730 can be passed to the residual block 520. In the residual block 520, residual join operations and / or layer normalization operations can be performed on the transformed attention block feature map 730. The output of the residual block 520 can be passed to the feedforward block 530. In the second processing step, the first feedforward block feature map 740 stored in the memory 120 can be loaded. In the second processing step, a second feedforward block feature map 745 can be generated through the operation of the feedforward block 530. The first feedforward block feature map 740 and the second feedforward block feature map 745 can be combined through a combination operation 750 to generate the transformed feedforward block feature map 760. The combination operation 750 may include at least one of the following operations: an average operation that calculates the average of the feature values ​​for the feature values ​​of the first feedforward feature map 740 and the feature values ​​of the second feedforward feature map 745; a weighted average operation that calculates the average by applying weights to the feature values; a summation operation that adds the feature values; an operation that selects the maximum value of the larger feature value among the feature values; and an operation that selects the minimum value of the smaller feature value among the feature values, but the embodiments are not limited thereto.

[0106] The transformed attention block feature map 730 and the transformed feedforward block feature map 760 can be used as inputs to attention block 510 and feedforward block 530, respectively, in the third processing step, which can be the next processing step after the second processing step (e.g., after the second processing step or following the second processing step).

[0107] Figure 8 This diagram illustrates the caching and transformation of a feature map according to processing steps, based on an embodiment.

[0108] refer to Figure 8 The diffusion-generative model may include a transformer model 800 as a denoising model. The transformer model 800 may include multiple attention blocks (e.g., attention block 802 and attention block 806) and multiple feedforward blocks (e.g., feedforward block 804 and feedforward block 808). The transformer model 800 may also include residual blocks located between the attention blocks and the feedforward blocks. The number of attention blocks 802 and 806 and the number of feedforward blocks 804 and 808 can vary. For ease of description, an example using two attention blocks 802 and 806 and two feedforward blocks 804 and 808 is described, but the embodiment is not limited thereto.

[0109] Electronic devices (e.g., Figure 1The electronic device 100 can sequentially execute processing steps to perform a denoising process on a noisy image, thereby generating a noise-reduced restored image. The processed steps performed can be selected from all processing steps included in the denoising process, and the number of selected processed steps can correspond to a target number of times the processing steps are to be executed (e.g., a target number of times the processing steps are executed). The selected processed steps may include a first processing step, a second processing step following the first processing step, and a third processing step following the second processing step.

[0110] In one embodiment, the electronic device may perform an inference process once for every k processing steps of the denoising process (where k is a natural number greater than or equal to two (“2”)) to reduce feature map computation in the diffusion generative model. Figure 8 In the example shown, k can be equal to three (“3”), but the embodiment is not limited thereto. In the first processing step, inference operations can be performed in each of the attention blocks 802 and 806 and the feedforward blocks 804 and 808 of the diffusion generative model to generate feature maps. Using the inference process in the first processing step, the electronic device can obtain an attention block feature map from attention block 802 at operation 810, a feedforward block feature map from feedforward block 804 at operation 820, an attention block feature map from attention block 806 at operation 830, and a feedforward block feature map from feedforward block 808 at operation 840. The electronic device can store (or cache) the attention block feature maps and feedforward block feature maps obtained from each of attention blocks 802 and 806 and feedforward blocks 804 and 808 in memory (e.g., Figure 1 In the memory 120).

[0111] In the second processing step (which may be a processing step following the first processing step), the electronic device may compare the transform determination parameter value determined for the second processing step (e.g., associated with the second processing step) with a threshold, and based on the comparison result determine whether to use the feature map cached in the first processing step as is or to use the feature map cached in the first processing step after transformation. The threshold may be a defined value or a selected value. The threshold may be changed. The transform determination parameter value for the second processing step may be determined during the learning process of the denoising process using the diffusion generative model. The transform determination parameter value may be defined for each processing step and for each block of the output feature map in the diffusion generative model. For example, the transformed determination parameter values ​​defined for each of the attention blocks 802 and 806 and the feedforward blocks 804 and 808 in the second processing step S2, and the transformed determination parameter values ​​defined for each of the attention blocks 802 and 806 and the feedforward blocks 804 and 808 in the third processing step S3, may be as shown in Table 1 below.

[0112] [Table 1]

[0113]

[0114] According to Table 1 above, in the second processing step S2, the transformation determination parameter value determined for attention block 802 can be 0.3, the transformation determination parameter value determined for feedforward block 804 can be 0.4, the transformation determination parameter value determined for attention block 806 can be 0.1, and the transformation determination parameter value determined for feedforward block 808 can be 0.8. In the third processing step S3, the transformation determination parameter value determined for attention block 802 can be 0.9, the transformation determination parameter value determined for feedforward block 804 can be 0.3, the transformation determination parameter value determined for attention block 806 can be 0.7, and the transformation determination parameter value determined for feedforward block 808 can be 0.2. The larger the transformation determination parameter value, the greater the probability that the feature map transformation process will be executed in the corresponding processing step; conversely, the smaller the transformation determination parameter value, the less likely the feature map transformation process will be executed in the corresponding processing step.

[0115] In this embodiment, the threshold can be set to 0.65. Therefore, in the second processing step, the transformation determination parameter value defined for feedforward block 808 can be greater than the threshold, and the transformation determination parameter values ​​defined for the remaining blocks (e.g., attention block 802, feedforward block 804, and attention block 806) can be less than or equal to the threshold. In the second processing step, at operation 812, the electronic device can use the attention block feature map generated from attention block 802 and stored in memory in the first processing step for attention block 802. For example, at operation 812, the stored attention block feature map can be provided as input to attention block 802. At operation 822, the electronic device can use the feedforward block feature map generated from feedforward block 804 and stored in memory in the first processing step for feedforward block 804. At operation 832, the electronic device can use the attention block feature map generated from attention block 806 and stored in memory in the first processing step for attention block 806. At operation 842, the electronic device can transform the feedforward block feature map generated from feedforward block 808 and stored in memory in the first processing step and use it for feedforward block 808. The transformation of the feedforward block feature map at operation 842 may, for example, include performing a linear (or nonlinear) transformation on the feedforward block feature map generated from feedforward block 808 and stored in memory in the first processing step based on transformation parameter values ​​defined for feedforward block 808 in the second processing step. The transformation parameter values ​​for each processing step can be determined during the learning process of the denoising process using a diffusion generative model. During the learning process of the denoising process, the objective function for determining the parameter values ​​and transformation parameter values ​​for each learned transformation in the attention block and feedforward block in each processing step can be set in various ways. For example, the objective function can be set or determined based on the mean square error between the output of the denoising model obtained by performing inference in all processing steps of the denoising process and the output of the denoising model obtained by applying the image processing method proposed herein. Based on the objective function, transformation parameter values ​​and transformation parameter values ​​can be determined for each attention block and feedforward block in each processing step to minimize the mean squared error. However, this is only an example, and the implementation is not limited to this. For example, in addition to mean squared error, objective functions based on L1 loss, cosine similarity, or regularization schemes can also be set.

[0116] The transformation parameter values ​​may include, for example, coefficients and constant terms included in the equation that performs a linear transformation on the eigenvalues ​​of the feature map. For example, the transformation parameter values ​​defined or determined for each of attention blocks 802 and 806 and feedforward blocks 804 and 808 in the second processing step S2, and for each of attention blocks 802 and 806 and feedforward blocks 804 and 808 in the third processing step S3, may be as shown in Table 2 below.

[0117] [Table 2]

[0118]

[0119] According to Table 2 above, for the second processing step S2, the first transformation parameter value (corresponding to the coefficients of the linear transformation equation) and the second transformation parameter value (corresponding to the constant term of the linear transformation equation) defined in attention block 802 can be -0.8 and -4.0, respectively, and the first transformation parameter value and the second transformation parameter value defined in feedforward block 804 can be 3.7 and 6.9, respectively. For the second processing step S2, the first transformation parameter value and the second transformation parameter value defined in attention block 806 can be -2.8 and 8.7, respectively, and the first transformation parameter value and the second transformation parameter value defined in feedforward block 808 can be 1.1 and 0.3, respectively. For the third processing step S3, the first transformation parameter value and the second transformation parameter value defined in attention block 802 can be 1.5 and 0.5, respectively, and the first transformation parameter value and the second transformation parameter value defined in feedforward block 804 can be 0.9 and 0.0, respectively. For the third processing step S3, the first transformation parameter value and the second transformation parameter value defined in the attention block 806 can be -2.1 and -0.8, respectively, and the first transformation parameter value and the second transformation parameter value defined in the feedforward block 808 can be 3.3 and 2.7, respectively.

[0120] According to Table 2 above, when a linear transformation is performed on the feedforward block feature map based on the first transformation parameter value 1.1 and the second transformation parameter value 0.3 defined for the feedforward block 808 in the second processing step, the electronic device can substitute each feature value of the feedforward block feature map as x into the linear transformation equation for all feature values ​​of the feedforward block feature map. The process of calculating the eigenvalues ​​y after linear transformation is performed to obtain the feature map after linear transformation.

[0121] In the third processing step (which may be a processing step following the second processing step), the electronic device can compare the transformation determination parameter value determined for the third processing step with a threshold, and determine, based on the comparison result, whether to use the cached feature map as is or to use the cached feature map after transformation. According to Table 1 above, in the third processing step, the transformation determination parameter value defined for each of attention blocks 802 and 806 can be greater than the threshold of 0.65, and the transformation determination parameter value defined for each of feedforward blocks 804 and 808 can be less than the threshold.

[0122] In the third processing step, at operation 814, the electronic device can transform the attention block feature map cached in the second processing step for attention block 802. According to Table 2 above, the first and second transformation parameter values ​​defined in attention block 802 for the third processing step S3 can be 1.5 and 0.5, respectively. The electronic device can transform the attention block feature map by substituting each feature value of the attention block feature map as x into the linear transformation equation. The process of calculating the eigenvalues ​​y after linear transformation is performed to obtain the feature map after linear transformation. At operation 824, the electronic device can use the feedforward block feature map generated from feedforward block 804 and stored in memory in the second processing step for feedforward block 804. For example, at operation 824, the stored feedforward block feature map can be provided as input to feedforward block 804. At operation 834, the electronic device can transform the attention block feature map cached in the second processing step for attention block 806. According to Table 2 above, the first transformation parameter value and the second transformation parameter value defined in attention block 806 for the third processing step S3 can be -2.1 and -0.8, respectively. The electronic device can substitute each eigenvalue of the attention block feature map as x into the linear transformation equation for all eigenvalues ​​of the attention block feature map. The process of calculating the eigenvalues ​​y after linear transformation is performed to obtain the feature map after linear transformation. At operation 844, the electronic device can use the feedforward block feature map generated from the feedforward block 808 and stored in memory in the second processing step for the feedforward block 808. For example, at operation 844, the stored feedforward block feature map can be provided as input to the feedforward block 808.

[0123] Figure 9 and Figure 10 This is a flowchart illustrating the operation of an image processing method for performing a denoising process according to an embodiment. This image processing method can be performed by the electronic device described herein (e.g., Figure 1 The electronic device 100) performs the operation.

[0124] refer to Figure 9 At operation 910, the electronic device can provide an image containing or including noise to the diffusion generative model (e.g., Figure 3 (Diffusion generative model 300). Electronic devices can input images containing noise into the diffusion generative model.

[0125] At operation 920, the electronic device can obtain a restored image with reduced noise compared to the image (e.g., compared to the image itself) by performing a denoising process to reduce noise in the image provided to the diffusion generative model. The electronic device can generate a restored image with progressively reduced noise by iteratively performing the image noise reduction process using the diffusion generative model. The denoising process may include multiple processing steps, which may be a process of obtaining output using the diffusion generative model, and these processing steps may be performed sequentially.

[0126] In an embodiment, the electronic device can acquire a restored image by executing some processing steps selected from all processing steps defined or included in the denoising process. The number of times a processing step is executed in the selected processing steps can correspond to a target number of times the processing steps are executed. The processing steps executed during the acquisition of the restored image can be pre-selected during the learning process of the denoising process, which determines which processing steps to execute based on the target number of times the processing steps are executed among all processing steps. During inference, the processing steps selected during the learning process can be executed for the denoising process of the image. Some processing steps executed in the denoising process can be selected based on the priority value of each processing step among all processing steps determined during the learning process of the denoising process. For example, the electronic device can select a number of processing steps corresponding to a target number (e.g., a target number of times the processing steps are executed) in descending order of the priority values ​​set for each processing step among all processing steps. Reference is made below. Figure 10 A more detailed example operation describes the denoising process performed through processing steps corresponding to the target number.

[0127] refer to Figure 10 At operation 1010, the electronic device may, in a first processing step within the processing steps, acquire a feature map corresponding to the input of the diffusion generative model by using a diffusion generative model. The diffusion generative model may include a transformer model. The transformer model may include an attention block and a feedforward block, wherein the attention block and the feedforward block may each output a feature map. For example, the attention block may output a first feature map, and the feedforward block may output a second feature map.

[0128] At operation 1020, the electronic device may store the feature map acquired in the first processing step in memory (e.g., Figure 1 The electronic device can store feature maps acquired from each of the attention blocks and / or feedforward blocks of the diffusion generative model in the memory (120).

[0129] At operation 1030, in the second processing step following the first processing step, the electronic device may compare the transform determination parameter value determined for the second processing step (e.g., associated with the second processing step) with a threshold. The threshold may be a defined value or an optional value. The transform determination parameter value for each processing step may be determined during the learning process of the denoising process using the diffusion generative model. The transform determination parameter value may be defined or determined for each processing step and also for each block of the output feature map in the diffusion generative model. For each processing step, the transform determination parameter value may be determined for each of the attention block and feedforward block in the diffusion generative model. The electronic device may determine whether to transform the feature map of the attention block acquired in the first processing step based on the transform determination parameter value determined for the attention block. The electronic device may determine whether to transform the feature map of the feedforward block acquired in the first processing step based on the transform determination parameter value determined for the feedforward block.

[0130] Based on the transformation parameter value determined for the second processing step (e.g., associated with the second processing step) being greater than a threshold ("Yes" at operation 1030), at operation 1040, the electronic device can transform the feature values ​​of the feature map stored in memory based on the transformation parameter values ​​determined for the second processing step. The transformation of the feature values ​​may include: inputting the feature values ​​into a defined linear or nonlinear transformation equation and obtaining the resulting value of the equation. The transformation parameter values ​​for each processing step can be determined during the learning process of the denoising process using a diffusion generative model. The transformation parameter values ​​may include, for example, coefficients and constant terms included in the equation that linearly transforms the feature values ​​of the feature map. For each processing step, transformation parameter values ​​may also be determined for each attention block and feedforward block of the output feature map in the diffusion generative model.

[0131] At operation 1050, the electronic device may, in a third processing step following the second processing step, provide a feature map including the transformed feature values ​​from operation 1040 to the diffusion generative model. The diffusion generative model can generate output by using the feature map including the transformed feature values ​​as one of the entire inputs.

[0132] Based on the transformation determination parameter value determined for the second processing step being less than or equal to a threshold ("No" at operation 1030), at operation 1060, the electronic device can provide the feature map stored in memory to the diffusion generative model in the third processing step. The electronic device can directly input the feature map stored in memory into the diffusion generative model without, for example, a linear or nonlinear transformation process.

[0133] Operations 1030, 1040, 1050, and 1060 described above can be performed individually for each attention block and feedforward block of the diffusion generative model. For example, based on the transformation of the attention block in the diffusion generative model determining that the parameter value is greater than a threshold, and the transformation of the feedforward block determining that the parameter value is less than or equal to a threshold, the electronic device can transform the feature map of the attention block stored in memory and provide the transformed feature map as input to the attention block in the next processing step. It can also provide the feature map of the feedforward block stored in memory as input to the feedforward block in the next processing step without undergoing a transformation process. Furthermore, based on the transformation of the attention block determining that the parameter value is less than or equal to a threshold, and the transformation of the feedforward block determining that the parameter value is greater than a threshold, the electronic device can provide the feature map of the attention block stored in memory as input to the attention block in the next processing step without undergoing a transformation process, and it can also provide the transformed feature map of the feedforward block stored in memory as input to the feedforward block in the next processing step.

[0134] In the third processing step, a process can be performed to compute and store feature maps in the attention block and feedforward block of the diffusion generative model (corresponding to operations 1010 and 1020), or a process can be performed to reuse and / or transform the feature maps based on the parameter values ​​and transformation parameter values ​​determined for the third processing step (corresponding to operations 1030 to 1060). The first, second, and third processing steps can be included in a subset of processing steps selected based on optimization of the processing steps in the denoising process. (See reference) Figure 10 The described series of operations can be performed iteratively until all processing steps in the selected processing steps have been performed, and can ultimately generate a noise-reduced restored image.

[0135] Figure 11 This is a block diagram illustrating the configuration of an inference accelerator system performing a denoising process according to an embodiment.

[0136] refer to Figure 11 The inference accelerator system 1100 can perform the denoising process described herein to reduce noise in images. The inference accelerator system 1100 can use a diffusion generative model (e.g., Figure 3 The diffusion generative model 300 is used to accelerate the denoising process.

[0137] The inference accelerator system 1100 may include a processor 1110, a cache memory 1115, and a memory 1120. The inference accelerator system 1100 may include other components besides those described above. For example, the inference accelerator system 1100 may also include display circuitry and / or communication circuitry for communicating with other devices.

[0138] Memory 1120 may store various information (e.g., data, values) required to perform the denoising process. Memory 1120 may store model information (e.g., model parameters, program code) used to implement the diffusion generative model. Memory 1120 may store transform determination parameter values ​​determined for each processing step and transform parameter values ​​determined for each processing step. Memory 1120 may include volatile memory or non-volatile memory. Memory 1120 may include, for example, DRAM.

[0139] Cache memory 1115 can cache and store feature maps (or eigenvalues) obtained from the diffusion generative model during the denoising process, as well as the resulting data of transformations (e.g., linear or nonlinear transformations) applied to the feature maps (or eigenvalues). Cache memory 1115 may be included within and operates within processor 1110, but is not limited to this, and cache memory 1115 may also operate outside of processor 1110.

[0140] Processor 1110 can execute programs or software to control other components connected to inference accelerator system 1100 and perform various data processing or operations. As at least part of the data processing or operations, processor 1110 can process instructions or data stored in memory 1120. Processor 1110 may include, for example, a CPU, GPU, and / or NPU. Inference accelerator system 1100 may correspond to electronic device 100 described herein, and processor 1110 can perform operations performed by electronic device 100.

[0141] Processor 1110 can perform a denoising process to reduce noise in the input image using a diffusion generative model. Processor 1110 can perform the denoising process on the input image based on data and / or values ​​stored in memory 1120 and / or cache memory 1115 to generate a restored image with reduced noise compared to the input image (e.g., compared to the input image). Processor 1110 can obtain the restored image by performing some (e.g., only some) of the processing steps selected from all processing steps defined or included in the denoising process.

[0142] In an embodiment, processor 1110 may acquire a feature map corresponding to the input of a diffusion generative model in a first processing step among the processing steps included in the denoising process by using a diffusion generative model. Processor 1110 may store the acquired feature map in cache memory 1115. The diffusion generative model may include a transformer model comprising attention blocks and feedforward blocks, and feature maps output from each of the attention blocks and feedforward blocks may be stored in cache memory 1115. In a second processing step following the first processing step, processor 1110 may compare a transform determination parameter value determined for the second processing step (e.g., associated with the second processing step) with a threshold. The transform determination parameter value determined for the second processing step may be loaded from cache memory 1115. Based on the transform determination parameter value determined for the second processing step being greater than the threshold, processor 1110 may transform the feature values ​​of the feature map stored in cache memory 1115 based on the transform parameter value determined for the second processing step. Processor 1110 may perform a linear or nonlinear transform on the feature values ​​of the feature map based on the transform parameter value. The transform parameter values ​​determined for the second processing step can be loaded from cache memory 1115. Processor 1110 can store the feature values ​​after linear or nonlinear transformation in cache memory 1115. The feature values ​​after linear or nonlinear transformation stored in cache memory 1115 can be used for subsequent processing steps. In a third processing step following the second processing step, processor 1110 can provide a feature map including the transformed feature values ​​to the diffusion generative model. Based on the transform determination parameter values ​​determined for the second processing step being less than or equal to a threshold, processor 1110 can provide the feature map stored in memory 1120 to the diffusion generative model in the third processing step. The first, second, and third processing steps can be included in a subset of processing steps selected according to the optimization of the processing steps for the denoising process. The selected processing steps (e.g., the first, second, and third processing steps) can be executed sequentially to gradually reduce the noise of the input image. When all processing steps in the selected processing steps have been executed, a noise-reduced restored image can be obtained.

[0143] The embodiments described herein can be implemented using hardware components, software components, and / or combinations thereof. The processing device can be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers and arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, field-programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other device capable of responding to and executing instructions in a defined manner. The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and generate data in response to the execution of software. For simplicity, the description of the processing device is singular; however, those skilled in the art will understand that the processing device can include multiple processing elements and various types of processing elements. For example, the processing device can include multiple processors, or a single processor and a single controller. Additionally, different processing configurations (e.g., parallel processors) are also possible.

[0144] Software may include computer programs, code segments, instructions, or combinations thereof, to independently or uniformly instruct or configure a processing device to operate as desired. Software and / or data may be permanently or temporarily embedded in any type of machine, component, physical or virtual device, or computer storage medium or device capable of providing instructions or data to or being interpreted by the processing device. Software may also be distributed across network-coupled computer systems, enabling the software to be stored and executed in a distributed manner. Software and data may be stored on one or more non-transitory computer-readable recording media.

[0145] The methods according to the above embodiments can be recorded in a non-transitory computer-readable medium including program instructions to implement the various operations of the above embodiments. The medium may also include data files, data structures, etc., alone or in combination with program instructions. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of this example, or program instructions well-known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include: magnetic media (e.g., hard disks, floppy disks, and magnetic tapes); optical media (e.g., compact disc read-only memory (CD-ROM) discs and digital versatile optical discs (DVDs)); magneto-optical media (e.g., optical-magnetic floppy disks); and hardware devices specifically configured to store and execute program instructions (e.g., read-only memory (ROM), random access memory (RAM), flash memory, etc.). Examples of program instructions include both machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0146] The aforementioned hardware device can be configured to function as one or more software modules to perform the operations of the above embodiments, and vice versa.

[0147] Although some exemplary embodiments have been described above with reference to the limited accompanying drawings, various technical modifications and variations can be applied based on them by those skilled in the art. For example, suitable results can be achieved without departing from the scope of this disclosure if the described techniques are performed in a different order and / or if components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented with other components or their equivalents.

[0148] Therefore, other implementations, embodiments, and equivalents of the claims are also within the scope of the appended claims.

Claims

1. An image processing method performed by an electronic device, comprising: The image, including noise, is provided to the diffusion generative model; as well as A noise-reduced restored image is obtained by performing a denoising process on the image using the diffusion generative model. The denoising process includes multiple processing steps, including a first processing step, a second processing step following the first processing step, and a third processing step following the second processing step. The process of obtaining the restored image includes: In the first processing step, the diffusion generative model is used to obtain a feature map corresponding to the input of the diffusion generative model. The feature map is stored in memory, and Based on the transformation associated with the second processing step, it is determined that the parameter value is greater than the threshold: In the second processing step, the feature values ​​included in the feature map stored in the memory are transformed based on the transformation parameter values ​​associated with the second processing step, and In the third processing step, a feature map including the transformed feature values ​​is provided to the diffusion generative model.

2. The image processing method according to claim 1, wherein, The restored image is obtained by performing only some of the processing steps selected from the plurality of processing steps included in the denoising process.

3. The image processing method according to claim 2, wherein, The number of times a processing step is executed in some of the processing steps corresponds to the target quantity.

4. The image processing method according to claim 1, wherein, Transforming the feature values ​​included in the feature map includes performing a linear or nonlinear transformation on the feature values ​​of the feature map based on the transformation parameter values.

5. The image processing method according to claim 1, wherein, The diffusion generative model includes a converter model. The transformer model includes an attention block and a feedforward block, and The attention block is configured to output a first feature map, and the feedforward block is configured to output a second feature map.

6. The image processing method according to claim 5, wherein, Storing the feature map in the memory includes storing the first feature map and the second feature map in the memory.

7. The image processing method according to claim 5, wherein, In each of the plurality of processing steps, a first transform determination parameter value is determined for the attention block, and a second transform determination parameter value is determined for the feedforward block. The process of obtaining the restored image includes: In the second processing step, based on the first transformation determination parameter value, it is determined whether to transform the first feature map obtained in the first processing step, and In the second processing step, the transformation parameter value is used to determine whether to transform the second feature map obtained in the first processing step.

8. The image processing method according to claim 1, wherein, Obtaining the restored image includes: Based on the transformation associated with the second processing step, if the parameter value is determined to be less than or equal to the threshold, in the third processing step, the feature map stored in the memory is provided to the diffusion generative model.

9. An electronic device, comprising: processor; as well as The memory is configured to store instructions. When the instruction is executed by the processor, it causes the electronic device to: The image, including noise, is fed into the diffusion generative model, and A noise-reduced restored image is obtained by performing a denoising process on the image using the diffusion generative model. The denoising process includes multiple processing steps, including a first processing step, a second processing step following the first processing step, and a third processing step following the second processing step. When the instruction is executed by the processor, it causes the electronic device to: In the first processing step, the diffusion generative model is used to obtain a feature map corresponding to the input of the diffusion generative model. The feature map is stored in the memory, and Based on the transformation associated with the second processing step, it is determined that the parameter value is greater than the threshold: In the second processing step, the feature values ​​included in the feature map stored in the memory are transformed based on the transformation parameter values ​​associated with the second processing step, and In the third processing step, a feature map including the transformed feature values ​​is provided to the diffusion generative model.

10. The electronic device according to claim 9, wherein, When the instructions are executed by the processor, the electronic device also causes: The restored image is obtained by performing only some of the processing steps selected from the plurality of processing steps included in the denoising process.

11. The electronic device according to claim 10, wherein, The number of times a processing step is executed in some of the processing steps corresponds to the target quantity.

12. The electronic device according to claim 9, wherein, When the instructions are executed by the processor, the electronic device also causes: The feature values ​​of the feature map are subjected to linear or nonlinear transformations based on the transformation parameter values.

13. The electronic device according to claim 9, wherein, The diffusion generative model includes a converter model. The transformer model includes an attention block and a feedforward block, and The attention block is configured to output a first feature map, and the feedforward block is configured to output a second feature map.

14. The electronic device according to claim 13, wherein, When the instructions are executed by the processor, the electronic device also causes: The first feature map and the second feature map are stored in the memory.

15. The electronic device according to claim 13, wherein, In each of the plurality of processing steps, a first transform determination parameter value is determined for the attention block, and a second transform determination parameter value is determined for the feedforward block. When the instruction is executed by the processor, it also causes the electronic device to: In the second processing step, based on the first transformation determination parameter value, it is determined whether to transform the first feature map obtained in the first processing step, and In the second processing step, the transformation parameter value is used to determine whether to transform the second feature map obtained in the first processing step.

16. The electronic device according to claim 9, wherein, When the instructions are executed by the processor, the electronic device also causes: Based on the transformation associated with the second processing step, if the parameter value is determined to be less than or equal to the threshold, in the third processing step, the feature map stored in the memory is provided to the diffusion generative model.

17. An inference accelerator system, comprising: The memory is configured to store model information used to implement the diffusion generative model; Cache memory; as well as The processor is configured to perform a denoising process that uses the diffusion generative model to reduce noise included in the input image. The denoising process includes multiple processing steps, including a first processing step, a second processing step following the first processing step, and a third processing step following the second processing step. The processor is further configured as follows: In the first processing step, the diffusion generative model is used to obtain a feature map corresponding to the input of the diffusion generative model. The feature map is stored in the cache memory, and Based on the transformation determined in the second processing step, the parameter value is determined to be greater than the threshold: In the second processing step, the feature values ​​of the feature map stored in the cache memory are transformed based on the transformation parameter values ​​associated with the second processing step, and In the third processing step, a feature map including the transformed feature values ​​is provided to the diffusion generative model.

18. The inference accelerator system according to claim 17, wherein, The diffusion generative model includes a converter model, and The converter model includes an attention block and a feedforward block.

19. The inference accelerator system according to claim 17, wherein, The processor is also configured to perform only some of the multiple processing steps included in the denoising process.

20. The inference accelerator system according to claim 17, wherein, The processor is further configured to perform a linear transformation on the feature values ​​of the feature map based on the transformation parameter values, and The eigenvalues ​​after linear transformation are stored in the cache memory.

Images