Method for generating digital zoom image using generative ai and electronic apparatus for performing same

Abstract

A method for generating a digital zoom image by using generative AI may comprise the steps of: acquiring a multi-frame input; performing demosaicing and denoising on the multi-frame input, wherein a denoising intensity is adjusted to be weaker as the number of a plurality of frames included in the multi-frame input increases; acquiring a base image by merging the plurality of frames on which the demosaicing and denoising have been performed; and acquiring a zoomed image by performing super-resolution processing on the base image.

Description

Method for generating digital zoom images using generative AI and electronic device for performing the same

[0001] The present disclosure relates to a method for generating a digital zoom image using generative AI, and more particularly, to a method for increasing the resolution and enlarging an image by performing super resolution using generative AI.

[0002] Digital zoom is simpler to implement than optical zoom, but suffers from poor image quality. To improve this, super-resolution technology is being utilized, and recently, super-resolution technology utilizing the diffusion model, a generative AI technique, has been attracting attention. While diffusion models are effective in improving image quality, the following issues can arise during the digital zoom process.

[0003] Using a diffusion model to increase image resolution can lead to hallucination, which creates details that don't actually exist. Furthermore, even if the diffusion model is lightweight for on-device processing, the computational complexity can make real-time processing difficult.

[0004] According to one aspect of the present disclosure, a method for generating a digital zoom image using generative AI may include the steps of obtaining a multi-frame input, performing demosaicing and denoising on the multi-frame input, and adjusting the denoising intensity to be weaker as the number of frames included in the multi-frame input increases, obtaining a base image by merging the multiple frames on which the demosaicing and denoising have been performed, and obtaining an enlarged image by performing super-resolution processing on the base image.

[0005] According to one aspect of the present disclosure, an electronic device includes a memory in which a program or at least one instruction is stored, and at least one processor operably coupled to the memory, wherein the at least one processor executes the program stored in the memory or the at least one instruction, thereby causing the electronic device to obtain a multi-frame input, perform demosaicing and denoising on the multi-frame input, and adjust the denoising intensity to be weaker as the number of frames included in the multi-frame input increases, and obtain a base image by merging the plurality of frames on which the demosaicing and denoising have been performed, and then perform super-resolution processing on the base image to obtain a zoomed image.

[0006] According to one aspect of the present disclosure, a computer-readable recording medium may have stored thereon a program for executing at least one of the embodiments of the disclosed method on a computer.

[0007] According to one aspect of the present disclosure, a computer program may be stored on a medium for performing at least one of the embodiments of the disclosed method on a computer.

[0008] FIG. 1 is a diagram illustrating a digital zoom module according to one embodiment of the present disclosure.

[0009] FIG. 2 is a diagram illustrating detailed configurations included in a multi-frame restoration module according to one embodiment of the present disclosure.

[0010] FIG. 3 is a diagram illustrating a process of processing an image in a demosaicing and denoising module according to one embodiment of the present disclosure.

[0011] FIG. 4 is a diagram illustrating a process of processing an image in a multi-frame restoration module according to one embodiment of the present disclosure.

[0012] FIG. 5 is a diagram illustrating detailed configurations of a super-resolution module according to one embodiment of the present disclosure.

[0013] FIG. 6 is a diagram illustrating a process of blending images in a super-resolution module according to one embodiment of the present disclosure.

[0014] FIG. 7 is a diagram illustrating components included in an electronic device according to one embodiment of the present disclosure.

[0015] FIGS. 8 to 15 are flowcharts illustrating a method of generating a digital zoom image using generative AI according to embodiments of the present disclosure.

[0016] Figure 16 is a drawing showing a basic image and an enlarged image acquired by a conventional digital zoom method.

[0017] FIG. 17 is a diagram illustrating a basic image and an enlarged image obtained by a digital zoom method according to embodiments of the present disclosure.

[0018] In describing this disclosure, descriptions of technical details that are well-known in the technical field to which this disclosure pertains and are not directly related to this disclosure will be omitted. This is to avoid obscuring the gist of this disclosure by omitting unnecessary explanations and to convey it more clearly. Furthermore, the terms described below are defined based on their functions in this disclosure and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the contents of this specification as a whole.

[0019] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.

[0020] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure of the present disclosure is complete and to fully inform those skilled in the art of the present disclosure of the scope of the disclosure. An embodiment of the present disclosure may be defined according to the claims. Like reference numerals denote like elements throughout the specification. In addition, when describing an embodiment of the present disclosure, if a detailed description of a related function or configuration is determined to unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, the terms described below are terms defined in consideration of the functions of the present disclosure and may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

[0021] In one embodiment, each block of the flowchart diagrams and combinations of the flowchart diagrams can be performed by computer program instructions. The computer program instructions can be installed on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, and the instructions, when executed by the processor of the computer or other programmable data processing apparatus, can create means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing apparatus to implement the functions in a particular manner, and the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing apparatus.

[0022] Additionally, each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0023] The term '~ unit' used in one embodiment of the present disclosure may represent software or a hardware component such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the '~ unit' may perform a specific role. Meanwhile, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. In one embodiment, the '~ unit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided through a specific component or a specific '~ unit' may be combined to reduce the number of components or separated into additional components. In addition, in one embodiment, the '~ unit' may include one or more processors.

[0024] Embodiments of the present disclosure relate to a method for generating a digital zoom image by performing super resolution processing using a generative AI model, and more particularly, to a method for generating a digital zoom image having improved image quality by increasing the resolution while preserving information or structural features included in the image.

[0025] An electronic device according to embodiments of the present disclosure can preserve detail information included in an image as much as possible by minimizing noise removal in a denoising step.

[0026] In addition, the electronic device according to embodiments of the present disclosure can preserve structural features included in a specific area of ​​an original image by blending an image whose resolution is increased using a generative model and an image whose resolution is increased without using a generative model during a process of performing super-resolution processing.

[0027] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0028] 1. Overall configuration and operation of the digital zoom module

[0029] FIG. 1 is a diagram illustrating a digital zoom module according to one embodiment of the present disclosure. Referring to FIG. 1, a digital zoom module (1) according to one embodiment of the present disclosure may include an Image Signal Processing (ISP) module (1000) and a super resolution (SR) module (400).

[0030] According to one embodiment of the present disclosure, a digital zoom module (1) may be included in an electronic device such as a smartphone, and an image captured using a camera provided in the electronic device may be enlarged through the digital zoom module (1).

[0031] The ISP module (1000) can process a raw image output from an image sensor to generate a base image, and the super-resolution module (400) can increase the resolution of the base image and enlarge its size to output it. The operations performed by the ISP module (1000) and the super-resolution module (400) are described in detail below.

[0032] A multi-frame input (10) input to a digital zoom module (1) may include a plurality of frames, and each of the plurality of frames may be a raw image output from an image sensor. For example, an image sensor of an electronic device may continuously capture a plurality of images with a single shutter operation to output raw images of a Bayer pattern, and a multi-frame input (10) including these raw images may be input to the digital zoom module (1).

[0033] According to one embodiment of the present disclosure, the ISP module (1000) may include a preprocessing module (100), a multi-frame restoration module (200), and a color processing module (300).

[0034] The preprocessing module (100) can perform various preprocessing tasks on the multi-frame input (10). For example, the preprocessing module (100) can perform preprocessing tasks such as BLS (Black Level Subtraction), LSC (Lens Shading Correction), BPC (Bad Pixel Correction), WB (White Balance), etc., but is not limited thereto and can also perform other types of preprocessing tasks as needed.

[0035] According to one embodiment of the present disclosure, the preprocessing module (100) may be omitted or may be implemented to be included in the multi-frame restoration module (200).

[0036] The multi-frame restoration module (200) can perform restoration work on multiple frames (raw images) included in the multi-frame input (10) to output a single image. Specifically, the multi-frame restoration module (200) can perform demosaicing and denoising on multiple frames included in the multi-frame input (10) and output a single image by merging the frames. According to one embodiment of the present disclosure, the single image output from the multi-frame restoration module (200) may be a linear RGB image. Of course, the present invention is not limited thereto, and other types of images may be output from the multi-frame restoration module (200).

[0037] According to one embodiment of the present disclosure, the multi-frame restoration module (200) can relatively weakly adjust the intensity of noise removal (denoising intensity) so that the basic image output from the ISP module (1000) contains as much information (e.g., information about details) as possible. In other words, the multi-frame restoration module (200) can preserve the details contained in the image by applying a minimum amount of denoising. If the basic image contains a lot of information about details, the occurrence of hallucination can be suppressed when the super-resolution module (400) increases the resolution using a diffusion model, and as a result, the quality of the enlarged image (20) can be improved.

[0038] According to one embodiment of the present disclosure, the multi-frame restoration module (200) can adjust the denoising intensity based on the number of multiple frames included in the multi-frame input (10). For example, the multi-frame restoration module (200) can adjust the denoising intensity to become weaker as the number of multiple frames included in the multi-frame input (10) increases.

[0039] The detailed configurations and specific operations included in the multi-frame restoration module (200) are described in detail below with reference to FIGS. 2 to 4.

[0040] The color processing module (300) can perform operations such as color processing and tone mapping on a single image output from the multi-frame restoration module (200) to output a basic image. For example, the color processing module (300) can receive one linear RGB image as input, and perform gamma correction, color space conversion, local tone mapping, global tone mapping, etc. to output one sRGB image (Standard RGB image).

[0041] The super-resolution module (400) can increase the resolution of the basic image output from the ISP module (1000) and enlarge its size to generate a zoomed image (20). According to one embodiment of the present disclosure, the super-resolution module (400) can increase the resolution of the basic image using a diffusion model.

[0042] According to one embodiment of the present disclosure, the super-resolution module (400) can improve processing speed through down / up scaling. Furthermore, according to one embodiment of the present disclosure, the super-resolution module (400) can improve the quality of the enlarged image (20) by performing super-resolution processing while preserving structural features for some areas of the base image.

[0043] The detailed configurations and specific operations included in the super-resolution module (400) are described in detail below with reference to FIGS. 5 and 6.

[0044] The modules (1, 100, 200, 300, 400, 1000) illustrated in Fig. 1 are configurations classified based on function or role. The modules (1, 100, 200, 300, 400, 1000) illustrated in Fig. 1 may be a software configuration implemented by a processor (730) of an electronic device (700) described later with reference to Fig. 7 executing a program or command stored in a memory (740), or may be a virtual configuration for which no matching hardware device actually exists.

[0045] In other words, the operations performed by the processor (730) of the electronic device (700) by executing a program or command stored in the memory (740) may be classified into a plurality of groups according to function or purpose, and the subjects performing the operations included in each classified group may be expressed as modules (1, 100, 200, 300, 400, 1000) of FIG. 1.

[0046] Accordingly, the operations described as being performed by the modules (1, 100, 200, 300, 400, 1000) illustrated in FIG. 1 in the present disclosure can be seen as actually being performed by the processor (730) of the electronic device (700) of FIG. 7 executing a program or command stored in the memory (740).

[0047] 2. Detailed configuration and operation of the multi-frame restoration module

[0048] FIG. 2 is a diagram illustrating detailed components included in a multi-frame restoration module (200) according to one embodiment of the present disclosure. Referring to FIG. 2, the multi-frame restoration module (200) may include a demosaicing and denoising module (210), a motion estimation and compensation module (220), and an MFP (Multi Frame Processing) module (230). The operation of each module is described in detail below.

[0049] (1) Demosaicing and denoising

[0050] The demosaicing and denoising module (210) can perform demosaicing and denoising on a plurality of frames included in a multi-frame input (10). According to one embodiment of the present disclosure, each of the plurality of frames input to the demosaicing and denoising module (210) may be a raw image of a Bayer pattern. The demosaicing and denoising module (210) can output a linear RGB image by performing demosaicing and denoising on the raw image of the Bayer pattern. Therefore, the demosaicing and denoising module (210) can output a plurality of linear RGB images.

[0051] The specific method by which the demosaicing and denoising module (210) performs demosaicing and denoising, particularly the method of adjusting the denoising intensity, is described in detail below with reference to FIG. 3.

[0052] FIG. 3 is a diagram illustrating a process of processing an image in a demosaicing and denoising module (210) according to one embodiment of the present disclosure.

[0053] Referring to Fig. 3, an image (frame) input to the demosaicing and denoising module (210) can be expressed as an input tetra (31). That is, the input tetra (31) corresponds to a frame included in the multi-frame input (10) and may be a raw image of a Bayer pattern as described above. In this case, 'tetra' may refer to a data structure that processes four pixels (2x2 blocks) of a Bayer pattern as a single unit.

[0054] The demosaicing and denoising module (210) can obtain a zero padded tetra (32) and a noise map (33) based on the input tetra (31).

[0055] The zero padding tetra (32) has a structure that expands the processing unit by adding padding with 0 around the pixel block included in the input tetra (31). The zero padding tetra (32) can solve the problem of information insufficiency that may occur at the boundary, and improve stability and accuracy during the interpolation and filtering process.

[0056] The demosaicing and denoising module (210) can generate a noise map (33) based on the input tetrahedron (31) and the zero padding tetrahedron (32), and a specific method for generating the noise map (33) is described in detail below with reference to the mathematical formula.

[0057] The demosaicing and denoising module (210) can perform demosaicing using a zero padding tetra (32) and denoising using a noise map (33). For example, the demosaicing and denoising module (210) can perform strided convolution (211, 212), convolution (213, 214, 216) and deconvolution (215) on the zero padding tetra (32) and the noise map (33) in the order shown in FIG. 3, and can output a plurality of linear RGB images (36) by performing concatenation or dot product on the intermediate results.

[0058] According to one embodiment of the present disclosure, various types of filters (34) may be used in the demosaicing and denoising processes. For example, the filter (34) may be used to interpolate surrounding pixel data during the demosaicing process or to emphasize color information in a specific direction to enhance restoration accuracy. Furthermore, for example, the filter (34) may be used to suppress a specific type of noise during the denoising process or to reduce noise while preserving details.

[0059] According to one embodiment of the present disclosure, various types of masks (35) may be used in the demosaicing and denoising processes. For example, the mask (35) may be used to select or emphasize specific pixel locations during the demosaicing process. Furthermore, for example, the mask (35) may be used to adjust the noise removal intensity for specific areas during the denoising process, or to protect specific areas.

[0060] The demosaicing and denoising module (210) can preserve details contained in an image by gently adjusting the denoising intensity. If the details contained in the image are preserved to a large extent, the occurrence of hallucination during super-resolution processing can be suppressed, thereby improving the quality of the output image.

[0061] According to one embodiment of the present disclosure, the demosaicing and denoising module (210) can preserve as much detail as possible by minimizing the amount of noise removed. In this case, minimizing the amount of noise removed may mean removing only the minimum amount of noise required to maintain the quality of the output image at a certain level or higher. The minimum amount of noise required to be removed to maintain the quality of the output image at a certain level or higher may be determined based on various conditions or needs.

[0062] According to one embodiment of the present disclosure, the demosaicing and denoising module (210) can adjust the denoising intensity based on the number of multiple frames included in the multi-frame input (10). For example, the demosaicing and denoising module (210) can adjust the denoising intensity to be weaker as the number of multiple frames included in the multi-frame input (10) increases. As the number of frames included in the multi-frame input (10) increases, the information of the image to be referenced in the super-resolution processing process increases, so that even if relatively less noise is removed, the quality of the output image (the image on which super-resolution processing has been performed) can be guaranteed.

[0063] For example, the demosaicing and denoising module (210) can remove noise by generating a noise map (33) according to the mathematical expression 1 below and performing noise modeling using the generated noise map (33). In other words, the demosaicing and denoising module (210) can perform noise modeling by adding random noise that follows a normal distribution according to σ calculated through the mathematical expression 1 to each pixel, and remove noise based on the result.

[0064]

[0065] σ: standard deviation of noise

[0066] λ shot : shot noise

[0067] λ read : read noise

[0068] I: pixel value

[0069] N: Number of frames included in the multi-frame input

[0070] Referring to the above mathematical expression 1, the demosaicing and denoising module (210) can generate a noise map (33) such that the amount of noise decreases as the number (N) of frames included in the multi-frame input (10) increases. Accordingly, the denoising intensity can be adjusted to be weaker as the number of frames included in the multi-frame input (10) increases.

[0071] The number of frames (N) included in the multi-frame input (10) may vary depending on various factors. For example, the number of frames (N) may be determined by the hardware specifications or settings of the camera that captures the image. Or, for example, even if the hardware specifications or settings of the camera are the same, the number of frames (N) may vary depending on selection. The camera captures multiple images in succession within a short period of time, and inputs the multi-frame input (10) including the captured multiple images (frames) to the digital zoom module (1). At this time, the multi-frame input (10) may be generated so as to exclude some images with low quality among the captured multiple images and include only the remaining images.

[0072] As described above, since the demosaicing and denoising module (210) weakly adjusts the denoising intensity, a plurality of linear RGB images (36) output after demosaicing and denoising can contain a lot of detail information. As a result, the detail information is well preserved in the basic image output from the ISP module (1000), and the effect of suppressing the occurrence of hallucination during super-resolution processing can be expected.

[0073] (2) Motion estimation and motion compensation

[0074] The motion estimation and compensation module (220) can align a plurality of linear RGB images (36) output from the demosaicing and denoising module (210) by performing motion estimation and motion compensation on the plurality of linear RGB images (36).

[0075] When a still image is captured in multiple frames, the pixel positions between frames may be slightly misaligned depending on the shooting conditions (e.g., camera shake, micro-movement, etc.) even for the same scene. These micro-differences can be corrected through motion estimation and motion compensation. For example, the motion estimation and compensation module (220) can analyze the relative movement (pixel movement) between frames (linear RGB images) to generate a motion vector (motion estimation), and align and correct the frames (motion compensation) using the generated motion vector.

[0076] According to one embodiment of the present disclosure, in order to determine weights for weighted summation of frames as described in the section '(3) Multi-frame processing' below, one of the multiple frames included in the multi-frame input (10) may be selected as a reference frame. In this case, the motion estimation and compensation module (220) may perform motion estimation and motion compensation and align the frames through comparison between the reference frame and the remaining frames.

[0077] (3) Multi Frame Processing (MFP)

[0078] Returning to FIG. 2, the MFP module (230) can merge multiple frames (linear RGB images) aligned through motion estimation and motion compensation after demosaicing and denoising are performed and output as a single frame.

[0079] According to one embodiment of the present disclosure, the MFP module (230) can merge frames by performing a weighted sum using weight values ​​determined through comparison between frames. A detailed description of the specific method by which the MFP module (230) determines the weight values ​​corresponding to each frame will be described in detail below with reference to FIG. 4.

[0080] FIG. 4 is a diagram illustrating a process of processing an image in a multi-frame restoration module (200) according to one embodiment of the present disclosure.

[0081] Referring to FIG. 4, as described above, raw images (41) of a Bayer pattern are converted into linear RGB images (42) by performing demosaicing and denoising while passing through a demosaicing and denoising module (210), and the linear RGB images (42) can be aligned by a motion estimation and compensation module (220).

[0082] Referring to FIG. 4, the electronic device can select any one of a plurality of frames (raw images of Bayer patterns) (41) input to the demosaicing and denoising module (210) as a reference frame. The reference frame can be used later when aligning frames and determining weight values.

[0083] According to one embodiment of the present disclosure, the electronic device can select the sharpest frame among the frames (raw images of Bayer patterns) (41) as a reference frame. For example, the electronic device can extract the boundary component of each frame using a Laplacian filter, calculate a sharpness score of each frame, and select the frame with the highest score as the reference frame. However, the electronic device is not limited thereto, and the electronic device can select the reference frame according to various criteria. For example, the electronic device can select the frame with the best quality as the reference frame, or can randomly select any one of a plurality of frames as the reference frame.

[0084] In Fig. 4, it is assumed that frame F1 is selected as the reference frame among the frames assigned reference numbers F0 to F4.

[0085] When the F1 frame is selected as the reference frame, the motion estimation and compensation module (220) can align the frames through comparison between the F1 frame and the remaining frames (F0, F2, F3, F4).

[0086] The MFP module (230) can calculate weight values ​​to be applied to each of the remaining frames (F0, F2, F3, F4) based on the result of comparing the F1 frame with the remaining frames (F0, F2, F3, F4). According to one embodiment of the present disclosure, the MFP module (230) can calculate weight values ​​according to the following mathematical expression 2.

[0087]

[0088] α and β are hyperparameters that assign high weights to frames that help reduce ghosting artifacts. d represents the difference in pixel values ​​between the reference frame and each frame. σ represents the standard deviation of pixel values ​​in the reference frame.

[0089] Referring to mathematical expression 2, if the size of the difference in pixel values ​​between a specific frame and a reference frame is less than β·σ, the frame is reflected in the weighted sum, and conversely, if the size of the difference in pixel values ​​between a specific frame and a reference frame is greater than β·σ, the frame is considered a ghost and is ignored.

[0090] By adjusting α and β, we can perform a weighted sum to maximize the required signal and discard unnecessary signals (e.g., ghosts). In other words, by adjusting α and β, we can choose whether to preserve detail information as much as possible even if it means adding a bit more noise, or conversely, to reduce noise and sacrifice some detail information.

[0091] In this way, the MFP module (230) determines weight values ​​(w0, w2, w3, w4) to be applied to each of the remaining frames (F0, F2, F3, F4) based on the result of comparing the reference frame (F1) with the remaining frames (F0, F2, F3, F4), and performs weighted summation on the frames (F0 to F4) using the determined weight values ​​(w0, w2, w3, w4), thereby generating a single linear RGB image (43).

[0092] Meanwhile, in FIG. 4, the reference frame is selected before the plurality of frames (41) are input to the demosaicing and denoising module (210), but this is not limited thereto, and the point in time at which the reference frame is selected may be changed. For example, the electronic device may select the reference frame immediately before performing weighted summation on the frames in the MFP module (230). In this case, the motion estimation and compensation module (220) can align the frames through comparison between the frames without the reference frame.

[0093] 3. Detailed configuration and operation of the super-resolution module

[0094] As previously described, when the color processing module (300) performs operations such as color processing and tone mapping on a single image output from the multi-frame restoration module (200) to output a basic image, the super-resolution module (400) can increase the resolution of the basic image and enlarge its size to output an enlarged image (20). The detailed configuration and operation of the super-resolution module will be described below with reference to FIGS. 5 and 6.

[0095] FIG. 5 is a diagram illustrating detailed configurations of a super-resolution module according to an embodiment of the present disclosure. Referring to FIG. 5, the super-resolution module (400) may include a Generative Super Resolution (GSR) module (460) and a structure preserved module (470). The super-resolution module (400) may receive a basic image (51) of 256x256 size as input, perform super-resolution processing, and generate and output an enlarged image (52) of 1024x1024 size.

[0096] According to one embodiment of the present disclosure, the super-resolution module (400) may be implemented based on a generative model. For example, the super-resolution module (400) may include a diffusion model (450), as illustrated in FIG. 5 .

[0097] Referring to FIG. 5, the diffusion model (450) can be implemented in the form of a latent diffusion model including an encoder (410), a latent space network (420), and a decoder (430). Of course, the present invention is not limited thereto, and the diffusion model (450) can also be implemented as another type of image-generating AI model.

[0098] The diffusion model (450) increases the resolution by performing a diffusion process on the base image (51) to generate an enlarged image (52). Therefore, if the base image (51) contains little information about details, it is difficult to improve the image quality through the diffusion model (450), and there is a risk of hallucination occurring. Therefore, the digital zoom module (1) according to one embodiment of the present disclosure can ensure that the base image (51) contains a lot of information about details by minimizing noise removal during the denoising process as described above.

[0099] According to one embodiment of the present disclosure, the diffusion model (450) can be lightweight and mounted on an electronic device to enable execution on-device. For example, a diffusion model trained for image generation can be distilled to reduce the number of parameters, thereby reducing the weight of the model and then mounting it on an electronic device.

[0100] A super-resolution module (400) according to one embodiment of the present disclosure can improve processing speed through down / up scaling, and can improve the quality of an enlarged image (52) by performing super-resolution processing while preserving structural features for some areas of a base image (51). This will be described in detail below.

[0101] (1) Down / Upscaling

[0102] As illustrated in FIG. 5, according to one embodiment of the present disclosure, a downscaler (441) and an upscaler (442) may be added to a diffusion model (450) to configure a GSR module (460).

[0103] The base image (51) is downscaled by a downscaler (441) and input to a diffusion model (450), and the image output from the diffusion model (450) can be upscaled by an upscaler (442). The reason for performing down / up-scaling before and after super-resolution processing is as follows.

[0104] 1) Improved processing speed

[0105] By downscaling the base image (51) and then processing it with the diffusion model (450), the processing speed can be improved. As described above, according to one embodiment of the present disclosure, the diffusion model (450) is lightweight and mounted on an electronic device for execution on-device. Additional weight reduction is possible by adding a down / upscaler (441, 442).

[0106] 2) Improved image quality of enlarged images (improved uniformity)

[0107] Down / up scaling can also improve the image quality of the enlarged image (52). The detailed reasons are as follows.

[0108] According to one embodiment of the present disclosure, the super-resolution module (400) can divide a basic image (51) into a plurality of tiles through tiling, and then perform super-resolution processing on each tile. This is to enable fast processing even on mobile devices (e.g., smartphones) with limited computational performance or resources.

[0109] Downscaling the base image (51) increases the receptive field of each tile, allowing a wider range of contextual information to be captured. If the receptive field of each tile is small, the amount of contextual information that can be captured may be insufficient, resulting in uneven uniformity among the tiles. However, according to one embodiment of the present disclosure, the amount of contextual information that can be captured for each tile increases, thereby improving the uniformity of the entire enlarged image (52).

[0110] In the embodiment illustrated in FIG. 5, the downscaler (441) and the upscaler (442) are respectively arranged at the input and output ends of the diffusion model (450). Alternatively, the downscaler (441) and the upscaler (442) may be implemented to be included in the encoder (410) and the decoder (430) of the diffusion model (450), respectively. For example, the diffusion model (450) may be trained such that some of the layers included in the encoder (410) perform downscaling, and similarly, some of the layers included in the decoder (430) perform upscaling.

[0111] As discussed above, although there are advantages resulting from down / up scaling, the following disadvantages may also arise due to down / up scaling, and to address these disadvantages, a super-resolution module (400) according to an embodiment of the present disclosure may include a structure preservation module (470).

[0112] 3) Disadvantages of down / up scaling

[0113] The size of the latent coefficient coming from the encoder (410) is the same, but since the image information included in one tile is relatively large, loss of fine detail may occur.

[0114] Additionally, downscaling may result in loss of structural information in the base image (51).

[0115] Due to the above problems, structural features may be expressed in a distorted manner in the enlarged image (52). However, when important elements such as text or a person's face are distorted, the user may feel more sensitive to the distortion than when the background or other objects are distorted. Therefore, according to one embodiment of the present disclosure, the super-resolution module (400) can perform super-resolution processing while preserving structural features for an area in the image where a specific type of object (e.g., text, a person's face, etc.) exists through the structure preservation module (470).

[0116] (2) Method for preserving the structural features of an image (structure preservation module)

[0117] Performing super-resolution using a generative AI-based image generation model, such as the diffusion model (450), has the advantage of expressing sophisticated and realistic details by adding new details that do not exist in the original. In the GSR module (460), since the resolution is increased using the diffusion model (450), details that do not exist in the base image (51) may be generated, and thus structural features included in the base image (51) may be distorted or lost. In addition, as explained above, downscaling may exacerbate this problem. In this case, the structural features of the image may refer to elements that define the shape, boundary, and form within the image.

[0118] According to one embodiment of the present disclosure, the above problem can be solved by blending an image output from a structure preservation module (470) and an image output from a GSR module (460) to generate an enlarged image (52).

[0119] Referring to Fig. 5, the image output from the GSR module (460) and the image output from the structure preservation module (470) are merged through weighted summation in the merge module (480), and the weighted summation can be performed by applying different weights to each region of the image. Details are as follows.

[0120] Just as the GSR module (460) performs the task of increasing the resolution of an image, the structure preservation module (470) can also perform the task of increasing the resolution of an image. That is, when a basic image (51) of size 256x256 is input to the structure preservation module (470) in FIG. 5, the structure preservation module (470) can increase the resolution and output an image of size 1024x1024.

[0121] However, the structure preservation module (470) differs from the GSR module (460) in that it outputs an image with an increased resolution of the base image (51) without creating or adding new details. Therefore, the structure preservation module (470) may be configured to increase the resolution while preserving the structural characteristics of the base image (51). For example, the structure preservation module (470) may be implemented as a simple resolution upscaler.

[0122] According to one embodiment of the present disclosure, the super-resolution module (400) gives a high weight to the output image of the structure preservation module (470) for an area where structural features are to be preserved, and gives a high weight to the output image of the GSR module (460) for other areas, thereby allowing realistic details to be expressed overall while maintaining the structural features of the base image (51) as much as possible for a specific area.

[0123] A specific embodiment of blending the output image of the GSR module (460) and the output image of the structure preservation module (470) is described with reference to FIG. 6. FIG. 6 is a diagram illustrating a process of blending images in a super-resolution module (400) according to one embodiment of the present disclosure.

[0124] According to one embodiment of the present disclosure, an electronic device can detect a preset type of object (e.g., text, a human face, etc.) from a base image (61) and determine a region where the preset type of object is detected as a preserved region. In the embodiment illustrated in FIG. 6, the electronic device can determine a region containing text (“sample text”) from the base image (61) as a preserved region (PR). Of course, the electronic device can also detect various types of objects other than text as needed and determine the detected region as a preserved region.

[0125] The GSR module (460) can increase the resolution of the basic image (61) and enlarge it to generate a first output image (62). In addition, the structure preservation module (470) can also increase the resolution of the basic image (61) and enlarge it to output a second output image (63).

[0126] Since the structural preservation module (470) simply increases the resolution without generating new details, the second output image (63) may have structural features better preserved than the first output image (62) generated based on generative AI. Accordingly, the text included in the second output image (63) may be less distorted than the text included in the first output image (62).

[0127] The electronic device can generate a first blend map (610) and a second blend map (620) based on the preservation region (PR). The first blend map (610) can store weight values ​​to be applied to pixels of the first output image (62), and the second blend map (620) can store weight values ​​to be applied to pixels of the second output image (63).

[0128] The weight values ​​stored in the first blend map (610) and the second blend map (620) can be determined according to at least one of the three rules below.

[0129] 1) The weight values ​​corresponding to the pixels included in the preservation area (PR) within the same blend map (610, 620) may be determined so as to be different from the weight values ​​corresponding to the pixels included in the remaining areas other than the preservation area (PR). For example, in the first blend map (610), the weight values ​​corresponding to the pixels included in the preservation area (PR) may be smaller than the weight values ​​corresponding to the pixels included in the remaining areas. Also, for example, in the second blend map (620), the weight values ​​corresponding to the pixels included in the preservation area (PR) may be larger than the weight values ​​corresponding to the pixels included in the remaining areas.

[0130] 2) For the preservation area (PR), weight values ​​may be determined so that more information of the second output image (63) is reflected than that of the first output image (62). For example, for pixels included in the preservation area (PR), the weight values ​​of the second blend map (620) may be greater than the weight values ​​of the first blend map (610).

[0131] 3) The weight values ​​can be determined so that the weight value of the first blend map (610) and the weight value of the second blend map (620) for each pixel add up to 1. For example, the blend maps (610, 620) can be determined so that the result (1-a) obtained by subtracting the weight value (a) of the first blend map (610) for each pixel from 1 becomes the weight value of the second blend map (620).

[0132] When blend maps (610, 620) are determined according to the above rules, the result of applying the first blend map (610) to the first output image (62) and the result of applying the second blend map (620) to the second output image (63) are merged in the merge module (480), thereby generating an enlarged image (64).

[0133] As described above, the super-resolution module (400) according to one embodiment of the present disclosure can improve the quality of an enlarged image by performing super-resolution processing while preserving structural features for some areas of a basic image.

[0134] 4. Quality comparison with enlarged images obtained using conventional digital zoom methods.

[0135] Hereinafter, with reference to FIGS. 16 and 17, the quality of an enlarged image acquired by a conventional digital zoom method and an enlarged image acquired by a digital zoom method according to embodiments of the present disclosure are compared.

[0136] FIG. 16 is a diagram illustrating a basic image (1610) and an enlarged image (1620) acquired by a conventional digital zoom method. FIG. 17 is a diagram illustrating a basic image (1710) and an enlarged image (1720) acquired by a digital zoom method according to embodiments of the present disclosure.

[0137] Referring to Fig. 16, a basic image (1610) can be obtained by performing processes such as demosaicing and denoising on multi-frame input according to a conventional digital zoom method. Subsequently, an enlarged image (1620) can be obtained by performing super-resolution using a generative model on the basic image (1610).

[0138] Referring to FIG. 17, a basic image (1710) may be obtained as a result of performing processes such as demosaicing and denoising on a multi-frame input according to a digital zoom method according to embodiments of the present disclosure. For example, the basic image (1710) may be output from the ISP module (1000) of FIG. 1. Subsequently, an enlarged image (1720) may be obtained by performing super-resolution using a generative model on the basic image (1710). For example, the enlarged image (1720) may be output from the super-resolution module (400) of FIG. 1.

[0139] As described above, according to embodiments of the present disclosure, the multi-frame restoration module (200) can adjust the denoising intensity to a lesser degree compared to conventional digital zoom methods. As a result, the base image (1710) of FIG. 17 may contain more noise than the base image (1610) of FIG. 16, but may also contain more information (e.g., detail information).

[0140] Since the basic image (1610) of FIG. 16 contains less noise than the basic image (1710) of FIG. 17, the basic image (1610) of FIG. 16 may have better visibility than the basic image (1710) of FIG. 17.

[0141] However, if super-resolution is performed on each of the two images (1610, 1710) using a generative model, the quality can be reversed. The base image (1710) of Fig. 17 contains more information than the base image (1610) of Fig. 16, and when super-resolution is performed using a generative model, the information contained in the base image (1710) of Fig. 17 can be used to generate image details. As a result, the details can be well expressed in the enlarged image (1720) generated from the base image (1710) of Fig. 17.

[0142] On the other hand, in the case of the basic image (1610) of Fig. 16, much information about the details of the image is lost in the process of removing noise, so in the process of performing super-resolution using a generative model, there is not enough information to use, so details may not be expressed well or distorted details (hallucination) may be generated.

[0143] Comparing the two drawings, it can be seen that the enlarged image (1720) of Fig. 17 has more realistically expressed details than the enlarged image (1620) of Fig. 16.

[0144] 5. Overall configuration and operation of the electronic device

[0145] Below, an electronic device for performing the operations described above will be described. An electronic device according to one embodiment of the present disclosure may be a device with a photographing function and a computational processing function, such as a smartphone or digital camera. Furthermore, even if it does not have a photographing function, it may be a variety of devices (e.g., a laptop or a cloud server) capable of receiving image or video files and performing image signal processing. The configuration of another electronic device according to one embodiment of the present disclosure will be described in detail below with reference to FIG. 7.

[0146] FIG. 7 is a diagram illustrating components included in an electronic device according to one embodiment of the present disclosure. Referring to FIG. 7, an electronic device (700) according to one embodiment of the present disclosure may include a photographing module (710), an input / output interface (720), a processor (730), and a memory (740).

[0147] The photographing module (710) is a component for photographing an image and may include a lens module, an image sensor (e.g., a CCD sensor, a CMOS sensor, etc.), etc. According to one embodiment of the present disclosure, the image sensor may output a raw image of a Bayer pattern having only one color channel per pixel. However, the present disclosure is not limited thereto and the image sensor may output various types of images.

[0148] The input / output interface (720) may include an input interface (e.g., a touch screen, a keyboard, a microphone, etc.) for receiving commands or information from a user, and an output interface (e.g., a display panel, a speaker, etc.) for displaying the result of an operation according to a user's command or the status of the electronic device (700). According to one embodiment of the present disclosure, the electronic device (700) may receive an input (e.g., a digital zoom request) from a user through the input / output interface (720), and when the operation is completed, may output the result of performing the operation (e.g., an enlarged image) through the input / output interface (720).

[0149] The processor (730) controls a series of processes to operate the electronic device (700) according to the embodiments described in the present disclosure, and may be composed of one or more processors. The one or more processors included in the processor (730) may be circuitry such as a System on Chip (SoC), an Integrated Circuit (IC), etc. The one or more processors included in the processor (730) may be a general-purpose processor such as a Central Processing Unit (CPU), a Micro Processor Unit (MPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), an artificial intelligence-only processor such as a Neural Processing Unit (NPU), or a communication-only processor such as a Communication Processor (CP). When the one or more processors included in the processor (730) are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0150] The processor (730) can write data to the memory (740) or read data stored in the memory (740), and in particular, process data according to predefined operation rules or artificial intelligence models by executing a program or at least one instruction stored in the memory (740). Accordingly, the processor (730) can perform the operations described in the embodiments of the present disclosure, and operations described as being performed by the electronic device (700) or modules included in the electronic device (700) in the present disclosure can be regarded as being performed by the processor (730) unless otherwise specifically described.

[0151] The memory (740) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (740) may not exist separately and may be configured to be included in the processor (730). The memory (740) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. A program or at least one instruction for performing operations according to embodiments described below may be stored in the memory (740). The memory (740) may also provide stored data to the processor (730) upon request of the processor (730).

[0152] The embodiments described above with reference to FIGS. 1 to 6 can be performed by an electronic device (700).

[0153] 6. Describe the process by referring to the flowcharts.

[0154] Hereinafter, with reference to the flowcharts of FIGS. 8 to 15, a method for generating a digital zoom image by an electronic device according to embodiments of the present disclosure will be described. The steps included in the flowcharts of FIGS. 8 to 15 can be performed by the electronic device (700) of FIG. 7, and therefore, the contents previously described with reference to FIGS. 1 to 7 may be equally applied to FIGS. 8 to 15, even if omitted below.

[0155] Referring to FIG. 8, at step 801, the electronic device can obtain a multi-frame input. The multi-frame input may include multiple frames, and each of the multiple frames may be a raw image output from an image sensor.

[0156] In step 802, the electronic device performs demosaicing and denoising on the multi-frame input. The denoising intensity can be adjusted to be weaker as the number of frames included in the multi-frame input increases. As previously described, by weakening the noise removal intensity, the electronic device can preserve the details contained in the image (frame).

[0157] In step 803, the electronic device can acquire a base image by merging multiple frames on which demosaicing and denoising have been performed. According to one embodiment of the present disclosure, the electronic device can merge the frames through weighted summation to ensure that the base image contains as much detail information as possible. The detailed steps included in step 803 are illustrated in FIG. 9.

[0158] Referring to FIG. 9, in step 901, the electronic device can determine a weight value corresponding to each of the multiple frames based on the results of comparing the multiple frames included in the multi-frame input. Detailed steps included in step 901 are illustrated in FIG. 10.

[0159] Referring to FIG. 10, at step 1001, the electronic device may determine one of the multiple frames included in the multi-frame input as a reference frame. According to one embodiment of the present disclosure, the electronic device may select the sharpest frame among the multiple frames included in the multi-frame input as the reference frame.

[0160] At step 1002, the electronic device can determine a weighting value based on the difference in pixel values ​​between the reference frame and the remaining frames and the standard deviation of the pixel values ​​of the reference frame.

[0161] Returning to FIG. 9 again, at step 902, the electronic device can generate a base image by performing a weighted sum on multiple frames using the determined weight values.

[0162] Returning to Figure 8, the electronic device can obtain an enlarged image by performing super-resolution processing on the base image in step 804. According to one embodiment of the present disclosure, the electronic device can perform super-resolution processing on the base image using a diffusion model. Detailed steps included in step 804 are illustrated in Figure 11.

[0163] Referring to FIG. 11, at step 1101, the electronic device can obtain a first output image by inputting a basic image into a generative super-resolution module based on a diffusion model.

[0164] Next, at step 1102, the electronic device can obtain a second output image by inputting the base image into a structure preservation module for increasing the resolution while preserving the structural features.

[0165] Finally, in step 1103, the electronic device can generate an enlarged image by performing a weighted sum on the first output image and the second output image. The detailed steps included in step 1103 are illustrated in FIG. 12.

[0166] Referring to FIG. 12, in step 1201, the electronic device can determine weighting values ​​to be applied to pixels of the first output image and pixels of the second output image based on the base image. Detailed steps included in step 1201 are illustrated in FIGS. 13 and 14.

[0167] Referring to FIG. 13, at step 1301, the electronic device can determine at least a portion of the base image as a preservation area.

[0168] At step 1302, the electronic device can determine weight values ​​for the preservation area such that the weight values ​​applied to the pixels of the first output image are less than the weight values ​​applied to the pixels of the second output image.

[0169] Referring to FIG. 14, in step 1401, the electronic device can determine at least a portion of the basic image as a preservation area. Detailed steps included in step 1401 are illustrated in FIG. 15.

[0170] Referring to FIG. 15, at step 1501, the electronic device can detect a preset type of object (e.g., text, human face, etc.) in the base image.

[0171] At step 1502, the electronic device can determine an area in which a preset type of object is detected as a preservation area.

[0172] Returning to FIG. 14, at step 1402, the electronic device may obtain a first blend map including weight values ​​applied to pixels of the first output image based on the preservation area. According to one embodiment of the present disclosure, the weight value corresponding to the preservation area in the first blend map may be smaller than the weight values ​​corresponding to the remaining areas excluding the preservation area.

[0173] At step 1403, the electronic device may obtain a second blend map including weight values ​​applied to pixels of the second output image based on the preservation area. According to one embodiment of the present disclosure, the weight value corresponding to the preservation area in the second blend map may be greater than the weight values ​​corresponding to the remaining areas excluding the preservation area.

[0174] Returning to FIG. 12 again, at step 1202, the electronic device can generate an enlarged image by performing a weighted sum on the first output image and the second output image based on the determined weight values.

[0175] A method for generating a digital zoom image using generative AI according to one embodiment of the present disclosure may include the steps of obtaining a multi-frame input, performing demosaicing and denoising on the multi-frame input, and adjusting the denoising intensity to be weaker as the number of frames included in the multi-frame input increases, obtaining a base image by merging the multiple frames on which the demosaicing and denoising have been performed, and obtaining a zoomed image by performing super resolution processing on the base image.

[0176] According to one embodiment, the step of obtaining the basic image may include the step of determining a weight value corresponding to each of the plurality of frames based on a result of comparing the plurality of frames included in the multi-frame input, and the step of generating the basic image by performing a weighted sum on the plurality of frames using the determined weight value.

[0177] According to one embodiment, the step of determining the weight value may include the step of determining one of the plurality of frames included in the multi-frame input as a reference frame, and the step of determining the weight value based on a difference in pixel values ​​between the reference frame and the remaining frames and a standard deviation of pixel values ​​of the reference frame.

[0178] According to one embodiment, the step of obtaining the enlarged image may generate the enlarged image by increasing the resolution and enlarging the size of the basic image while preserving structural features for at least some areas of the basic image.

[0179] According to one embodiment, the step of obtaining the enlarged image may include the step of obtaining a first output image by inputting the basic image into a generative super resolution module based on a diffusion model, the step of obtaining a second output image by inputting the basic image into a structure preserved module for increasing the resolution while preserving structural features, and the step of generating the enlarged image by performing a weighted sum on the first output image and the second output image.

[0180] According to one embodiment, the step of obtaining the first output image may include performing downscaling on the base image, inputting the base image to the super-resolution module, and performing upscaling on the output of the super-resolution module to generate the first output image.

[0181] According to one embodiment, the step of generating the enlarged image by performing the weighted sum may include the step of determining, based on the base image, weight values ​​to be applied to pixels of the first output image and pixels of the second output image, and the step of generating the enlarged image by performing weighted sum on the first output image and the second output image based on the determined weight values.

[0182] According to one embodiment, the step of determining the weight values ​​may include the step of determining at least a portion of the base image as a preserved region, and the step of determining the weight values ​​for the preserved region such that a weight value applied to pixels of the first output image is smaller than a weight value applied to pixels of the second output image.

[0183] According to one embodiment, the step of determining the weight values ​​includes the steps of determining at least a portion of the base image as a preservation area, obtaining a first blend map including weight values ​​applied to pixels of the first output image based on the preservation area, and obtaining a second blend map including weight values ​​applied to pixels of the second output image based on the preservation area, wherein a weight value corresponding to the preservation area in the first blend map may be smaller than a weight value corresponding to a remaining area excluding the preservation area, and a weight value corresponding to the preservation area in the second blend map may be larger than a weight value corresponding to a remaining area excluding the preservation area.

[0184] According to one embodiment, the step of determining the preservation area may include the step of detecting a preset type of object in the basic image and the step of determining an area in which the preset type of object is detected as the preservation area.

[0185] An electronic device according to one embodiment of the present disclosure includes a memory in which a program or at least one instruction is stored, and at least one processor operably coupled to the memory, wherein the at least one processor executes the program stored in the memory or the at least one instruction, thereby causing the electronic device to obtain a multi-frame input, perform demosaicing and denoising on the multi-frame input, and adjust the denoising intensity to be weaker as the number of frames included in the multi-frame input increases, and obtain a base image by merging the plurality of frames on which the demosaicing and denoising have been performed, and then perform super-resolution processing on the base image to obtain a zoomed image.

[0186] According to one embodiment, the electronic device can generate the basic image by determining a weight value corresponding to each of the plurality of frames based on a result of comparing the plurality of frames included in the multi-frame input when obtaining the basic image, and then performing a weighted sum on the plurality of frames using the determined weight value.

[0187] According to one embodiment, in determining the weight value, the electronic device may determine one of a plurality of frames included in the multi-frame input as a reference frame, and then determine the weight value based on a difference in pixel values ​​between the reference frame and the remaining frames and a standard deviation of pixel values ​​of the reference frame.

[0188] According to one embodiment, the electronic device can generate the enlarged image by increasing the resolution and enlarging the size of the basic image while preserving structural features for at least some areas of the basic image when obtaining the enlarged image.

[0189] According to one embodiment, the electronic device may obtain a first output image by inputting the basic image into a generative super resolution module based on a diffusion model to obtain the enlarged image, obtain a second output image by inputting the basic image into a structure preserved module for increasing the resolution while preserving structural features, and then generate the enlarged image by performing a weighted sum on the first output image and the second output image.

[0190] According to one embodiment, the electronic device can generate the first output image by performing downscaling on the basic image, inputting the downscaling onto the super-resolution module, and performing upscaling on the output of the super-resolution module when obtaining the first output image.

[0191] According to one embodiment, the electronic device can generate the enlarged image by performing the weighted sum, by determining weight values ​​to be applied to pixels of the first output image and pixels of the second output image based on the base image, and then performing a weighted sum on the first output image and the second output image based on the determined weight values.

[0192] According to one embodiment, in determining the weight values, the electronic device may determine at least a portion of the basic image as a preserved region, and then determine the weight values ​​for the preserved region such that a weight value applied to pixels of the first output image is smaller than a weight value applied to pixels of the second output image.

[0193] According to one embodiment, in determining the weight values, the electronic device determines at least a portion of the base image as a preservation area, obtains a first blend map including weight values ​​applied to pixels of the first output image based on the preservation area, and then obtains a second blend map including weight values ​​applied to pixels of the second output image based on the preservation area, wherein a weight value corresponding to the preservation area in the first blend map may be smaller than a weight value corresponding to a remaining area excluding the preservation area, and a weight value corresponding to the preservation area in the second blend map may be larger than a weight value corresponding to a remaining area excluding the preservation area.

[0194] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, "application" and "program" may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code, including source code, object code, and executable code. "Computer-readable medium" may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0195] Additionally, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, and removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0196] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0197] The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present disclosure. For example, suitable results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the systems, structures, devices, circuits, etc. described are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form.

[0198] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method for generating a digital zoom image using generative AI, A step of acquiring multi frame input; A step of performing demosaicing and denoising on the multi-frame input, wherein the denoising intensity is adjusted to be weaker as the number of multiple frames included in the multi-frame input increases; A step of obtaining a base image by merging multiple frames on which the demosaicing and denoising have been performed; and A method comprising the step of obtaining a zoomed image by performing super resolution processing on the above basic image.

2. In paragraph 1, The steps for obtaining the above basic image are: A step of determining a weight value corresponding to each of the plurality of frames based on a result of comparing the plurality of frames included in the multi-frame input; and A method characterized by comprising a step of generating the basic image by performing a weighted sum on the plurality of frames using the determined weight values.

3. In either paragraph 1 or paragraph 2, The step of determining the above weight value is: A step of determining one of the plurality of frames included in the above multi-frame input as a reference frame; and A method characterized by comprising a step of determining the weight value based on the difference in pixel values ​​between the reference frame and the remaining frames and the standard deviation of the pixel values ​​of the reference frame.

4. In any one of paragraphs 1 to 3, The step of obtaining the above enlarged image is: A step of obtaining a first output image by inputting the above basic image into a generative super resolution module based on a diffusion model; A step of obtaining a second output image by inputting the basic image into a structure preserved module for increasing the resolution while preserving structural features; and A method characterized by comprising a step of generating the enlarged image by performing a weighted sum on the first output image and the second output image.

5. In any one of paragraphs 1 to 4, The step of obtaining the first output image is as follows: A method characterized in that the first output image is generated by performing downscaling on the basic image, inputting the same into the super-resolution module, and performing upscaling on the output of the super-resolution module.

6. In any one of paragraphs 1 to 5, The step of generating the enlarged image by performing the above weighted sum is: A step of determining weight values ​​to be applied to pixels of the first output image and pixels of the second output image based on the base image; and A method comprising the step of generating the enlarged image by performing a weighted sum on the first output image and the second output image based on the determined weight values.

7. In any one of paragraphs 1 to 6, The step of determining the above weight values ​​is: A step of determining at least a portion of the above basic image as a preserved region; and A method characterized in that, for the above preservation area, the method comprises a step of determining the weight values ​​so that the weight values ​​applied to the pixels of the first output image are smaller than the weight values ​​applied to the pixels of the second output image.

8. In any one of paragraphs 1 to 7, The step of determining the above weight values ​​is: A step of determining at least a portion of the above basic image as a preservation area; A step of obtaining a first blend map including weight values ​​applied to pixels of the first output image based on the preservation area; and A step of obtaining a second blend map including weight values ​​applied to pixels of the second output image based on the preservation area, In the first blend map, the weight value corresponding to the preservation area is smaller than the weight value corresponding to the remaining areas excluding the preservation area, A method characterized in that the weight value corresponding to the preservation area in the second blend map is greater than the weight value corresponding to the remaining areas excluding the preservation area.

9. In any one of paragraphs 1 to 8, The step of determining the above preservation area is: A step of detecting a preset type of object in the above basic image; and A method characterized by comprising a step of determining an area in which an object of the preset type is detected as the preservation area.

10. In electronic devices (700), a memory (740) in which a program or at least one instruction is stored; and comprising at least one processor (730) operably coupled to the memory; The electronic device (700) executes the program stored in the memory (740) or the at least one instruction by the at least one processor (730). Obtain multi frame input (10), Demosaicing and denoising are performed on the above multi-frame input (10), and the denoising intensity is adjusted to be weaker as the number of multiple frames included in the above multi-frame input (10) increases. After obtaining a base image by merging multiple frames on which the above demosaicing and denoising have been performed, An electronic device that obtains a zoomed image (20) by performing super resolution processing on the above basic image.

11. In paragraph 10, The above electronic device (700) obtains the basic image, After determining a weight value corresponding to each of the plurality of frames based on the result of comparing the plurality of frames included in the above multi-frame input (10), An electronic device characterized in that the basic image is generated by performing a weighted sum on the plurality of frames using the determined weight values.

12. In either of paragraphs 10 or 11, The above electronic device (700) obtains the enlarged image (52), By inputting the above basic image (51) into a generative super resolution module (460) based on a diffusion model, a first output image is obtained, After obtaining a second output image by inputting the above basic image (51) into a structure preserved module (470) to increase the resolution while preserving the structural features, An electronic device characterized in that the enlarged image (52) is generated by performing a weighted sum on the first output image and the second output image.

13. In any one of paragraphs 10 to 12, The above electronic device (700) obtains the first output image, An electronic device characterized in that the first output image is generated by performing downscaling on the basic image (51) and then inputting it to the super-resolution module, and performing upscaling on the output of the super-resolution module.

14. In any one of paragraphs 10 to 13, The electronic device (700) generates the enlarged image (64) by performing the weighted sum. Based on the above basic image (61), after determining the weight values ​​to be applied to the pixels of the first output image (62) and the pixels of the second output image (63), An electronic device characterized in that the enlarged image (64) is generated by performing a weighted sum on the first output image (62) and the second output image (63) based on the determined weight values.

15. In any one of paragraphs 10 to 14, The electronic device (700) determines the weight values, After determining at least a portion of the above basic image (61) as a preserved region, An electronic device characterized in that, for the above preservation area, the weight values ​​are determined so that the weight values ​​applied to the pixels of the first output image (62) are smaller than the weight values ​​applied to the pixels of the second output image (63).

Images