Methods for generating target images, electronic devices, and non-transitory computer-readable media

Abstract

A method for generating a target image according to an embodiment of the present disclosure includes: acquiring a captured blurred image by a camera assembly including optics; applying a restoration process to the captured blurred image using a restoration filter to generate a restored sharp image; generating a regular fusion mask based on the captured blurred image; masking the restored sharp image using the regular fusion mask to generate a first intermediate image; generating a reverse fusion mask; applying a noise reduction process to the captured blurred image to reduce noise in the captured blurred image and generate a denoised image, wherein the noise reduction process is adjusted based on adjustment information; masking the denoised image using the reverse fusion mask to generate a second intermediate image; and combining the first intermediate image and the second intermediate image to generate the target image.

Description

Technical Field

[0001] This disclosure relates to methods for generating target images, electronic devices, and non-transitory computer-readable media. Background Technology

[0002] Smartphones and tablets are widely used in our daily lives. Many electronic devices today are equipped with camera components for capturing images. Some are portable, making them easy to carry. Therefore, users of these devices can easily take photos of objects anytime, anywhere using their built-in camera components.

[0003] When images are captured using camera components, image sharpness is degraded by optical aberrations such as coma and astigmatism. Over the past decade, numerous academic papers have proposed techniques to improve blurry images captured due to these optical aberrations. However, most of the published techniques are highly complex or computationally expensive. Therefore, these techniques are impractical and unsuitable for implementation in electronic devices such as smartphones and tablets, where the computing power is insufficient. Summary of the Invention

[0004] This disclosure aims to solve at least one of the aforementioned technical problems. Therefore, this disclosure requires providing a method for generating a target image, an electronic device, and a non-transitory computer-readable medium.

[0005] According to this disclosure, a method for generating a target image may include:

[0006] Acquire a blurred image captured by a camera assembly, which includes optics;

[0007] The restoration process is applied to the captured blurred image using a restoration filter to generate a restored sharp image. This restoration filter is a filter that restores the sharpness of the captured blurred image based on the optical properties of the optical device.

[0008] Generate a regular fusion mask based on the captured blurred image;

[0009] The recovered sharp image is masked using a regularized fusion mask to generate a first intermediate image;

[0010] Generate a reverse fusion mask, wherein the regular fusion mask is reversed;

[0011] The noise reduction process is applied to the captured blurred image to reduce noise in the captured blurred image and generate a noise-reduced image, wherein the noise reduction process is adjusted based on adjustment information depending on the filtering characteristics of the recovery filter and / or the optical characteristics of the optics.

[0012] The denoised image is masked using a reverse fusion mask to generate a second intermediate image; and

[0013] The first intermediate image and the second intermediate image are combined to generate the target image.

[0014] According to this disclosure, the electronic device may include:

[0015] Camera assemblies including optical components; and

[0016] The processor is configured as follows:

[0017] Acquire a blurred image captured by a camera assembly, which includes optics;

[0018] The restoration process is applied to the captured blurred image using a restoration filter to generate a restored sharp image. The restoration filter is a filter that restores the sharpness of the captured blurred image based on the optical characteristics of the optical device.

[0019] Generate a regular fusion mask based on the captured blurred image;

[0020] The recovered sharp image is masked using the regularized fusion mask to generate a first intermediate image;

[0021] Generate an inverse fusion mask, wherein the regular fusion mask is inverted;

[0022] The noise reduction process is applied to the captured blurred image to reduce noise in the captured blurred image and generate a noise-reduced image, wherein the noise reduction process is adjusted based on adjustment information that depends on the filtering characteristics of the recovery filter and / or the optical characteristics of the optics.

[0023] The denoised image is masked using a reverse fusion mask to generate a second intermediate image; and the first intermediate image and the second intermediate image are combined to generate a target image.

[0024] According to this disclosure, a non-transitory computer-readable medium stores program instructions, wherein when executed by an electronic device, the program instructions cause the electronic device to perform at least the following operations:

[0025] Acquire a blurred image captured by a camera assembly, which includes optics;

[0026] Restoration filters are used to apply the restoration process to a captured blurred image to generate a restored sharp image. Restoration filters are filters that restore the sharpness of a captured blurred image based on the optical properties of optical devices.

[0027] Generate a regular fusion mask based on the captured blurred image;

[0028] The recovered sharp image is masked using a regular fusion mask to generate a first intermediate image;

[0029] Generate an inverse fusion mask, wherein the regular fusion mask is inverted;

[0030] The noise reduction process is applied to the captured blurred image to reduce noise in the captured blurred image and generate a noise-reduced image, wherein the noise reduction process is adjusted based on adjustment information that depends on the filtering characteristics of the recovery filter and / or the optical characteristics of the optics.

[0031] A second intermediate image is generated by masking the denoised image using a reverse fusion mask; and

[0032] Combine the first intermediate image and the second intermediate image to generate the target image. Attached Figure Description

[0033] These and / or other aspects and advantages of the embodiments of this disclosure will become apparent and more readily understood from the following description with reference to the accompanying drawings, in which:

[0034] Figure 1 This is a plan view of the first side of an electronic device according to an embodiment of the present disclosure;

[0035] Figure 2 This is a plan view of the second side of an electronic device according to an embodiment of the present disclosure;

[0036] Figure 3 This is a block diagram of an electronic device according to an embodiment of the present disclosure;

[0037] Figure 4 This is an illustration of optical aberrations in optical devices;

[0038] Figure 5 A formula indicating the relationship between a captured blurry image and an ideal sharp image is shown;

[0039] Figure 6 This is a visual illustration of a method for improving the sharpness of a blurred image captured in an electronic device according to embodiments of the present disclosure;

[0040] Figure 7 The cost function c(L) including the fidelity term and the regularization term is shown;

[0041] Figure 8 This is a formula showing how to calculate the inverse filter based on the cost function c(L);

[0042] Figure 9The image and inverse filter of the blur kernel K in the spatial domain are shown. Images;

[0043] Figure 10 The problem of restoring a clear image is shown;

[0044] Figure 11A An outline of the fusion process of an electronic device according to an embodiment of the present disclosure is shown;

[0045] Figure 11B An example of a filter used in the noise reduction process is shown, namely a bilateral filter, and its formula is given.

[0046] Figure 11C An example of a portion of the captured blurred image is shown.

[0047] Figure 11D The spatial weights W in the bilateral filter are shown. s And intensity weight W i An example of a formula.

[0048] Figure 11E The image shows a blurred image captured before the noise reduction process was applied using a bilateral filter, and a noise-reduced image after the noise reduction process was applied using a bilateral filter.

[0049] Figure 11F An example of a recovery filter array is shown.

[0050] Figure 12 An example of how to generate a regular fusion mask in an electronic device according to an embodiment of the present disclosure is shown;

[0051] Figure 13 An example of a lookup table for non-linearly modulating a captured blurred image is shown;

[0052] Figure 14 This is a visual illustration of the lens shadow model in the electronic device according to the embodiments of this disclosure;

[0053] Figure 15 This is an intuitive illustration of the first choice for reflecting shadow properties on a regularized blending mask;

[0054] Figure 16 This is an intuitive illustration of the second choice for reflecting shadow properties on a regularized blending mask;

[0055] Figure 17 This is an intuitive illustration of the third option for reflecting shadow properties on a regularized blending mask;

[0056] Figure 18 It is a visual illustration of the smooth transition in the final fusion mask;

[0057] Figure 19 This is a visual illustration of a method for generating a target image in an electronic device according to embodiments of the present disclosure;

[0058] Figure 20 This is a flowchart of the target image generation process in an electronic device according to embodiments of the present disclosure; and

[0059] Figure 21 A comparison is shown between a target image generated by prior art and a target image generated by an electronic device according to an embodiment of the present disclosure.

[0060] Figure 22 An example is shown of applying the noise reduction process to the chromaticity components (U, V) in addition to the luminance component (Y). Detailed Implementation

[0061] Various embodiments of this disclosure will be described in detail, and examples of each embodiment will be shown in the accompanying drawings. Throughout this specification, the same or similar elements and elements having the same or similar functions are denoted by the same reference numerals. The embodiments described herein with reference to the accompanying drawings are illustrative and intended to illustrate this disclosure, but should not be construed as limiting this disclosure.

[0062] Figure 1 This is a plan view of the first side of the electronic device 10 according to an embodiment of the present disclosure. Figure 2 This is a plan view of the second side of an electronic device 10 according to an embodiment of the present disclosure. The first side may be referred to as the rear side of the electronic device 10, and the second side may be referred to as the front side of the electronic device 10.

[0063] like Figure 1 and Figure 2 As shown, the electronic device 10 may include a display 20 and a camera assembly 30. In this embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34, and a sub-camera 36. The first main camera 32 and the second main camera 34 can capture images on a first side of the electronic device 10, and the sub-camera 36 can capture images on a second side of the electronic device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called external cameras, while the sub-camera 36 is a so-called internal camera. As an example, the electronic device 10 may be a mobile phone, a tablet computer, a personal digital assistant, etc.

[0064] Each of the first main camera 32, the second main camera 34, and the sub-camera 36 has an imaging sensor that converts light that has passed through a color filter into an electrical signal. The signal value of the electrical signal depends on the amount of light that has passed through the color filter.

[0065] Although the electronic device 10 according to this embodiment has three cameras, the electronic device 10 may have fewer than three cameras or more than three cameras. For example, the electronic device 10 may have two, four, five cameras, and so on.

[0066] Figure 3 This is a block diagram of the electronic device 10 according to this embodiment. (See diagram below.) Figure 3 As shown, in addition to the display 20 and camera assembly 30, the electronic device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46, and a communication circuit 48. The display 20, camera assembly 30, main processor 40, image signal processor 42, memory 44, power supply circuit 46, and communication circuit 48 are interconnected via a bus 50.

[0067] The main processor 40 executes one or more program instructions stored in the memory 44. The main processor 40 implements various applications and data processing of the electronic device 10 by executing these program instructions. The main processor 40 can be one or more computer processors. The main processor 40 is not limited to a single CPU core, but can have multiple CPU cores. The main processor 40 can be the main CPU of the electronic device 10, an image processing unit (IPU), or a DSP provided with the camera assembly 30.

[0068] The image signal processor 42 controls the camera assembly 30 and processes various image data captured by the camera assembly 30 to generate target image data. For example, the image signal processor 42 can apply processes such as demosaicing, noise reduction, automatic exposure, automatic focus, automatic white balance, and high dynamic range to the image data captured by the camera assembly 30.

[0069] In this embodiment, the main processor 40 and the image signal processor 42 cooperate with each other to generate target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture images of the object by means of the camera assembly 30 and apply various image processing techniques to the captured image data.

[0070] Memory 44 stores program instructions and various data to be executed by the main processor 40. For example, memory 44 may also store data of captured images.

[0071] Memory 44 may include high-speed RAM and / or non-volatile memory such as flash memory and disk storage. That is, memory 44 may include non-transitory computer-readable media that stores program instructions.

[0072] The power supply circuit 46 may include a battery, such as a lithium-ion rechargeable battery, and a battery management unit (BMU) for managing the battery.

[0073] Communication circuit 48 is configured to receive and transmit data to communicate with base stations of a telecommunications network system, the Internet, or other devices via wireless communication. Wireless communication can employ any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTM), LTE-Advanced (LTE-A), and 5th Generation (5G). Communication circuit 48 may include an antenna and radio frequency (RF) circuitry.

[0074] Figure 4 This is an illustration of optical aberrations. In other words, when an image is captured in the camera assembly 30, the image sharpness is degraded due to optical aberrations (such as coma, astigmatism, etc.) in the optics. Therefore, the point light source image is diffused by optical aberrations in the optical system, and the point light source image is no longer a point on the image plane.

[0075] Typically, the diffused point light source is modeled by a function called the point spread function (PSF), which represents the way the captured image degrades and its degradation characteristics; this is known as optical blur. Hereafter, the blurred image captured by camera assembly 30 is also referred to as the captured blurred image.

[0076] Figure 5 A formula indicating the relationship between the captured blurred image B and the ideal sharp image L is shown. (As follows) Figure 5 As shown, the sharpness of the captured blurred image B is represented by K*L+n. K indicates the same blur kernel as PSF. L indicates the ideal sharp image without noise, i.e., the ideal image. n indicates noise. "*" indicates circular convolution. This formula shows that images captured by camera assembly 30 always include noise. Since noise is unavoidable when capturing images, this noise is also called shot noise.

[0077] Figure 6This is a visual illustration of a method for improving the sharpness of a captured blurred image in an electronic device 10 according to an embodiment of the present disclosure. In this embodiment, the electronic device 10 obtains a recovered sharp image L_res through a filtering recovery process using an inverse filter calculated based on the PSF. The recovered sharp image L_res is closer to the ideal sharp image L. However, the filtering recovery process also filters out noise in the captured blurred image B, and thus also increases the noise. The inverse filter is one example of a recovery filter, which is a filter that recovers the sharpness of a captured blurred image based on the optical properties of an optics device.

[0078] Next, we will explain how to calculate the inverse filter based on the blur kernel K, where the blur kernel K is the PSF. Figure 7 The cost function c(L) in this embodiment is shown. For example... Figure 7 As shown, the cost function c(L) is essentially represented by the minimum independent variable of "the captured blurred image B - the blurred kernel K * the ideal sharp image L". That is, the term within the quotation marks is the fidelity term. Furthermore, in Figure 7 In the formula shown, a regularization term is also introduced into the cost function c(L) to apply a penalty. That is, by introducing a regularization term into the cost function c(L), overfitting can be avoided. Therefore, overfitting can be suppressed to reduce noise in the recovered sharp image L_res. In this paper, D indicates the regularization function, and ρ indicates the regularization gain.

[0079] Figure 8 This is a formula illustrating how to calculate the inverse filter based on the cost function c(L). It can be solved in the frequency domain. Figure 7 The cost function c(L) is used to obtain Figure 8 The Fourier transform F(L) is used. By performing an inverse Fourier transform F(L), the recovered sharp image L_res can be obtained. This can be achieved through an inverse filter. The captured blurred image B is used to obtain the recovered sharp image L_res, therefore the inverse filter can be specified in this formula.

[0080] Figure 9 The image and inverse filter of the blur kernel K in the spatial domain are shown. The image. Given the inverse filter K. -1 The reason for adding the estimated value (hat) is that the blur kernel K is not an actual measured value, but a design value. Due to assembly errors and dimensional errors in the components of the camera assembly 30 and the electronic device 10, the design value of the blur kernel K is not equal to the actual measured value of the blur kernel K. Therefore, the inverse filter... With the actual inverse filter K in electronic device 10 -1Slightly different.

[0081] For electronic device 10 and its users, obtaining the actual measured value of the fuzzy kernel K is a very complex and tedious process. Furthermore, the inverse filter... The filter is sufficiently fine to recover the captured blurred image B. Therefore, the electronic device 10 according to this embodiment calculates the inverse filter based on the design value of the blur kernel K.

[0082] Figure 10 The problem with the recovered clear image is shown. For example... Figure 10 As shown, the captured blurred image contains noise. Therefore, if an inverse filter is used... Applying a circular convolution to a captured blurred image improves the sharpness of the recovered image, although it increases noise in the captured blurred image.

[0083] Furthermore, the captured blurred image contains noise, so it is desirable to reduce the noise in the captured blurred image as much as possible before generating the target image. Noise can be reduced by applying a denoising process to the captured blurred image, from which a denoised image can then be generated.

[0084] Therefore, in order to solve these problems, the electronic device 10 according to an embodiment of the present disclosure introduces a fusion process for fusing a restored clear image with a filtering restoration process applied and a denoised image with a noise reduction process applied but not a filtering restoration process.

[0085] Figure 11A An outline of the fusion process of an electronic device 10 according to an embodiment of the present disclosure is shown. During the fusion process, an inverse filter is utilized. A recurrent convolution is applied to the captured blurred image to generate the recovered sharp image.

[0086] Even if noise is present in the high-frequency regions of an image, the human eye cannot detect it. Conversely, if noise is present in the low-frequency regions of an image, the human eye can detect it and easily notice it.

[0087] Therefore, in the electronic device 10 according to an embodiment of the present disclosure, a regular fusion mask is used to mask the low-frequency regions in the recovered clear image to generate a first intermediate image.

[0088] On the other hand, the captured blurred image also contains noise. Therefore, in order to reduce the noise in the captured blurred image, a denoising process is applied to the captured blurred image. After applying the denoising process to the captured blurred image, a denoised image is obtained.

[0089] A second intermediate image is generated by masking the denoised image using a reversed fusion mask. The reversed fusion mask is a mask that is the reverse of a regular fusion mask. For example, a reversed fusion mask can be obtained by inverting a regular fusion mask. Subsequently, the electronic device 10 according to this embodiment combines the first intermediate image and the second intermediate image to generate a target image.

[0090] The target image is generated through this fusion process, replacing noisy low-frequency regions in the restored sharp image with noise-free low-frequency regions in the denoised image due to the denoising process. Therefore, the low-frequency regions of the target image are free of noise. On the other hand, the sharpness of the high-frequency regions in the target image is improved due to the restoration process. In other words, the texture in the high-frequency regions of the target image can be fine.

[0091] Furthermore, a noise reduction process is used to reduce noise in the low-frequency regions of the captured blurred image to generate a denoised image. Therefore, the target image generated by combining the recovered sharp image and the denoised image has very little noise in the low-frequency regions.

[0092] Typically, texture in high-frequency regions degrades during the denoising process. However, in this embodiment, the texture in the high-frequency regions of the denoised image is replaced with the texture in the high-frequency regions of the recovered sharp image. Therefore, the denoising process can be adjusted to effectively reduce noise in the low-frequency regions of the captured blurred image. This improves user satisfaction with the target image.

[0093] Figure 11B The formula for a bilateral filter, which serves as an example of a filter used in the noise reduction process, is shown. Figure 11C An example of a portion of a captured blurred image is shown. Bilateral filters are known to remove noise while preserving image edges. However, bilateral filters are just one example used in the noise reduction process, and other noise reduction methods can be applied to reduce noise.

[0094] like Figure 11B As shown, a bilateral filter can be represented as:

[0095]

[0096] q i，j The values ​​of the denoised pixels and the center pixel in the filter kernel are indicated. The first term on the right-hand side of the formula is the filter kernel used as normalization weights, and the image block I(i,j) in the second term is the image block to be processed in the captured blurred image. i and j indicate the x and y positions in the captured blurred image, respectively, and (i,j) indicates the center coordinates of the filter kernel. ".*" indicates element-wise multiplication.

[0097] W s Indicator spatial weights, W i Indicator intensity weight. Figure 11D The spatial weights W in the bilateral filter are shown. s And intensity weight W i An example of a formula. For example... Figure 11D As shown, the spatial weight W s It is a predetermined Gaussian filter kernel, and the size of the Gaussian filter kernel is the same as the size of the image patch I(i,j). H indicates the size of the Gaussian filter kernel in the x-direction, and V indicates the size of the Gaussian filter kernel in the y-direction.

[0098] Spatial weight W s It is represented as:

[0099]

[0100] Where n indicates the x-position of the Gaussian filter kernel, and m indicates the y-position of the Gaussian filter kernel. Parameter δ s This represents the variance of the intensity weights. As understood from this formula, the spatial weights W... s Through δ s Control. That is, parameter δ. s The larger the value, the stronger the smoothing effect.

[0101] Intensity weight W i It is represented as:

[0102]

[0103] Wherein, parameter δ i Indicates the spatial weight variance; the second power indicates element-wise multiplication. (I(i,j)-p) i，j ) 2 Indicates the intensity and p of each pixel i，j The intensity difference between the values. In this embodiment, as understood from the formula, the intensity weight W i It can be achieved through the parameter δ i To control it. That is, the parameter δ i The smaller the value, the better the edge preservation effect. In other words, the parameter δ... i The smaller the value, the weaker the smoothing effect.

[0104] Figure 11E The image shown is a blurred image captured before the noise reduction process is applied using the aforementioned bilateral filter, and a noise-reduced image after the noise reduction process is applied using the aforementioned bilateral filter.

[0105] like Figure 11EAs shown, a denoising process is applied using a bilateral filter to reduce noise in planar regions, while preserving edges in the denoised image. Additionally, the spatial weights W in the bilateral filter... s And intensity weight W i It can be achieved through the parameter δ s and parameter δ i To control this. Therefore, in this embodiment, the parameter δ is adjusted based on adjustment information that depends on the filtering characteristics of the recovery filter used by the recovery process and / or the optical characteristics of the optics in the camera assembly 30. s and parameter δ i .

[0106] For example, the adjustment information may depend on the kernel size of the recovery filter, the filter strength of the recovery filter, the frequency characteristics of the recovery filter, the spatial characteristics of the optics of the camera assembly 30, and / or the point spread function (PSF) of the optics of the camera assembly 30.

[0107] Figure 11F An example of a recovery filter array is shown. For example... Figure 11F As shown, in the central region of the restoration filter array, the point spread function (PSF) is small and isotropic, exhibiting high brightness. Therefore, even after the restoration process is applied, a well-restored sharp image with good sharpness recovery and relatively low noise can be expected. Therefore, the parameter δ... s It can be small to not very smooth, the parameter δ i It can be so large that the edges are not preserved much.

[0108] On the other hand, in the corner regions of the restoration filter array, the point spread function (PSF) is large and anisotropic, and exhibits low brightness (lens shadow). Therefore, even after applying the restoration process, good sharpness restoration cannot be expected. Furthermore, the low brightness introduces significant noise. Therefore, the parameter δ... s It should be larger to make it smoother, parameter δ i It should be smaller to better preserve the edges.

[0109] From the center region to the corner region in the captured blurred image, the parameter δ s It may gradually increase, the parameter δ i It may gradually become smaller.

[0110] Figure 12 An example of how a regular fusion mask is generated in an electronic device 10 according to an embodiment of the present disclosure is shown. Figure 12As shown, a regularized fusion mask is generated based on the captured blurred image. More specifically, the electronic device 10 acquires the captured blurred image, for example, from the camera assembly 30 or the image signal processor 42. Subsequently, the electronic device 10 applies a circular convolution to the captured blurred image using an inverse filter to improve the sharpness of the captured blurred image and generate a restored sharp image. However, if the restored sharp image has already been generated in another process, this process can be omitted.

[0111] In addition, the electronic device 10 performs an averaging process on the captured blurry image to reduce the sharpness of the captured blurry image and generate an average image.

[0112] Then, the electronic device 10 subtracts the recovered sharp image from the average image to generate a subtracted image, and then calculates the absolute value of the subtracted image to generate a difference image. Through these processes, a difference image can be obtained, which demonstrates the degree to which the filtering restoration process using an inverse filter alters the pixel brightness. In other words, each pixel in the difference image has a specific value indicating the level of brightness change caused by the filtering restoration process.

[0113] exist Figure 12 In the diagram, gray areas indicate regions where brightness has been significantly altered by the filtering and restoration process using an inverse filter. Conversely, black areas indicate regions where brightness has not been significantly altered by the filtering and restoration process using an inverse filter.

[0114] On the other hand, the electronic device 10 non-linearly modulates the captured blurred image to generate a temporary threshold map. In this embodiment, a lookup table (LUT) is used to non-linearly modulate the captured blurred image.

[0115] Figure 13 An example of a lookup table (LUT) for a blurred image captured by nonlinear modulation is shown. In this example, if the pixel values ​​of the captured blurred image are low, the pixel values ​​of the temporary threshold map are increased from their original values. Conversely, if the pixel values ​​of the captured blurred image are in the middle range, the pixel values ​​of the temporary threshold map are decreased from their original values. Furthermore, if the pixel values ​​of the captured blurred image are greater than a certain value, the pixel values ​​of the temporary threshold map are restricted to that value.

[0116] Figure 13 This is an example of a lookup table LUT, which is not limited to... Figure 13 The example shown is not limited to using a lookup table (LUT) to nonlinearly modulate a captured blurred image. Various other methods can be applied to nonlinearly modulate the captured blurred image.

[0117] Next, as Figure 12 As shown, according to an embodiment of this disclosure, the electronic device 10 multiplies a temporary threshold map by a certain value to adjust the gain of the temporary threshold map, and then can calculate a final threshold map. This certain value can be less than one or greater than one. In the final threshold map, each pixel is used to indicate a threshold for whether pixels of the recovered sharp image should be masked when generating the first intermediate image.

[0118] More specifically, the electronic device 10 generates a regular fusion mask based on the thresholded difference image and the final threshold map. During the thresholding process, if the value of `diff_img`, which represents the pixel value of the difference image, is equal to or greater than the value of `th_map`, which represents the pixel value of the final threshold map, then the value of `blend_mask`, which represents the pixel value of the regular fusion mask, is `blend_ratio_high`. For example, `blend_ratio_high` is 95%.

[0119] On the other hand, if the value of diff_img is less than the value of th_map, then the value of blend_mask is blend_ratio_low. For example, blend_ratio_low is 5%.

[0120] In this embodiment, when using a regular blending mask, if the blend_mask of the pixels of the regular blending mask is blend_ratio_high (95%), a first intermediate image is generated such that the pixels of the first intermediate image contain 95% of the corresponding pixels of the recovered clear image and 5% of the corresponding pixels of the captured blurred image.

[0121] On the other hand, if the blend_mask of the pixels of the regular blend mask is blend_ratio_low (5%), a first intermediate image is generated such that the pixels of the first intermediate image contain 5% of the corresponding pixels of the recovered sharp image and 95% of the corresponding pixels of the captured blurred image. In this embodiment, blend_ratio_high is increased until blend_ratio_low should be 1.

[0122] The values ​​of the recovered sharp image and the captured blurry image are fused in the same pixel, making it impossible for the human eye to distinguish the boundary between the area used by the recovered sharp image and the area used by the captured blurry image in the target image.

[0123] The thresholding process eliminates small grayscale areas, such as noise in the difference image. When the sharpness of the captured blurred image is restored, the bright areas in the restored sharp image contain noise. Therefore, in this embodiment, the values ​​of the bright areas in the final threshold map are high. Consequently, the bright areas in the regular fusion mask have a low fusion ratio, and the bright areas in the restored sharp image are masked by the regular fusion mask.

[0124] An inverted blend mask can be generated by inverting a regular blend mask. That is, by inverting a regular blend mask, the blend mask of the pixels with blend_ratio_high (95%) of the regular blend mask is converted into the blend mask of the pixels with blend_ratio_low (5%) of the inverted blend mask, and the blend mask of the pixels with blend_ratio_low (5%) of the regular blend mask is converted into the blend mask of the pixels with blend_ratio_high (95%) of the inverted blend mask.

[0125] Incidentally, in the example mentioned above, `blend_ratio_high` is less than 100%, and `blend_ratio_low` is greater than 0%. However, in a regularized blend mask, `blend_ratio_high` can be 100%, while `blend_ratio_low` can be 0%. In this case, in a reversed blend mask, `blend_ratio_high` is also 100%, and `blend_ratio_low` is also 0%. However, the value of `blend_ratio_high` is higher than the value of `blend_ratio_low`.

[0126] Due to the characteristics of the optics in camera assembly 30, the brightness around the corners of the captured image is reduced. That is, the amount of light in the corners is low due to lens shading (also known as vignetting). Therefore, in some cases, the captured blurred image has been compensated to correct for the vignetting characteristics. To compensate, the electronic device 10 multiplies the captured image by an appropriate gain. However, noise characteristics are strongly dependent on this compensation. Therefore, noise characteristics can be taken into account when generating a regular fusion mask.

[0127] Optionally, in the electronic device 10 according to embodiments of the present disclosure, for example, a lens shading model (LSM) is introduced. Figure 14 This is a visual illustration of the lens shadow model LSM in this embodiment.

[0128] like Figure 14As shown, the captured image, following the original lens shadow characteristics of the optics, has a brighter area in the center and a darker area in the corners. In this embodiment, a lens shadow model (LSM) is generated based on the original lens shadow characteristics. That is, the LSM is generated through a nonlinear transformation of the original lens characteristics. The purpose of the nonlinear transformation is to adjust the brightness based on the LSM to obtain a regular fusion mask suitable for masking noisy areas.

[0129] <First Choice>

[0130] Figure 15 This is an intuitive illustration of the first choice of reflective shadow characteristics on a regularized fusion mask. In the first choice, the electronic device 10 performs pixel-wise multiplication for the differential image and the lens shadow model LSM.

[0131] By correcting the shadow characteristics of the captured image, the brightness around the corners of the captured image has been improved. That is, the brightness around the corners of the captured blurred image has also been improved. Therefore, according to the first choice, the brightness around the corners of the difference image is reduced by using the Lens Shadow Model (LSM). Thus, the correction effect of the shadow characteristics of the regularized fusion mask can be compensated.

[0132] <Second Choice>

[0133] Figure 16 This is an intuitive illustration of the second choice regarding the reflective shadow characteristics on a regularized fusion mask. In this second choice, the electronic device 10 performs pixel-by-pixel multiplication on the final threshold map and the inverted lens shadow model to modify the final threshold map. The inverted lens shadow model is calculated using the inverted lens shadow model LSM. Therefore, the area around the center of the inverted lens shadow model is dark, while the area around the corners of the inverted lens shadow model is bright.

[0134] By correcting the shadow characteristics of the captured image, the amplitude of noise around the corners of the captured image has been increased. Therefore, by increasing the corner values ​​of the final threshold map, the corner regions of the regularized fusion mask can be easily set to blend_ratio_low and masked by the regularized fusion mask.

[0135] <Third Choice>

[0136] Figure 17This is an intuitive illustration of the third option for the reflection shadow characteristics on a regularized blend mask. In this third option, the thresholding process mentioned above is modified. That is, the value of blend_mask is calculated by multiplying blend_ratio_high by the lens shadow model LSM(i,j) or by multiplying blend_ratio_low by the inverse lens shadow model. As a result of this calculation, the ratio of the recovered sharp image increases around the center of the regularized blend mask. In other words, the ratio of the recovered sharp image decreases in the corners of the regularized blend mask.

[0137] By correcting the shadow features of the captured image, a significant amount of noise is introduced into the corner regions. Therefore, to eliminate noise in the corner regions, the ratio of the recovered sharp image decreases when pixels are located near the corners of the regularized fusion mask. In other words, the ratio of the captured blurry image increases when pixels are located closer to the corners of the regularized fusion mask. Therefore, when generating the target image, noise in the corners of the recovered sharp image can be suppressed.

[0138] Alternatively, in the electronic device 10 according to an embodiment of the present disclosure, it is possible to smooth the transition in the final fusion mask. Figure 18 It provides a visual illustration of the smooth transition in the final fusion mask.

[0139] like Figure 18 As shown, the final blend mask has two blend_mask values: blend_ratio_high and blend_ratio_low. Therefore, the boundary between the blend_ratio_high and blend_ratio_low regions is very sharp and clear. When using a sharp and clear final blend mask to generate the target image, the target image may appear unnatural to the human eye.

[0140] Therefore, the electronic device 10 according to embodiments of the present disclosure can increase the blurriness of the final blend mask to naturally mix regions with blend_ratio_high and blend_ratio_low. For example, the electronic device 10 increases the blurriness of the final blend mask by performing a simple convolution of a Gaussian blur kernel.

[0141] A blurred blend mask is generated by introducing blur into the final blend mask. In the blurred blend mask, the boundary between the blend_ratio_high and blend_ratio_low regions is blurred. In other words, the pixel values ​​in the boundary regions have transitional values ​​between blend_ratio_low and blend_ratio_high. That is, in the boundary regions, the blend_mask value gradually increases from blend_ratio_low to blend_ratio_high.

[0142] Figure 19 This is a visual explanation of how the target image is generated. For example... Figure 19 As shown and as already based on Figure 11A As briefly described, the target image is generated by masking the recovered sharp image with a regular fusion mask, masking the denoised image with an inverse fusion mask, and combining the recovered sharp image and the denoised image.

[0143] More specifically, a first intermediate image can be generated by performing pixel-wise multiplication of the recovered sharp image and a regularized fusion mask. A second intermediate image can be generated by performing pixel-wise multiplication of the denoised image and an inverted fusion mask. In other words, the masking process can be implemented through pixel-wise multiplication. Therefore, the target image can be generated by combining the first and second intermediate images.

[0144] Figure 20 This is a flowchart of a target image generation process in an electronic device 10 according to an embodiment of the present disclosure. The target image generation process can be executed by a main processor 40 or an image signal processor 42. Optionally, the target image generation process can be executed by a combination of the main processor 40 and the image signal processor 42.

[0145] In this embodiment, for example, the main processor 40 acquires the captured blurred image from the output port of the image signal processor 42. Then, the main processor 40 performs a target image generation process on the captured blurred image and inputs the generated target image to the input port of the image signal processor 42.

[0146] Furthermore, program instructions for implementing the target image generation process can be stored on a non-transitory computer-readable medium. The main processor 40 reads the program instructions from the non-transitory computer-readable medium and executes the program instructions to implement the target image generation process.

[0147] Furthermore, the target image generation process can be performed on the luminance plane of the YUV standard. That is, the luminance plane (Y plane) can undergo the target image generation process in the electronic device 10 according to embodiments of the present disclosure. Of course, other planes or images can undergo the target image generation process disclosed herein.

[0148] like Figure 20 As shown, the main processor 40 of the electronic device 10 obtains a captured blurred image from, for example, an image signal processor 42 (step S10) captured by a camera assembly 30, which includes optics.

[0149] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 applies a circular convolution to the captured blurred image using an inverse filter to generate the recovered sharp image (step S12). The details of this process have been explained above.

[0150] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 generates a regular fusion mask based on the captured blurred image (step S14). The details of this process have been explained above.

[0151] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 uses a regular fusion mask to mask the recovered sharp image to generate a first intermediate image (step S16). The details of this process have been explained above.

[0152] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 generates a reverse fusion mask that reverses the regular fusion mask (step S18). The details of this process have been explained above.

[0153] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 applies a noise reduction process to the captured blurred image to generate a noise-reduced image (step S19). The details of this process have been explained above.

[0154] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 uses a reverse fusion mask to mask the captured blurred image to generate a second intermediate image (step S20). The details of this process have been explained above.

[0155] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 combines the first intermediate image and the second intermediate image to generate the target image (step S22). The details of this process have been explained above. After completing step S22, the target image generation process according to this embodiment ends.

[0156] Figure 21 A comparison is shown between a target image generated by prior art and a target image generated by an electronic device 10 according to an embodiment of this disclosure. Figure 21 As shown, in the prior art, the target image contains noise due to the filtering and restoration process that improves sharpness, or it is blurred because it has not undergone the filtering and restoration process.

[0157] On the other hand, in the target image generated by the electronic device 10 according to this embodiment, noise caused by the filtering recovery process that improves the sharpness of the captured blurry image is eliminated by masking the noise area of ​​the recovered clear image and by using the captured blurry image instead.

[0158] On the other hand, during the generation of the target image, the clarity of the target image's texture is improved by a filtering restoration process that does not mask the superior texture regions of the restored sharp image. Therefore, a more natural and clearer target image can be obtained for the user without increasing costs.

[0159] Furthermore, the electronic device 10 according to this embodiment can simplify the optics of the camera assembly 30 and reduce the number of lens elements of the optics of the camera assembly 30, while obtaining natural and high-quality target images without requiring large and expensive optics of the camera assembly 30.

[0160] like Figure 22 As shown, if the captured blurred image can be defined by the YUV color model, then... Figure 20 The target image generation process shown is applied to the luminance component (Y). However, in addition to the luminance component (Y), other components can be used... Figures 11A to 11E The noise reduction process shown is applied to the chroma components (U, V). Applying the noise reduction process to the chroma components (U, V) can also reduce the noise in the chroma components. The output is then... Figure 20 The target image shown is generated by the luminance component (Y) and the chrominance component (U, V) after the noise reduction process, and is used as an image defined by the YUV color model.

[0161] In the description of embodiments of this disclosure, it should be understood that terms such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” should be interpreted as referring to the direction or position described or shown in the drawings. These related terms are used only to simplify the description of this disclosure and do not indicate or imply that the mentioned devices or elements must have a particular orientation or must be constructed or operated in a particular orientation. Therefore, these terms should not be construed as limiting this disclosure.

[0162] Furthermore, the use of terms such as “first” and “second” herein is for descriptive purposes and is not intended to indicate or imply relative importance or significance, or to imply the number of technical features indicated. Therefore, a feature defined as “first” and “second” may include one or more of that feature. In this disclosure, unless otherwise specified, “a plurality” means “two or more.”

[0163] In the description of embodiments of this disclosure, unless otherwise specified or limited, terms such as “mounted,” “connected,” and “coupled” are used extensively, and may be, for example, a fixed connection, a detachable connection, or an integral connection; may be a mechanical or electrical connection; may be a direct or indirect connection through an intermediate structure; or may be internal communication between two elements under specific circumstances, as would be understood by those skilled in the art.

[0164] In embodiments of this disclosure, unless otherwise specified or limited, a structure where the first feature is "on" or "under" the second feature can include embodiments where the first and second features are in direct contact, or embodiments where the first and second features are not in direct contact but are in contact through an additional feature formed between them. Furthermore, a first feature "on (in contact with) the second feature," "above the second feature (not in contact)," or "on the second feature (in contact through an additional feature)" can include embodiments where the first feature is orthogonally or obliquely "on (in contact with) the second feature," "above the second feature (not in contact)," or "on the second feature (in contact through an additional feature)," or simply means that the first feature is at a height higher than the second feature; while a first feature "below the second feature," "directly below the second feature," or "at the bottom of the second feature" can include embodiments where the first feature is orthogonally or obliquely "below the second feature," "directly below the second feature," or "at the bottom of the second feature," or simply means that the first feature is at a height lower than the second feature.

[0165] The foregoing description provides various embodiments and examples to implement different structures of this disclosure. To simplify this disclosure, specific elements and arrangements have been described above. However, these elements and arrangements are merely examples and are not intended to limit this disclosure. Furthermore, reference numerals and / or lettering may be repeated in different examples of this disclosure. Such repetition is for simplification and clarity and does not indicate a relationship between different embodiments and / or arrangements. Additionally, this disclosure provides examples of different processes and materials. However, those skilled in the art should understand that other processes and / or materials may also be applied.

[0166] Throughout this specification, references to “embodiment,” “some embodiments,” “exemplary embodiments,” “exemplary embodiments,” “example,” “specific example,” or “some examples” mean that a specific feature, structure, material, or characteristic associated with an embodiment or example is included in at least one embodiment or example of this disclosure. Therefore, the foregoing phrases appearing throughout this specification do not necessarily refer to the same embodiment or example of this disclosure. Furthermore, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0167] Any process or method described in the flowchart or otherwise herein can be understood as including one or more modules, segments, or portions of code comprising executable instructions for implementing specific logical functions or steps in the process, and the scope of the preferred embodiments of this disclosure includes other implementations in which those skilled in the art will understand that functionality can be implemented in a different order than that shown or discussed, including in substantially the same order or in the reverse order.

[0168] The logic and / or steps otherwise described herein or shown in flowcharts, such as a specific sequential list of executable instructions for implementing logical functions, may be embodied in any computer-readable medium and used by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, including a processor, or other system capable of obtaining instructions from an instruction execution system, apparatus, or device that executes instructions). For the purposes of this specification, "computer-readable medium" can be any device suitable for including, storing, communicating, propagating, or transmitting a program used by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of computer-readable media include, but are not limited to: electronic connections (electronic devices) having one or more wires, portable computer casings (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optic devices, and compact disk read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because, for example, when it is necessary to obtain the program electronically, the paper or other suitable medium can be optically scanned, and then compiled, decoded or processed in other suitable ways, and then the program can be stored in computer memory.

[0169] It should be understood that each part of this disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, similar to another embodiment, the steps or methods can be implemented by one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing data signal logic functions, application-specific integrated circuits (ASICs) having suitable combinations of logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0170] Those skilled in the art will understand that all or part of the steps in the exemplary methods described above can be implemented by using program commands to related hardware. The program, when executed on a computer, can be stored in a computer-readable storage medium, and includes one or a combination of the steps of the method embodiments of this disclosure.

[0171] Furthermore, each functional unit in the embodiments of this disclosure can be integrated into the processing module, or these units can be physically separate, or two or more units can be integrated into the processing module. The integrated module can be implemented in hardware or as a software functional module. When the integrated module is implemented as a software functional module and sold or used as a standalone product, the integrated module can be stored in a computer-readable storage medium.

[0172] The aforementioned storage media can be read-only memory, disk, and CD, etc.

[0173] Although embodiments of the present disclosure have been shown and described, those skilled in the art will understand that these embodiments are illustrative and should not be construed as limiting the present disclosure, and that changes, modifications, substitutions and variations may be made to the embodiments without departing from the scope of the present disclosure.

Claims

1. A method of generating a target image, comprising: obtaining a captured blurred image captured by a camera assembly, the camera assembly comprising optics; applying a restoration process to the captured blurred image with a restoration filter to generate a restored sharp image, the restoration filter being a filter that restores sharpness of the captured blurred image based on optical characteristics of the optics; generating a regular fusion mask based on the captured blurred image; masking the restored sharp image with the regular fusion mask to generate a first intermediate image; generating an inverted fusion mask, wherein the regular fusion mask is inverted; applying a denoising process to the captured blurred image to reduce noise in the captured blurred image and generate a denoised image, wherein the denoising process is adjusted based on adjustment information that depends on filter characteristics of the restoration filter and / or optical characteristics of the optics; masking the denoised image with the inverted fusion mask to generate a second intermediate image; and combining the first intermediate image and the second intermediate image to generate the target image.

2. The method of claim 1, wherein, The adjustment information depends on a kernel size of the restoration filter.

3. The method of claim 1, wherein, The adjustment information depends on a filter strength of the restoration filter.

4. The method of claim 1, wherein, The adjustment information depends on frequency characteristics of the restoration filter.

5. The method of claim 1, wherein, The adjustment information depends on spatial characteristics of the optics.

6. The method of claim 1, wherein, The adjustment information depends on a point spread function (PSF) of the optics.

7. The method of claim 1, wherein, The denoising process uses a bilateral filter, the bilateral filter being expressed as: where q i,j values of denoising pixels in the filter kernel, i and j indicate an x position and a y position in the captured blurred image, respectively, and (i, j) indicates a center coordinate of the filter kernel, I(i, j) indicates an image block to be processed in the captured blurred image, Indicates element-wise multiplication. W s indicates a spatial weight, W i indicates an intensity weight, and H indicates a size of the filter kernel in the x direction, and V indicates a size of the filter kernel in the y direction.

8. The method of claim 7, wherein, The spatial weight W s is expressed as: where n indicates an x position in the filter kernel, m indicates a y position in the filter kernel, and the parameter indicates the intensity weight variance.

9. The method of claim 8, wherein, The intensity weight W i is expressed as: wherein, Parameter indicates spatial weight variance, power of two indicates element-wise multiplication, p i,j represents the value of the center pixel of the filter kernel.

10. The method of claim 9, wherein, the parameters and the parameters are adjusted based on the adjustment information, which depends on the filtering properties of the recovery filter used by the recovery process and / or the optical properties of the optics in the camera assembly.

11. The method of claim 10, wherein, the parameter gradually increases from a center region to a corner region in the captured blurred image gradually decreases from the center region to the corner region in the captured blurred image 12. The method of claim 1, wherein, Generating the inverted fusion mask comprises inverting the regular fusion mask.

13. The method of claim 1, wherein, Generating the regular fusion mask comprises: generating an average image by degrading the captured blurred image; subtracting the restored sharp image from the average image to generate a subtracted image; and computing an absolute value of the subtracted image to generate a difference image.

14. The method of claim 13, wherein, Generating the regular fusion mask comprises generating a threshold map based on the captured blurred image data, wherein in the threshold map each pixel indicates a threshold value to determine whether the pixel of the restored sharp image should be masked to generate the first intermediate image.

15. The method of claim 14, wherein, Generating the threshold map comprises: non-linearly modulating the captured blurred image to produce an interim threshold map; and multiplying the interim threshold map by a value to generate a final interim threshold map, the final interim threshold map being the threshold map.

16. The method of claim 15, wherein, The regular fusion mask is generated based on the difference image and the threshold map that are subjected to a thresholding process.

17. The method of claim 16, wherein, In the thresholding process, if the value of the pixel of the difference image is equal to or greater than the value of the corresponding pixel of the threshold map, the value of blend_mask of the pixel of the regular fusion mask is blend_ratio_high, the blend_ratio_high indicating a ratio of the pixel of the recovered sharp image in the first intermediate image, and if the value of the pixel of the difference image is less than the value of the corresponding pixel of the threshold map, the value of blend_mask of the pixel of the regular fusion mask is blend_ratio_low, the blend_ratio_low indicating a ratio of the pixel of the recovered sharp image in the first intermediate image, wherein the value of the blend_ratio_high is higher than the value of the blend_ratio_low.

18. The method of claim 17, wherein, the value of the blend_ratio_high is less than 100%.

19. The method of claim 18, wherein, masking the recovered sharp image with the regular fusion mask to generate the first intermediate image comprises generating the first intermediate image such that the pixel of the first intermediate image contains the blend_ratio_high of the corresponding pixel of the recovered sharp image and the blend_ratio_low of the corresponding pixel of the captured blurred image.

20. The method of claim 19, wherein, generating the regular fusion mask comprises: generating a lens shading model based on lens shading characteristics of the optics; and performing a pixel-wise multiplication of the difference image and the lens shading model to generate the regular fusion mask.

21. The method of claim 19, further comprising: generating a lens shading model based on lens shading characteristics of the optics; inverting the lens shading model to generate an inverted lens shading model; and performing a pixel-wise multiplication of the final temporary threshold map and the inverted lens shading model to modify the final temporary threshold map. the value of the blend_ratio_low increases when the pixel is located closer to a corner of the regular fusion mask during the thresholding process.

22. The method of claim 19, wherein, adding blur to the regular fusion mask to smooth a transition of a boundary between an area of the blend_ratio_high and an area of the blend_ratio_low of the regular fusion mask.

23. The method of claim 17, further comprising:

24. An electronic device, comprising: a camera assembly including optics; and a processor configured to: obtain a captured blurred image captured by a camera assembly, the camera assembly including optics; apply a recovery process to the captured blurred image with a recovery filter to generate a recovered sharp image, the recovery filter being a filter that restores sharpness of the captured blurred image based on optical characteristics of the optics; generate a regular fusion mask based on the captured blurred image; if the value of the pixel of the difference image is equal to or greater than the value of the corresponding pixel of the threshold map, the value of blend_mask of the pixel of the regular fusion mask is blend_ratio_high, the blend_ratio_high indicating a ratio of the pixel of the recovered sharp image in the first intermediate image, and if the value of the pixel of the difference image is less than the value of the corresponding pixel of the threshold map, the value of blend_mask of the pixel of the regular fusion mask is blend_ratio_low, the blend_ratio_low indicating a ratio of the pixel of the recovered sharp image in the first intermediate image, wherein the value of the blend_ratio_high is higher than the value of the blend_ratio_low. the value of the blend_ratio_high is less than 100%. masking the recovered sharp image with the regular fusion mask to generate the first intermediate image comprises generating the first intermediate image such that the pixel of the first intermediate image contains the blend_ratio_high of the corresponding pixel of the recovered sharp image and the blend_ratio_low of the corresponding pixel of the captured blurred image. generating the regular fusion mask comprises: generating a lens shading model based on lens shading characteristics of the optics; and performing a pixel-wise multiplication of the difference image and the lens shading model to generate the regular fusion mask.

21. The method of claim 19, further comprising: generating a lens shading model based on lens shading characteristics of the optics; inverting the lens shading model to generate an inverted lens shading model; and performing a pixel-wise multiplication of the final temporary threshold map and the inverted lens shading model to modify the final temporary threshold map. the value of the blend_ratio_low increases when the pixel is located closer to a corner of the regular fusion mask during the thresholding process. adding blur to the regular fusion mask to smooth a transition of a boundary between an area of the blend_ratio_high and an area of the blend_ratio_low of the regular fusion mask.

24. An electronic device, comprising: a camera assembly including optics; and a processor configured to: obtain a captured blurred image captured by a camera assembly, the camera assembly including optics; apply a recovery process to the captured blurred image with a recovery filter to generate a recovered sharp image, the recovery filter being a filter that restores sharpness of the captured blurred image based on optical characteristics of the optics; generate a regular fusion mask based on the captured blurred image; masking the recovered sharp image with the regular fusion mask to generate a first intermediate image; generating an inverted fusion mask, wherein the regular fusion mask is inverted; applying a denoising process to the captured blurry image to reduce noise in the captured blurry image and generate a denoised image, wherein the denoising process is adjusted based on adjustment information dependent on a filtering characteristic of the recovery filter and / or an optical characteristic of the optical device; masking the denoised image with the inverted fusion mask to generate a second intermediate image; and combining the first intermediate image and the second intermediate image to generate a target image.

25. A non-transitory computer readable medium storing program instructions, wherein, When the program instructions are executed by an electronic device, the program instructions cause the electronic device to perform at least the following operations: obtaining a captured blurry image captured by a camera assembly, the camera assembly comprising an optical device; applying a recovery process to the captured blurry image with a recovery filter to generate a recovered sharp image, the recovery filter being a filter that restores sharpness of the captured blurry image based on an optical characteristic of the optical device; generating a regular fusion mask based on the captured blurry image; masking the recovered sharp image with the regular fusion mask to generate a first intermediate image; generating an inverted fusion mask, wherein the regular fusion mask is inverted; applying a denoising process to the captured blurry image to reduce noise in the captured blurry image and generate a denoised image, wherein the denoising process is adjusted based on adjustment information dependent on a filtering characteristic of the recovery filter and / or an optical characteristic of the optical device; masking the denoised image with the inverted fusion mask to generate a second intermediate image; and combining the first intermediate image and the second intermediate image to generate a target image. When the program instructions are executed by an electronic device, the program instructions cause the electronic device to perform at least the following operations: obtaining a captured blurry image captured by a camera assembly, the camera assembly comprising an optical device; applying a recovery process to the captured blurry image with a recovery filter to generate a recovered sharp image, the recovery filter being a filter that restores sharpness of the captured blurry image based on an optical characteristic of the optical device; generating a regular fusion mask based on the captured blurry image; masking the recovered sharp image with the regular fusion mask to generate a first intermediate image; generating an inverted fusion mask, wherein the regular fusion mask is inverted; applying a denoising process to the captured blurry image to reduce noise in the captured blurry image and generate a denoised image, wherein the denoising process is adjusted based on adjustment information dependent on a filtering characteristic of the recovery filter and / or an optical characteristic of the optical device; masking the denoised image with the inverted fusion mask to generate a second intermediate image; and combining the first intermediate image and the second intermediate image to generate a target image.

Images