Method, electronic device, and non-transitory computer-readable medium for generating a target image

Abstract

A method of generating a target image includes obtaining a captured blurred image captured by a camera assembly including optics; applying a cyclic convolution to the captured blurred image with a restoration filter to generate a restored sharp image, the restoration filter being a filter that restores sharpness of the captured blurred image based on optical characteristics of the optics; generating a regular blending mask based on the captured blurred image; masking the restored sharp image with the regular blending mask to generate a first intermediate image; generating an inverse blending mask in which the regular blending mask is inverted; masking the captured blurred image with the inverse blending mask to generate a second intermediate image; and combining the first intermediate image and the second intermediate image to generate the target image.

Description

Technical Field

[0001] This disclosure relates to methods for generating target images, electronic devices, and non-transitory computer-readable media. Background Technology

[0002] Electronic devices such as smartphones and tablets are widely used in our daily lives. Many electronic devices today are equipped with camera components for capturing images. Some electronic devices are portable, making them easy to carry. Therefore, users of electronic devices can easily take pictures of objects anytime, anywhere by using the camera components of their devices.

[0003] When images are captured using camera components, image sharpness is degraded due to optical aberrations such as coma and astigmatism. Over the past decade, numerous academic papers have proposed techniques to improve the blurry images captured due to these optical aberrations. However, most of the published techniques are very complex or require high computational costs. Therefore, these techniques are impractical and unsuitable for implementation in electronic devices such as smartphones and tablets due to their insufficient computing power. Summary of the Invention

[0004] This disclosure aims to solve at least one of the aforementioned technical problems. Accordingly, this disclosure requires providing a method for generating a target image, an electronic device, and a non-transitory computer-readable medium.

[0005] According to this disclosure, a method for generating a target image may include:

[0006] Obtain a blurred image captured by a camera assembly including optics;

[0007] A restoration filter is used to apply a circular convolution to a captured blurred image to generate a restored sharp image. The restoration filter is a filter that restores the sharpness of a captured blurred image based on the optical properties of optical devices.

[0008] Rule-based blending masks are generated based on captured blurred images;

[0009] The restored sharp image is masked using a regular blending mask to generate a first intermediate image;

[0010] Generate an inverse blending mask, in which the regular blending mask is inverted;

[0011] The captured blurred image is masked using an inverse blending mask to generate a second intermediate image; and

[0012] Combine the first intermediate image and the second intermediate image to generate the target image.

[0013] According to this disclosure, the electronic device may include:

[0014] Camera components, including optics; and

[0015] The processor is configured as follows:

[0016] Obtain a blurred image captured by the camera component;

[0017] A restoration filter is used to apply a circular convolution to a captured blurred image to generate a restored sharp image. The restoration filter is a filter that restores the sharpness of a captured blurred image based on the optical properties of optical devices.

[0018] Rule-based blending masks are generated based on captured blurred images;

[0019] The restored sharp image is masked using a regular blending mask to generate a first intermediate image;

[0020] Generate an inverse blending mask, in which the regular blending mask is inverted;

[0021] The captured blurred image is masked using an inverse blending mask to generate a second intermediate image; and

[0022] Combine the first intermediate image and the second intermediate image to generate the target image.

[0023] According to this disclosure, a non-transitory computer-readable medium may include program instructions stored thereon, wherein, when executed by an electronic device, the program instructions cause the electronic device to perform at least the following operations:

[0024] Obtain a blurred image captured by a camera assembly including optics;

[0025] A restoration filter is used to apply a circular convolution to a captured blurred image to generate a restored sharp image. The restoration filter is a filter that restores the sharpness of a captured blurred image based on the optical properties of optical devices.

[0026] Rule-based blending masks are generated based on captured blurred images;

[0027] The restored sharp image is masked using a regular blending mask to generate a first intermediate image;

[0028] Generate an inverse blending mask, in which the regular blending mask is inverted;

[0029] The captured blurred image is masked using an inverse blending mask to generate a second intermediate image; and

[0030] Combine the first intermediate image and the second intermediate image to generate the target image. Attached Figure Description

[0031] These and / or other aspects and advantages of the embodiments of this disclosure will become apparent and more readily understood from the following description taken with reference to the accompanying drawings, in which:

[0032] Figure 1 This is a plan view of the first side of an electronic device according to an embodiment of the present disclosure;

[0033] Figure 2 This is a plan view of the second side of an electronic device according to an embodiment of the present disclosure;

[0034] Figure 3 This is a block diagram of an electronic device according to embodiments of the present disclosure;

[0035] Figure 4 This is an illustration of optical aberrations in optical devices;

[0036] Figure 5 The diagram illustrates a formula that indicates the relationship between a captured blurry image and an ideal sharp image;

[0037] Figure 6 This is a visual illustration of a method for improving the clarity of a captured blurry image in an electronic device according to embodiments of the present disclosure;

[0038] Figure 7 The diagram illustrates the cost function c(L), which includes a fidelity term and a regularization term.

[0039] Figure 8 This is a formula illustrating how to calculate the inverse filter based on the cost function c(L);

[0040] Figure 9 The image and inverse filter of blur kernel K are shown. Images in the spatial domain;

[0041] Figure 10 The diagram illustrates the problem of restoring a clear image;

[0042] Figure 11 The illustration outlines a hybridization process for an electronic device according to an embodiment of the present disclosure;

[0043] Figure 12 An example of how to generate a rule-based blending mask in an electronic device according to an embodiment of the present disclosure is illustrated;

[0044] Figure 13 The illustration shows an example of a look-up table (LUT) used for non-linear modulation of a captured blurred image;

[0045] Figure 14This is a visual illustration of a lens shading model (LSM) in an electronic device according to an embodiment of this disclosure.

[0046] Figure 15 This is a visual description of the first option for reflective light-blocking properties on a regular blending mask;

[0047] Figure 16 This is a visual description of the second option for reflecting light-blocking properties on a regular blending mask;

[0048] Figure 17 This is a visual description of the third option for reflective light-blocking properties on a regular blending mask;

[0049] Figure 18 It is a visual illustration of the smooth transition in the final blending mask;

[0050] Figure 19 This is a visual illustration of a method for generating a target image in an electronic device according to embodiments of the present disclosure;

[0051] Figure 20 This is a flowchart of a target image generation process in an electronic device according to embodiments of the present disclosure; and

[0052] Figure 21 The illustration shows a comparison between a target image generated by the prior art and a target image generated by an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0053] Embodiments of this disclosure will be described in detail, and examples of embodiments will be illustrated in the accompanying drawings. Identical or similar elements and elements having the same or similar functions are referred to by the same reference numerals throughout the specification. The embodiments described herein with reference to the accompanying drawings are illustrative and intended to illustrate this disclosure, but should not be construed as limiting the scope of this disclosure.

[0054] Figure 1 This is a plan view of the first side of an electronic device 10 according to an embodiment of the present disclosure, and Figure 2 This is a plan view of the second side of an electronic device 10 according to an embodiment of the present disclosure. The first side may be referred to as the rear side of the electronic device 10, and the second side may be referred to as the front side of the electronic device 10.

[0055] like Figure 1 and Figure 2As shown, the client device 10 may include a display 20 and a camera assembly 30. In this embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34, and a sub-camera 36. The first main camera 32 and the second main camera 34 can capture images from a first side of the electronic device 10, while the sub-camera 36 can capture images from a second side of the electronic device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called external cameras, while the sub-camera 36 is a so-called internal camera. As an example, the electronic device 10 may be a mobile phone, a tablet computer, a personal digital assistant, etc.

[0056] Each of the first main camera 32, the second main camera 34, and the sub-camera 36 has an image sensor that converts light that has passed through a color filter into an electrical signal. The signal value of the electrical signal depends on the amount of light that has passed through the color filter.

[0057] Although the electronic device 10 according to this embodiment has three cameras, the electronic device 10 may have fewer than three cameras or more than three cameras. For example, the electronic device 10 may have two, four, five, etc. cameras.

[0058] Figure 3 This is a block diagram of the electronic device 10 according to this embodiment; as shown Figure 3 As shown, in addition to the display 20 and camera assembly 30, the electronic device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46, and a communication circuit 48. The display 20, camera assembly 30, main processor 40, image signal processor 42, memory 44, power supply circuit 46, and communication circuit 48 are interconnected via a bus 50.

[0059] The main processor 40 executes one or more program instructions stored in the memory 44. The main processor 40 implements various application programs and data processing of the electronic device 10 by executing these program instructions. The main processor 40 can be one or more computer processors. The main processor 40 is not limited to a single CPU core, but can have multiple CPU cores. The main processor 40 can be the main CPU of the electronic device 10, an image processing unit (IPU), or a DSP provided together with the camera assembly 30.

[0060] The image signal processor 42 controls the camera assembly 30 and processes various image data captured by the camera assembly 30 to generate target image data. For example, the image signal processor 42 can apply de-mosaic processing, noise reduction processing, automatic exposure processing, automatic focus processing, automatic white balance processing, high dynamic range processing, etc., to the image data captured by the camera assembly 30.

[0061] In this embodiment, the main processor 40 and the image signal processor 42 cooperate with each other to generate target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture images of the object by means of the camera assembly 30 and apply various image processing techniques to the captured image data.

[0062] Memory 44 stores program instructions and various data to be executed by the main processor 40. For example, data of captured images is also stored in memory 44.

[0063] Memory 44 may include high-speed RAM and / or non-volatile memory such as flash memory and disk storage. That is, memory 44 may include non-transitory computer-readable media that stores program instructions.

[0064] The power supply circuit 46 may include a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.

[0065] Communication circuit 48 is configured to receive and transmit data to communicate wirelessly with base stations of telecommunications network systems, the Internet, or other devices. The wireless communication can adopt any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), Advanced LTE, and 5th Generation (5G). Communication circuit 48 may include an antenna and radio frequency (RF) circuitry.

[0066] Figure 4 This is an illustration of optical aberrations. Specifically, when an image is captured in the camera assembly 30, the image sharpness is degraded due to optical aberrations (such as coma, astigmatism, etc.) in the optics. Therefore, the point light source image is extended by optical aberrations in the optics, and the point light source image is no longer a point on the image plane.

[0067] Typically, the extended point light source is modeled by a function called the Point Spread Function (PSF), which represents the way the captured image is degraded and its degradation characteristics. This is known as optical blur. Subsequently, the blurred image captured by camera component 30 is also referred to as the captured blurred image.

[0068] Figure 5The diagram illustrates a formula indicating the relationship between the captured blurred image B and the ideal sharp image L. (See diagram for example.) Figure 5 As shown, the sharpness of the captured blurred image B is represented by K*L+n. K indicates the same blur kernel as PSF. L indicates an ideal sharp image without noise (i.e., ideal). n indicates noise. "*" indicates circular convolution. This formula shows that images captured by camera assembly 30 always include noise. This noise is also called shot noise because noise is unavoidable when capturing images.

[0069] Figure 6 This is a visual illustration of a method for improving the sharpness of a captured blurred image in an electronic device 10 according to an embodiment of the present disclosure. In this embodiment, the electronic device 10 obtains a restored sharp image L_res via a filtering restoration process using an inverse filter calculated based on the PSF. The restored sharp image L_res is closer to the ideal sharp image L. However, noise in the captured blurred image B is also filtered by the filtering restoration process, and thus the noise is also increased. The inverse filter is one example of a restoration filter, which is a filter that restores the sharpness of a captured blurred image based on the optical characteristics of an optics device.

[0070] Next, we will explain how to calculate the inverse filter based on the fuzzy kernel K, which is the SPF. Figure 7 The cost function c(L) in this embodiment is illustrated. For example... Figure 7 As shown, the cost function c(L) is essentially expressed by the minimum independent variable of "captured blurred image B - blurred kernel K * ideal sharp image L". That is, this term is the fidelity term. Furthermore, Figure 7 The formula shown introduces a regularization term into the cost function c(L) to apply a penalty. That is, by introducing a regularization term into the cost function c(L), overfitting can be avoided. As a result, this regularization term can be suppressed to increase the noise in the recovered sharp image L_res. In this paper, D indicates the regularization function, and ρ indicates the regularization gain.

[0071] Figure 8 This is a formula illustrating how to calculate the inverse filter based on the cost function c(L). It can be solved in the frequency domain. Figure 7 The cost function c(L) is obtained in Figure 8 The Fourier transform F(L) in the image is used. The recovered sharp image L_res can be obtained by performing an inverse Fourier transform F(L). This is because an inverse filter can be used... *The captured blurred image B yields the recovered sharp image L_res, obtained through an inverse filter. This can be specified in this formula.

[0072] Figure 9 The image and inverse filter of blur kernel K are shown. Image in the spatial domain. Inverse filter K -1 The reason for adding the cap is that the blur kernel K is not an actual measured value, but a specified value. Due to assembly and dimensional errors of the components of the camera assembly 30 and electrical equipment 10, the specified value of the blur kernel K is not equal to the actual measured value of the blur kernel K. Therefore, the inverse filter... The actual inverse filter K of electrical equipment 10 -1 Slightly different.

[0073] Obtaining the actual measured value of the fuzzy kernel K is a very complex and tedious process for the electronic device 10 and its users. Furthermore, the inverse filter... The filter is sufficiently fine to recover the captured blurred image B. Therefore, according to this embodiment, the electronic device 10 calculates an inverse filter based on a specified value of the blur kernel K.

[0074] Figure 10 The illustration shows the problem of restoring a clear image. For example... Figure 10 As shown, the captured blurred image contains noise. As a result, if an inverse filter is used... Applying a circular convolution to a captured blurred image increases noise in the captured blurred image, but improves the sharpness of the recovered clear image.

[0075] Therefore, in order to solve this problem, the electronic device 10 according to an embodiment of the present disclosure introduces a mixing process for mixing the clear image recovered by the filtering recovery process and the blurry image of the original capture that was not applied to the filtering recovery process.

[0076] Figure 11 The illustration outlines a mixing process of an electronic device 10 according to an embodiment of the present disclosure. During the mixing process, an inverse filter is utilized. A circular convolution is applied to the captured blurred image to generate a restored sharp image L_res.

[0077] Even if noise is contained in the high-frequency region of an image, the human eye cannot detect it. Conversely, if noise is contained in the low-frequency region of an image, it can be detected by the human eye and is easily noticed.

[0078] Therefore, in the electronic device 10 according to this embodiment, a regular blending mask is used to mask the low-frequency regions in the recovered clear image to generate a first intermediate image. On the other hand, an inverse blending mask is used to mask the captured blurred image to generate a second intermediate image. This inverse blending mask is the inverse of the regular blending mask. For example, the inverse blending mask can be obtained by inverting the regular blending mask. Thereafter, the electronic device 10 according to this embodiment combines the first intermediate image and the second intermediate image to generate a target image.

[0079] This blending process generates the target image by replacing noisy low-frequency regions in the recovered sharp image with noise-free low-frequency regions in the captured blurred image. Therefore, noise is not present in the low-frequency regions of the target image. On the other hand, the sharpness of high-frequency regions in the target image is improved. That is, the texture in the high-frequency regions of the target image can be fine, thus increasing user satisfaction with the target image.

[0080] Figure 12 An example of how a rule-based blending mask is generated in an electronic device 10 according to an embodiment of the present disclosure is illustrated. Figure 12 As shown, a rule-based mixing mask is generated based on the captured blurred image. More specifically, the electronic device 10 obtains the captured blurred image, for example, from the camera assembly 30 or the image signal processor 42. The electronic device 10 then applies a circular convolution to the captured blurred image using an inverse filter to improve the sharpness of the captured blurred image and generate a restored sharp image. However, if the restored sharp image has already been generated in another process, that process can be omitted.

[0081] In addition, the electronic device 10 performs averaging processing on the captured blurry image to degrade the sharpness of the captured blurry image and generate an average image.

[0082] Then, the electronic device 10 subtracts the recovered sharp image from the average image to generate a subtracted image, and then calculates the absolute value of the subtracted image to generate a difference image. The difference image obtained through these processes represents the degree to which the filtering and restoration process of the inverse filter changes the pixel brightness. In other words, each pixel in the difference image has a specific value indicating the level of brightness change caused by the filtering and restoration process.

[0083] exist Figure 12 In the diagram, gray areas indicate regions where brightness has been significantly altered by the inverse filter's filtering and recovery process. Conversely, black areas indicate regions where brightness has not been significantly altered by the inverse filter's filtering and recovery process.

[0084] On the other hand, the electronic device 10 non-linearly modulates the captured blurred image to generate a temporary threshold mapping. In this embodiment, a lookup table (LUT) is used for non-linearly modulating the captured blurred image.

[0085] Figure 13 The illustration shows an example of a lookup table (LUT) used for non-linearly modulating a captured blurred image. In this example, if the pixel values ​​of the captured blurred image are low, the pixel values ​​of the temporary thresholding map are increased from their original values. Conversely, if the pixel values ​​of the captured blurred image are in the middle range, the pixel values ​​of the temporary thresholding map are decreased from their original values. Furthermore, if the pixel values ​​of the captured blurred image are greater than a certain value, the upper limit of the pixel values ​​of the temporary thresholding map is set to that certain value.

[0086] Figure 13 This is one example of a lookup table LUT, which is not limited to... Figure 13 The example shown is illustrated. Furthermore, methods for non-linearly modulating a captured blurred image are not limited to using a lookup table (LUT). Various other methods can be applied to non-linearly modulate a captured blurred image.

[0087] Next, as Figure 12 As shown, the electronic device 10 according to an embodiment of this disclosure multiplies a temporary threshold map by a specific value to adjust the gain of the temporary threshold map, and then can calculate a final threshold map. This specific value can be less than one or greater than one. In this final threshold map, each pixel indicates a threshold regarding whether pixels of the recovered sharp image should be masked when generating the first intermediate image.

[0088] More specifically, the electronic device 10 generates a regular blending mask based on the difference image and the final threshold map through a thresholding process. In the thresholding process, if the value of `diff_img`, which is the pixel value of the difference image, is equal to or greater than the value of `th_map`, which is the pixel value of the final threshold map, then the value of `blend_mask`, which is the pixel value of the regular blending mask, is `blend_ratio_high`. For example, `blend_ratio_high` is 95%.

[0089] On the other hand, if the value of diff_img is less than the value of th_map, then the value of blend_mask is blend_ratio_low. For example, blend_ratio_low is 5%.

[0090] In this embodiment, when using a regular blending mask, if the blend_mask of the pixels of the regular blending mask is blend_ratio_high (95%), a first intermediate image is generated such that the pixels of the first intermediate image contain 95% of the corresponding pixels of the restored clear image and 5% of the corresponding pixels of the captured blurred image.

[0091] On the other hand, if the blend_mask of the pixels of the regular blending mask is blend_ratio_low (5%), a first intermediate image is generated such that the pixels of the first intermediate image contain 5% of the corresponding pixels of the restored sharp image and 95% of the corresponding pixels of the captured blurred image. In this embodiment, blend_ratio_high is added to blend_ratio_low, which should be 1.

[0092] The values ​​of the restored sharp image and the captured blurry image are mixed in the same pixel, making it impossible for the human eye to distinguish the boundary between the areas using the restored sharp image and the areas using the captured blurry image in the target image.

[0093] Thresholding eliminates small grayscale areas of noise, such as those found in the difference image. When restoring the sharpness of the captured blurred image, bright areas in the restored sharp image contain noise. Therefore, in this embodiment, the final thresholded value for bright areas is high. As a result, the blending ratio of bright areas in the regular blending mask becomes lower, and the bright areas in the restored sharp image are masked by the regular blending mask.

[0094] An inverse blending mask can be generated by inverting a regular blending mask. That is, by inverting the regular blending mask, the blend_mask of the pixels with blend_ratio_high (95%) in the regular blending mask is converted into the blend_mask of the pixels with blend_ratio_low (5%) in the inverse blending mask, and the blend_mask of the pixels with blend_ratio_low (5%) in the regular blending mask is converted into the blend_mask of the pixels with blend_ratio_high (95%) in the inverse blending mask.

[0095] Incidentally, in the example mentioned above, `blend_ratio_high` is less than 100%, and `blend_ratio_low` is greater than 0%. However, in a regular blending mask, `blend_ratio_high` can be 100%, and `blend_ratio_low` can be 0%. In this case, in an inverse blending mask, `blend_ratio_high` is also 100%, and `blend_ratio_low` is also 0%. However, the value of `blend_ratio_high` is higher than the value of `blend_ratio_low`.

[0096] Due to the characteristics of the optics in camera assembly 30, the brightness is reduced around the corners of the captured image. That is, the amount of light in the corners is low due to lens vignetting (also known as vignetting). Therefore, in some cases, the captured blurred image has been compensated to correct for the vignetting characteristics. To compensate, the electronic device 10 multiplies the captured image by an appropriate gain. However, noise characteristics depend primarily on this compensation. Therefore, noise characteristics are taken into account when generating a regular blending mask.

[0097] Optionally, in the electronic device 10 according to embodiments of the present disclosure, for example, a lens shading model (LSM) is introduced. Figure 14 This is a visual description of the lens shading model LSM in this embodiment.

[0098] like Figure 14 As shown, the captured image, following the original lens shading characteristics of the optics, has a brighter area in the center and a darker area in the corners. In this embodiment, a lens shading model (LSM) is generated based on the original lens shading characteristics. That is, the LSM is generated through a nonlinear transformation of the original lens characteristics. The purpose of the nonlinear transformation is to adjust the brightness based on the LSM to obtain a regular blending mask suitable for masking noisy areas.

[0099] <First Option>

[0100] Figure 15 This is a visual description of the first option for reflective shading properties on a regular blending mask. In the first option, the electronic device 10 performs pixel-wise multiplication of the differential image and the lens shading model LSM.

[0101] By correcting the shading characteristics of the captured image, the brightness around the corners of the captured image has been increased. That is, the brightness around the corners of the captured blurred image has also been increased. Therefore, according to the first option, the brightness around the corners of the difference image is reduced by using the Lens Shading Model (LSM). As a result, the correction effect of the shading characteristics of the regular blending mask can be compensated.

[0102] <Second Option>

[0103] Figure 16 This is a visual illustration of the second option for reflective shading properties on a regular blending mask. In this second option, the electronic device 10 performs pixel-by-pixel multiplication of the final threshold mapping and the inverse lens shading model to modify the final threshold mapping. The inverse lens shading model is computed by inverting the lens shading model LSM. Therefore, the area around the center of the inverse lens shading model is dark, while the area around the corners of the inverse lens shading model is bright.

[0104] By correcting the occlusion characteristics of the captured image, the amplitude of noise around the corners of the captured image has been increased. Therefore, by increasing the value of the corner in the final threshold mapping, the corner region of the regular blend mask can be easily set to blend_ratio_low and masked using the regular blend mask.

[0105] <Third Option>

[0106] Figure 17 This is a visual description of the third option for the reflective shading properties on the regular blending mask. In this third option, the thresholding process mentioned above is modified. Specifically, the value of `blend_mask` is calculated by multiplying `blend_ratio_high` by the lens shading model `LSM(i,j)` or by multiplying `blend_ratio_low` by the inverse lens shading model. As a result of this calculation, the ratio of the recovered sharp image increases around the center of the regular blending mask. In other words, the ratio of the recovered sharp image decreases at the corners of the regular blending mask.

[0107] By correcting the occlusion characteristics of the captured image, a significant amount of noise has already been introduced into the corner regions of the captured image. Therefore, to eliminate noise in the corner regions, the rate of recovering a sharp image decreases when pixels are located closer to the corner of the regular blending mask. In other words, the rate of capturing a blurred image increases when pixels are located closer to the corner of the regular blending mask. Therefore, when generating the target image, noise in the corners of the recovered sharp image can be suppressed.

[0108] Alternatively, in the electronic device 10 according to an embodiment of the present disclosure, the transition in the final blending mask can be smoothed. Figure 18 It is a visual illustration of the smooth transition in the final blending mask.

[0109] like Figure 18As shown, the final blend mask has two values: blend_ratio_high and blend_ratio_low. Therefore, the boundary between the regions of blend_ratio_high and blend_ratio_low is very clear and sharp. When using a clear and sharp final blend mask to generate the target image, the target image may appear unnatural to the human eye.

[0110] Therefore, the electronic device 10 according to embodiments of the present disclosure can add blur to the final blending mask to naturally blend regions of blend_ratio_high and blend_ratio_low. For example, the electronic device 10 adds blur to the final blending mask by performing a simple convolution of a Gaussian blur kernel.

[0111] A blurred blending mask is generated by introducing blurring into the final blending mask. In the blurred blending mask, the boundary between the regions of blend_ratio_high and blend_ratio_low is blurred. In other words, the pixel values ​​in the boundary regions have transitional values ​​between blend_ratio_low and blend_ratio_high. That is, in the boundary regions, the value of blend_mask gradually increases from blend_ratio_low to blend_ratio_high.

[0112] Figure 19 It is a visual description of the method used to generate the target image. For example... Figure 19 As shown, and as based on Figure 11 As briefly explained, the target image is generated by masking the recovered sharp image using a regular blending mask, masking the captured blurred image using an inverse blending mask, and combining them.

[0113] More specifically, a first intermediate image can be generated by performing pixel-wise multiplication of the restored sharp image and a regular blending mask. A second intermediate image can be generated by performing pixel-wise multiplication of the captured blurred image and an inverse blending mask. That is, masking processing can be implemented through pixel-wise multiplication. Therefore, the target image can be generated by combining the first and second intermediate images.

[0114] Figure 20 This is a flowchart of a target image generation process in an electronic device 10 according to an embodiment of the present disclosure. The target image generation process may be executed by a main processor 40 or an image signal processor 42. Alternatively, the target image generation process may be executed by a combination of the main processor 40 and the image signal processor 42.

[0115] In this embodiment, for example, the main processor 40 obtains the captured blurred image from the output port of the image signal processor 42. Then, the main processor 40 performs a target image generation process on the captured blurred image and inputs the generated target image to the input port of the image signal processor 42.

[0116] Furthermore, the program instructions for implementing the target image generation process can be stored on a non-transitory computer-readable medium. The main processor 40 reads the program instructions from the non-transitory computer-readable medium and executes the program instructions to implement the target image generation process.

[0117] Furthermore, the target image generation process can be performed on the luminance plane of the YUV standard. That is, in the electronic device 10 according to an embodiment of this disclosure, the luminance plane (Y plane) can undergo the target image generation process. Of course, other planes or images can undergo the target image generation process disclosed herein.

[0118] like Figure 20 As shown, the main processor 40 of the electronic device 10 obtains, for example, the blurred image captured by the camera assembly 30 including optics from the image signal processor 42 (step S10).

[0119] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 applies a circular convolution to the captured blurred image using an inverse filter to generate a restored sharp image (step S12). The details of this process have already been explained above.

[0120] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 generates a rule-based blending mask based on the captured blurred image (step S14). The details of this process have already been explained above.

[0121] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 uses a regular blending mask to mask the recovered clear image to generate a first intermediate image (step S16). The details of this process have already been explained above.

[0122] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 generates an inverse blending mask, wherein the regular blending mask is inverted (step S18). The details of this process have already been explained above.

[0123] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 uses an inverse blending mask to mask the captured blurred image to generate a second intermediate image (step S20). The details of this process have already been explained above.

[0124] Next, as Figure 20 As shown, the main processor 40 of the electronic device 10 combines the first intermediate image and the second intermediate image to generate the target image (step S22). The details of this process have been explained above. After completing step S22, the target image generation process according to this embodiment ends.

[0125] Figure 21 The illustration shows a comparison between a target image generated by prior art and a target image generated by an electronic device 10 according to an embodiment of the present disclosure. Figure 21 As shown, in the prior art, the target image contains noise due to the filtering and restoration process used to improve sharpness, or is blurred because it has not undergone a filtering and restoration process.

[0126] On the other hand, in the target image generated by the electronic device 10 according to this embodiment, noise caused by the filtering restoration process used to improve the sharpness of the captured blurry image is eliminated by masking the noise area of ​​the restored clear image and replacing the captured blurry image.

[0127] On the other hand, the texture clarity of the target image is improved by a filtering restoration process that does not mask the fine texture regions of the restored sharp image when generating the target image. As a result, a more natural and clearer target image can be obtained for the user without increasing costs.

[0128] Similarly, the electronic device 10 according to this embodiment can simplify the optics of the camera assembly 30 and reduce the number of lens elements in the optics of the camera assembly 30. At the same time, natural and high-quality target images can be obtained without the need for large and expensive optics in the camera assembly 30.

[0129] In the description of embodiments of this disclosure, it should be understood that terms such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” should be interpreted as referring to the direction or position as described or shown in the discussed drawings. These related terms are used only to simplify the description of this disclosure and do not indicate or imply that the mentioned devices or elements must have a particular orientation, or be constructed or operated in a particular orientation. Therefore, these terms should not be construed as limiting this disclosure.

[0130] Furthermore, the use of terms such as “first” and “second” herein is for descriptive purposes and is not intended to indicate or imply relative importance or significance, nor is it intended to imply the number of technical features indicated. Therefore, a feature defined by “first” and “second” may include one or more of those features. In the description of this disclosure, “multiple” means two or more, unless otherwise stated.

[0131] In the description of embodiments of this disclosure, unless otherwise specified or limited, the terms “mounted,” “connected,” “coupled,” etc. are used extensively and can be, for example, a fixed connection, a detachable connection, or an integral connection; it can also be a mechanical or electronic connection; it can also be a direct or indirect connection via an intermediate structure; or it can be internal communication between two elements, which can be understood by those skilled in the art based on the specific circumstances.

[0132] In embodiments of this disclosure, unless otherwise specified or limited, a structure where the first feature is "on" or "under" the second feature may include embodiments where the first feature and the second feature are in direct contact, and may also include embodiments where the first feature and the second feature are not in direct contact with each other, but rather contact each other via an additional feature formed between them. Furthermore, "on," "above," or "top" the second feature may include embodiments where the first feature is directly opposite or tilted "on," "above," or "top" the second feature, or simply means that the first feature is at a height higher than the second feature; while "below," "under," or "under" the second feature may include embodiments where the first feature is directly opposite or tilted "below," "under," or "under" the second feature, or simply means that the first feature is at a height lower than the second feature.

[0133] Various embodiments and examples have been provided in the foregoing description to implement different structures of this disclosure. To simplify this disclosure, certain elements and arrangements have been described above. However, these elements and arrangements are merely examples and are not intended to limit this disclosure. Furthermore, in different examples of this disclosure, numbers and letters of the reference numerals may be repeated. This repetition is for simplification and clarity, and does not indicate a relationship between different embodiments and / or arrangements. In addition, examples of different processes and materials are provided in this disclosure. However, those skilled in the art will understand that other processes and / or materials may also be applied.

[0134] Throughout this specification, references to "an embodiment," "some embodiments," "an exemplary embodiment," "an example," "a specific example," or "some examples" mean that a particular feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this disclosure. Therefore, the appearance of these phrases throughout this specification does not necessarily refer to the same embodiment or example of this disclosure. Furthermore, in one or more embodiments or examples, a particular feature, structure, material, or characteristic may be combined in any suitable manner.

[0135] Any process or method described in the flowchart or otherwise herein can be understood as including one or more modules, segments, or portions of code comprising executable instructions for implementing a particular logical function or step in the process, and the scope of the preferred embodiments of this disclosure includes other implementations, wherein those skilled in the art will understand that functionality may be implemented in a sequence different from the sequence shown or discussed (including in substantially the same or opposite sequences).

[0136] The logic and / or steps otherwise described herein or shown in the flowcharts, such as a specific list of executable instructions for implementing logical functions, may be embodied in any computer-readable medium that will be used by or in conjunction with an instruction execution system, device, or apparatus (such as a computer-based system including a processor or other system capable of obtaining and executing instructions from and with the instruction execution system, device, or apparatus). For the purposes of this specification, "computer-readable medium" can be any device that adaptively includes, stores, communicates, propagates, or transmits a program that will be used by or in conjunction with an instruction execution system, device, or apparatus. More specific examples of computer-readable media include, but are not limited to: electronic connections (electronic devices) having one or more wires, portable computer casings (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because, for example, when a program needs to be obtained electronically, it can be optically scanned on paper or other suitable media, then edited, decrypted or processed by other suitable methods, and then the program can be stored in computer memory.

[0137] It should be understood that each part of this disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, similarly in another embodiment, the steps or methods can be implemented by one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing data signals, application-specific integrated circuits (ASICs) having suitable combinations of logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0138] Those skilled in the art will understand that all or part of the steps in the exemplary methods described above can be implemented by using program commands to related hardware. These programs can be stored in a computer-readable storage medium, and when run on a computer, they include one or a combination of the steps in the method embodiments of this disclosure.

[0139] Furthermore, each functional unit in the embodiments of this disclosure can be integrated into a processing module, or these units can be separate physical entities, or two or more units can be integrated into a processing module. The integrated module can be implemented in hardware or as a software functional module. When the integrated module is implemented as a software functional module and sold or used as a standalone product, the integrated module can be stored in a computer-readable storage medium.

[0140] The aforementioned storage media can be read-only memory, disk, CD, etc.

[0141] Although embodiments of the present disclosure have been shown and described, those skilled in the art will understand that these embodiments are illustrative and should not be construed as limiting the present disclosure, and that changes, modifications, substitutions and variations may be made in the embodiments without departing from the scope of the present disclosure.

Claims

1. A method for generating a target image, comprising: Obtain a blurred image captured by a camera assembly including optics; A restored sharp image is generated by applying a circular convolution to the captured blurred image using a restoration filter, wherein the restoration filter is a filter that restores the sharpness of the captured blurred image based on the optical properties of the optical device; the optical properties of the optical device are the point spread function (PSF). A rule-based blending mask is generated based on the captured blurred image; wherein the rule-based blending mask is generated by thresholding based on a difference image and a threshold mapping, or by a difference image and a lens shading model; the difference image is generated by performing an averaging process on the captured blurred image to generate an average image, subtracting the recovered sharp image from the average image to generate a subtracted image, and calculating the absolute value of the subtracted image. The restored clear image is masked using the rule-based mixing mask to generate a first intermediate image; Generate an inverse mixing mask, in which the regular mixing mask is inverted; The captured blurred image is masked using the inverse blending mask to generate a second intermediate image; and The first intermediate image and the second intermediate image are combined to generate the target image.

2. The method of claim 1, wherein, The generation of the rule-based mixing mask includes: The threshold map is generated based on the captured blurred image data, wherein each pixel in the threshold map indicates a threshold to determine whether the pixels of the recovered sharp image should be masked to generate the first intermediate image.

3. The method of claim 2, wherein, The generation of the threshold mapping includes: The captured blurred image is non-linearly modulated to generate a temporary threshold mapping; and The temporary threshold mapping is multiplied by a specific value to generate a final threshold mapping, which is the threshold mapping itself.

4. The method of claim 3, wherein, In the thresholding process, If the pixel value of the difference image is equal to or greater than the corresponding pixel value of the threshold mapping, then the pixel value of the regular blending mask is blend_ratio_high, where blend_ratio_high indicates the ratio of the pixels in the recovered sharp image in the first intermediate image, and If the pixel value of the difference image is less than the corresponding pixel value of the threshold mapping, then the pixel value of the regular blending mask is blend_ratio_low, where blend_ratio_low also indicates the ratio of the pixels in the recovered sharp image in the first intermediate image. The value of blend_ratio_high is higher than the value of blend_ratio_low.

5. The method of claim 4, wherein, The value of blend_ratio_high is less than 100%.

6. The method of claim 5, wherein, The step of masking the recovered clear image using the rule-based mixing mask to generate the first intermediate image includes: Generate the first intermediate image such that the pixels of the first intermediate image contain the blend_ratio_high value of the corresponding pixel of the restored clear image and the 1-blend_ratio_high value of the corresponding pixel of the captured blurred image; or, make the pixels of the first intermediate image contain the blend_ratio_low value of the corresponding pixel of the restored clear image and the 1-blend_ratio_low value of the corresponding pixel of the captured blurred image.

7. The method of claim 1, wherein, The generation of the rule-based mixing mask includes: The lens shading model is generated based on the lens shading characteristics of the optical device; and Perform pixel-wise multiplication of the difference image and the lens occlusion model to generate the regular blending mask.

8. The method according to claim 6, further comprising: The lens shading model is generated based on the lens shading characteristics of the optical device. Inverse the lens shading model to generate the inverse lens shading model; as well as Perform pixel-by-pixel multiplication of the final threshold mapping and the reverse lens shading model to modify the final threshold mapping.

9. The method of claim 6, wherein, In the thresholding process, the value of blend_ratio_low increases when the pixel is located closer to the corner of the regular blending mask.

10. The method of claim 4, further comprising adding a blur to the regular blending mask to smooth the transition of the boundary between the blend_ratio_high region and the blend_ratio_low region of the regular blending mask.

11. An electronic device, comprising: Camera components, including optics; as well as The processor is configured as follows: Obtain a blurred image captured by the camera component; A restored sharp image is generated by applying a circular convolution to the captured blurred image using a restoration filter, wherein the restoration filter is a filter that restores the sharpness of the captured blurred image based on the optical properties of the optical device; the optical properties of the optical device are the point spread function (PSF). A rule-based blending mask is generated based on the captured blurred image; wherein the rule-based blending mask is generated by thresholding based on a difference image and a threshold mapping, or by a difference image and a lens shading model; the difference image is generated by performing an averaging process on the captured blurred image to generate an average image, subtracting the recovered sharp image from the average image to generate a subtracted image, and calculating the absolute value of the subtracted image. The restored clear image is masked using the rule-based mixing mask to generate a first intermediate image; Generate an inverse mixing mask, in which the regular mixing mask is inverted; The captured blurred image is masked using the inverse blending mask to generate a second intermediate image; and The first intermediate image and the second intermediate image are combined to generate the target image.

12. A non-transitory computer readable medium comprising program instructions stored thereon, wherein, When the program instructions are executed by the electronic device, the program instructions cause the electronic device to perform at least the following operations: Obtain a blurred image captured by a camera assembly including optics; A restored sharp image is generated by applying a circular convolution to the captured blurred image using a restoration filter, wherein the restoration filter is a filter that restores the sharpness of the captured blurred image based on the optical properties of the optical device; the optical properties of the optical device are the point spread function (PSF). A rule-based blending mask is generated based on the captured blurred image; wherein the rule-based blending mask is generated by thresholding based on a difference image and a threshold mapping, or by a difference image and a lens shading model; the difference image is generated by performing an averaging process on the captured blurred image to generate an average image, subtracting the recovered sharp image from the average image to generate a subtracted image, and calculating the absolute value of the subtracted image. The restored clear image is masked using the rule-based mixing mask to generate a first intermediate image; Generate an inverse mixing mask, in which the regular mixing mask is inverted; The captured blurred image is masked using the inverse blending mask to generate a second intermediate image; and The first intermediate image and the second intermediate image are combined to generate the target image.

Images