Processing Image Data in a Synthetic Image

Abstract

A method for processing image data in a synthetic image is described. The method includes: analyzing a multi-level mixing of the image of the synthetic image; identifying a sharp transition boundary region of the synthetic image; applying less attenuation to a higher level of the multi-level mixing of the synthetic image in the sharp transition boundary region; and applying more attenuation to a lower level of the multi-level mixing of the synthetic image in the sharp transition boundary region.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to devices having multiple cameras, and more particularly, to devices and methods for processing image data in a composite image. Background Art

[0002] Digital cameras are electronic devices that capture images stored in digital format. Other electronic devices, such as smart phones, tablets, or other portable devices, are typically equipped with cameras to enable them to capture images in digital format. As the demand for improved functionality of cameras or electronic devices having cameras has increased, multiple cameras with different functions have been implemented in electronic devices. According to some implementations, a dual camera module in an electronic device may include two different lenses / sensors, or may have different camera characteristics, such as exposure settings.

[0003] Multiple scenes captured by a device having a digital camera include a very large dynamic range (i.e., there are very bright and very dark regions in the same scene). It is difficult for a camera to record a scene using a single exposure, and thus multiple exposures are typically used, where a high dynamic range (HDR) algorithm is used to combine image frames. These algorithms can select portions of each image frame that are optimally exposed during the multiple exposures. Blending is used to synthesize these portions together. In blending, multiple image frames are combined together using corresponding blending weights to control how each frame should contribute to the final result. A particular artifact called a halo artifact typically appears around the transition point between the bright and dark regions of a scene.

[0004] Therefore, devices and methods for blending images that reduce transition artifacts, such as halo artifacts, are beneficial. Summary of the Invention

[0005] Technical Solution for Solving the Problem

[0006] A method for processing image data in a composite image is described. The method includes: analyzing multi-level blending of an image of the composite image; identifying a sharp transition boundary region of the composite image; applying less attenuation to a higher level of the multi-level blending of the composite image in the sharp transition boundary region; and applying more attenuation to a lower level of the multi-level blending of the composite image in the sharp transition boundary region.

[0007] Also described is a device for enhancing image quality. The device includes a processor circuit configured to: analyze a multi-level blending of an image of a synthetic image; identify a sharp transition boundary region of the synthetic image; apply less attenuation to a higher level of the multi-level blending of the synthetic image in the sharp transition boundary region; and apply more attenuation to a lower level of the multi-level blending of the synthetic image in the sharp transition boundary region.

[0008] A non-transitory computer-readable storage medium storing data representing software executable by a computer to enhance image quality, the non-transitory computer-readable storage medium including: analyzing a multi-level blending of an image of a synthetic image; identifying a sharp transition boundary region of the synthetic image; applying less attenuation to a higher level of the multi-level blending of the synthetic image in the sharp transition boundary region; and applying more attenuation to a lower level of the multi-level blending of the synthetic image in the sharp transition boundary region. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an exemplary electronic device having multiple cameras;

[0010] Figure 2 is a diagram showing different levels associated with multiple images;

[0011] Figure 3 is a diagram showing blending using weights of different levels associated with multiple images;

[0012] Figure 4 is a block diagram for generating a blended output image based on two input images;

[0013] Figure 5 is showing for Figure 4 a graph of applying an attenuation signal to two input images;

[0014] Figure 6 shows an example of images being blended to form an output image with a reduced halo effect; and

[0015] Figure 7 is a flowchart showing a method of processing image data in a synthetic image. DETAILED DESCRIPTION

[0016] The devices and methods described below detect transition boundaries associated with scenarios where blending control can quickly transition from a dark region to a bright region. After detecting a transition boundary, the devices and methods provide control of the blend to suppress halation artifacts. The devices and methods can detect where halation artifacts will appear and can control the blend to prevent the blended result from exhibiting halation artifacts. For example, the devices and methods can look for large changes in blend weights, where regions with large changes can mark sharp transition boundaries. The circuits and methods are beneficial when used with images having different exposures or for blending an image generated using a flash with an image generated without using a flash.

[0017] According to some implementations, the devices and methods provide an extension to pyramid blending, where detection can be applied at multiple levels of the blending pyramid. Fine detail is recorded at the high-resolution levels of the blending pyramid. Less control is applied at these levels to avoid blurring the fine detail in the output image. Large-scale transitions are recorded at the lower levels, so detection and more attenuation (also referred to as suppression) can be applied to the regions detected at the lower levels to control halation artifacts. Thus, the devices and methods allow for halation attenuation to be performed in the pyramid without losing the fine detail in the output image. Attenuation can be applied at each level using a controllable threshold. The detail level in the low-resolution levels can be reduced at points where there are rapid transitions in the blend weights. The control adaptation can be scene-content based. The devices and methods allow for the detection of halation regions and control of how much high-pass energy is passed to the output to avoid halation overshoot and undershoot.

[0018] Furthermore, while this specification includes claims that define features that are regarded as novel for one or more implementations of the invention, it is believed that the circuits and methods will be better understood from a consideration of the description in conjunction with the accompanying drawings. While various circuits and methods are disclosed, it should be understood that the circuits and methods are merely examples of the arrangements of the invention, which can be implemented in various forms. Thus, the specific structural and functional details disclosed in this specification should not be construed as limiting, but rather as a basis for the claims and as a representative basis for teaching one of ordinary skill in the art to employ the inventive arrangements in any appropriate detailed structure in various ways. Additionally, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of the circuits and methods.

[0019] First, turn to Figure 1, which shows a block diagram of an electronic device having multiple cameras. The electronic device (e.g., mobile device 100) can be any type of device having one or more cameras. The mobile device 100 can include a processor 102 coupled to multiple cameras 104 and 105. The cameras 104 and 105 can have different characteristics (such as different focal lengths or different operating characteristics (such as different exposure times or use of different flashlights)), where the blending of images can involve images with different characteristics as described in more detail below. The mobile device 100 can be any type of device suitable for sending and receiving information, such as a smartphone, a tablet, or other electronic devices that receive or provide information, such as wearable devices. The processor 102 can be an ARM processor, an X86 processor, a MIPS processor, a graphics processing unit (GPU), a general-purpose GPU, or any other processor configured to execute instructions stored in a memory. The processor 102 can be implemented in one or more processing devices, where the processors can be different. For example, the electronic device can include a central processing unit (CPU) and, for example, a GPU. The operation of the processing circuit can depend on both software and hardware to implement various features of the circuits and methods described below.

[0020] The processor 102 can be coupled to a display 106 for displaying information to a user. The processor 102 can also be coupled to a memory 108 capable of storing information related to data or information associated with image data. As is well known, the memory 108 can be implemented as part of the processor 102 or can be implemented as a memory other than any cache memory of the processor. The memory 108 can include any type of memory, such as a solid-state drive (SSD), flash memory, read-only memory (ROM), or any other memory element providing long-term memory, where the memory can be any type of electronically driven internal memory or external memory accessible to the electronic device.

[0021] A user interface 110 is also provided to enable a user to input data and receive data. Some aspects of recording images may require manual input from the user. The user interface 110 can include a touchscreen user interface and other input / output (I / O) elements (such as speakers and microphones) commonly used on portable communication devices (such as smartphones, smartwatches, or tablet computers). The user interface 110 can also include devices for inputting or outputting data that can be attached to the mobile device via an electrical connector or via a wireless connection (such as a Bluetooth or near-field communication (NFC) connection).

[0022] Processor 102 may also be coupled to other elements that receive input data or provide data, including various sensors 120 for activity tracking, an inertial measurement unit (IMU) 112, and a global positioning system (GPS) device 113. For example, the inertial measurement unit (IMU) 112 may provide various information related to the movement or orientation of the device, while the GPS 113 provides location information associated with the device. Sensors that may be part of or coupled to the mobile device may include, for example, light intensity (such as ambient light or UV light) sensors, proximity sensors, ambient temperature sensors, humidity sensors, heart rate detection sensors, galvanic skin response sensors, skin temperature sensors, barometers, speedometers, altimeters, magnetometers, Hall sensors, gyroscopes, WiFi transceivers, or any other sensors that may provide information relevant to achieving the objective. Processor 102 may receive input data through an input / output (I / O) port 114 or a transceiver 116 coupled to an antenna 118. Although the elements of the electronic device are shown by way of example, it should be understood that other elements may be implemented in the Figure 1 electronic device, or the electronic device may be arranged differently to implement the methods set forth below. Although Figure 1 multiple cameras are shown, consecutive frames with different characteristics (such as different exposure characteristics) may be captured by a single camera. For example, multiple frames may be from a single camera that changes the exposure between frames. It should be noted that for any reference to multiple images below, a single camera or multiple cameras may be used to capture the multiple images.

[0023] Turning now to Figure 2 , which shows different levels associated with multiple images from different cameras or from a single camera, where multiple images captured by a single camera may have different exposure characteristics. According to some implementations, different cameras may have different exposure times or different uses of supplementary light (such as a flash). It should also be understood that each of Image 1 and Image 2 may be a multi-frame image. To avoid transition artifacts, a multi-scale blending algorithm associated with a blending pyramid may be used. Conventional blending algorithms typically have difficulty transitioning quickly between dark and light regions. A particular artifact called halation typically appears around the transition point between dark and light regions. On the bright side of the transition, the blended result becomes brighter, while on the dark side of the transition, the blended result becomes too dark. This region around the scene transition looks very unnatural, indicating that the camera may have applied synthetic processing to generate the final image. The circuits and methods described in more detail below address this transition artifact so that the blended output result looks more natural.

[0024] According to some implementations, as will be referencedFigure 3 The circuits and methods for implementing a hybrid pyramid, described in more detail, accept two different images (such as a long exposure input and a short exposure input, and a frame of hybrid weights). A high weight for an image at a given level (shown here as having n levels) indicates that the resulting mixture should use a higher contribution from one of the images (such as the short exposure image). A low hybrid weight indicates that the resulting mixture should use a higher contribution from the other image (such as the long exposure image). The circuits and methods set forth below improve the resulting mixture output in regions of the composite image where the hybrid weights transition rapidly from low values to high values.

[0025] According to Figure 2 the example, n levels are provided, where level 1 provides a high detail level and the resolution level decreases to level n, level n represents a low-level pass, and level n can represent a base level. The resolution of the levels can be generated by downsampling (such as by reducing the resolution of the image to, for example, one quarter), where the resolution of each of the height and weight dimensions is reduced by half, where only one quarter of the pixels of the next higher resolution level will be used. That is, compared to level 1, level 2 can only have one quarter of the number of pixels associated with the image. For example, for each image, for each square of 4 pixels in level 1, only one of the pixels will be used to represent the image in level 2, and for each square of 4 pixels in level 2, only one pixel will be used to represent the image in level 3, and so on. The base image in level n is an image with a very low resolution. Although two images are shown by way of example, it should be understood that more than two images can be mixed using the circuits and methods set forth below. Refer to Figure 3 provides more details related to mixing two images in a hybrid pyramid.

[0026] Now turning to Figure 3 , which shows the mixing of multiple images using weights at different levels associated with a hybrid pyramid, where images 1 and 2 can be implemented as described above to represent images from different cameras with different exposures, and can be multi-frame images. Alternatively, images 1 and 2 can be captured by a single camera, where the multiple images can have different exposure characteristics. The pair of images is passed together with a set of hybrid weights that define how the input images should be mixed together to form an output image. At each level of the pyramid, the input image can be divided into a base image and a detail image. The base image is a low-resolution version of the input image, while the detail image contains high-pass information. The base image is passed as input to the next level of the pyramid. After all the levels have been computed, there is a single low-resolution base image and a set of detail images from the pyramid.

[0027] In a reconstruction pyramid, when the base image and the detail images are combined, the original input image is reconstructed. For a blending application, each level of the image pyramid also includes a blending block that combines information from two input images to create a blended detail image. The lowest level of the pyramid contains the low-frequency views of the two input images. These low-frequency views are also blended together to form a blended base layer. The blended base layer is given to the reconstruction pyramid together with the blended detail layers from the blending pyramid to form a final blended output image. It should be noted that the correction of transition effects (such as the halo effect) can be performed during the blending process rather than after blending is complete. According to one implementation, the blending pyramid can include 8 levels, where blending can be performed at specific levels. For example, blending at the parts of the image detected to have transition artifacts may not be performed at the first 3 levels (i.e., Figure 3 levels 1 - 3 of the example shown) with the highest resolution to avoid loss of texture. Blending can also be avoided at the lowest level (i.e., level n) (which is generally a blurred version of the image and is used to identify the location of transition artifacts). For example, if n = 8, each pixel in the eighth level of the image will represent 256×256 pixels in the original image and thus has a rather low resolution. As Figure 3 shown, the weights are generated on the right side and are used to implement the blending. Different from traditional devices that only receive weights, as will be described in more detail below, the blending according to Figure 3 the implementation uses weights W0 - Wn and derivative weights Wd0 - N (generally referred to as Wdi for a given level i). Although the blending pyramid is shown by way of example, it should be understood that the circuits and methods for processing image data in a composite image can be used for any multi-scale blending operation.

[0028] Now turning to Figure 4 , which shows a block diagram of a blending control block 402 for generating a blended output image based on two input images. The blending control block 402 can be used to generate an output image based on a blending pyramid with multiple levels as Figure 3 shown. The blending control block 402 receives multiple images, shown here by way of example as a first image (I1) and a second image (I2). The blending control block 402 also receives weight values (Wi and Wdi)), and the weight values are applied as referred to above with reference to Figure 3Images at different levels described to generate a blended output image (Out). The weight value Wi represents a weight map, where the weight value Wi describes the weight of each level from the first image. As will be described in more detail below, Wdi represents high-pass interpolation and is used to determine the attenuation value.

[0029] As Figure 5 shown in the graph of, the attenuation value A for each level of the two input images can be generated based on the value of Wdi. That is, based on different values of Wdi at different levels, each level can have a different attenuation value A. In addition, one or more equations for determining different attenuation values can be different for each level (i.e., for each level, Figure 4 the graph of can be different). The method of correcting the halo artifact is based on detecting large transitions in the blending weights at each pyramid level. According to the circuit and method for processing image data in the composite image, each weight level should produce a base output and a detail output. The Wdi value provides the derivative or change amount of the weight at each blending level. According to one implementation, the blending pyramid can be a Laplacian pyramid. When there is a large change in the weight of a given part of the image, halos may appear. To correct such halo effects, an attenuation gain is provided to the output detail level of the same pyramid at that part of the image. By attenuating or suppressing the blended detail level, halo overshoot in the blended output is prevented. Wdi represents the detail output from the weight pyramid at the current level I and can include the second derivative associated with the Laplacian function. That is, Wi can include the first derivative of the pixels of the image, and Wdi can include the second derivative of the pixels of the image and represents the difference with respect to the image of the layer with higher resolution above. The output generated by the blending control circuit 402 can be expressed as: Figure 5 Out = (Wi * I1 + (1 - Wi) * I2) * A (1)

[0030] Out=(Wi*I1+(1-Wi)*I2)*A (1)

[0031] As Figure 5 shown, the attenuation factor A can be calculated using a linear curve. According to one implementation, the attenuation value can be calculated as follows:

[0032] y = 1 - m(x - t1), (2)

[0033] where the slope m can be:

[0034] m = (1 - Amin) / (t2 - t1) (3).

[0035] Thus, the output after mixing in the region identified as having transition artifacts is attenuated by multiplying the standard output of the stage by an attenuation value A that is less than or equal to 1. According to one implementation, the value of A can be equal to 1.0 until a first threshold t1 associated with the Wdi value, and then decreased to a minimum Amin until a second threshold t2 is reached.

[0036] To determine the values of t1 and t2, a difference image Wd is determined, where

[0037] Wi - Wdi = Wd, (4)

[0038] where Wi can be based on the input image to a low - pass filter (whose output is the downsampled image), and Wdi is based on the upsampled version of the corresponding downsampled Wi image. According to one implementation, the Wdi value can be generated, for example, after a Laplace transform (to generate the difference image based on images of the same resolution). The values of t1 and t2 are selected such that the resulting difference image Wd avoids the halo effect at the transition edges between bright and dark regions. Although Laplace techniques can be employed, other techniques (such as wavelet decomposition) can also be used.

[0039] Figure 5 The parameters of the graph (i.e., the controllable thresholds Amin, t1, and t2) can be adjustable and can be different for each stage of the pyramid. The mixing control circuit 402 provides the attenuation factor when the detail input is mixed. At the highest stage of the pyramid, the detail information also contains important information about the image texture and fine details. The technique of mixing using the Wdi value can be selectively applied. For example, the technique can be not applied at one or more high - resolution stages to avoid losing these important details. The parameters can also be changed based on the scene, where the parameters are optimized for the scene. According to one implementation, a parameter library can be stored, where a group of parameters from the library is selected based on the determination of the characteristics of the scene. When an image enters, the parameters can be selected (e.g., on a per - image basis) to be applied to perform the mixing.

[0040] Now turning to Figure 6 , which shows an example image that is mixed to form an output image with a reduced halo effect. As Figure 6 shown, as described above, the mixing control block receives two images 602 and 604, and receives a weight map 606 to generate an output image 608. It can be seen that the hard transition between the bright and dark regions in the generated output image 608 has a reduced halo effect.

[0041] Now turning to Figure 7, which is a flowchart showing a method of processing image data in a composite image. At block 702, multi-level blending of the image of the composite image is analyzed, such as by using the Wdi value as described above. At block 704, the sharp transition boundary region of the composite image is identified. At block 706, less attenuation is applied to the higher levels of the multi-level blending of the composite image in the sharp transition boundary region. In some cases, attenuation may not be applied at a particular level. At block 708, more attenuation is applied to the lower levels of the multi-level blending of the composite image in the sharp transition boundary region.

[0042] According to some implementations, analyzing the multi-level blending of an image may include analyzing the pyramid blending of the image of the composite image. The method may also include selectively applying attenuation at a predetermined level among multiple levels, including applying attenuation using a controllable threshold. For example, the controllable threshold may include a first threshold for starting attenuation, a second threshold for ending attenuation, and a minimum attenuation value applied during attenuation. Different controllable thresholds may be applied to different levels of the pyramid blending of the image of the composite image. The method may also include: after applying less attenuation to the higher levels of the multi-level blending of the composite image and more attenuation to the lower levels of the multi-level blending of the composite image in the sharp transition boundary region, generating a blended output of the image of the composite image. Although specific elements of the method are described, it should be understood that additional elements of the method or additional details related to the elements may be implemented according to Figure 1-6 the disclosure of

[0043] Thus, the halo attenuation technique can be implemented as a modification to the traditional blending standard algorithm. However, the same technique can be applied in any blending algorithm where there is a difference between the base-level image and the detail-level image. Modifying the details at a lower resolution reduces the halo effect because the halo is an artifact that affects very high-degree transitions in the scene. This technique is proposed in the context of high dynamic range imaging (where multiple exposures are captured to increase the dynamic range of the combined image output). The technique can also be used to help reduce artifacts when combining, for example, flash and no-flash images. This technique avoids overshoot and undershoot around the transition region between different exposures.

[0044] Thus, it can be understood that new circuits and methods for implementing a device with a focused camera have been described. Those skilled in the art should understand that there will be various alternatives and equivalents incorporated into the disclosed invention. Therefore, the present invention is not limited by the foregoing implementations, but only by the appended claims.

Claims

1. A method for processing image data in a composite image generated by mixing multiple input images obtained by using different cameras or different operating characteristics, the method comprises: analyzing a multi-level hybrid pyramid of the input images of the composite image; identifying a transition boundary region in the composite image where a transition exists in the hybrid weight values of the input images; and using a processor to apply attenuation to the composite image at different levels of the multi-level hybrid pyramid of the composite image in the transition boundary region, where more attenuation is applied to the first level of the multi-level hybrid pyramid than to the second level of the multi-level hybrid pyramid, and the first level of the multi-level hybrid pyramid corresponds to an image with a lower resolution than the second level of the multi-level hybrid pyramid, Among them, applying attenuation at different levels of the multilevel hybrid pyramid of the composite image includes applying attenuation using a threshold of the derivative weight value, where the derivative weight value represents the derivative of the weight of the input image at a given level of the multilevel hybrid pyramid, and where the threshold includes a first threshold (t 1 ) below which attenuation is not applied and a second threshold (t 2 ) above which a minimum attenuation value is applied during attenuation, where the first threshold has a value lower than the second threshold.

2. The method according to claim 1, wherein different thresholds are applied to different levels of the multi-level hybrid pyramid of the input images of the composite image.

3. The method according to claim 1, further comprising generating a mixed output of the input images of the composite image after applying attenuation to different levels of the multi-level hybrid pyramid of the composite image in the transition boundary region.

4. An apparatus for enhancing the quality of a composite image generated by mixing multiple input images obtained by using different cameras or different operating characteristics, the apparatus comprises: a processor circuit configured to: analyze a multi-level hybrid pyramid of the input images of the composite image; identify a transition boundary region in the composite image where a transition exists in the hybrid weight values of the input images; and apply attenuation to the composite image at different levels of the multi-level hybrid pyramid of the composite image in the transition boundary region; where more attenuation is applied to the first level of the multi-level hybrid pyramid than to the second level of the multi-level hybrid pyramid, and the first level of the multi-level hybrid pyramid corresponds to an image with a lower resolution than the second level of the multi-level hybrid pyramid, Among them, applying attenuation at different levels of the multilevel hybrid pyramid of the synthetic image includes applying attenuation using a threshold of the derivative weight value, where the derivative weight value represents the derivative of the weight of the input image at a given level of the multilevel hybrid pyramid, and where the threshold includes a first threshold (t 1 ) below which attenuation is not applied and a second threshold (t 2 ) above which a minimum attenuation value is applied during attenuation, and where the first threshold has a value lower than the second threshold.

5. The apparatus according to claim 4, wherein different thresholds are applied to different levels of the multi-level hybrid pyramid of the input images of the composite image.

6. The apparatus according to claim 4, further comprising generating a mixed output of the input images of the composite image after applying attenuation to different levels of the multi-level hybrid pyramid of the composite image in the transition boundary region.

7. A computer-readable storage medium comprising program code which, when executed by a processor of a device, causes the device to execute the method according to any one of claims 1-3.

Images