Hybrid greyscale image enhancement

Abstract

The present disclosure relates to mixed gray image enhancement. A method for enhancing an image and an image enhancement device are disclosed. The method for enhancing an image includes capturing an initial image including a plurality of pixels and performing, for each pixel of the plurality of pixels, a pixel-wise defogging operation. The performing includes, for each pixel of the plurality of pixels, generating a value of a mixed gray image based on a color channel of the pixel, for each pixel of the plurality of pixels, generating a value of a transmission map based on the mixed gray image, and for each pixel of the plurality of pixels, generating an output color channel of a processed image based on the value of the transmission map, the processed image being an enhanced version of the initial image.

Description

Technical Field

[0001] The present invention generally relates to a system and method for image processing, and in a particular embodiment, to a system and method for enhancing images. Background Technology

[0002] Images of outdoor scenes can be degraded by murky media, such as particles or water droplets in the atmosphere. Light can be scattered and absorbed by these particles, resulting in haze in the captured image. Haze in captured images is problematic because it reduces visibility. For example, haze can be caused by smog, fog, drizzle, smoke, or other airborne particles in images captured outdoors. Haze increases with distance. In other words, distant objects in a hazy image will have reduced visibility compared to closer objects in the same image.

[0003] The loss of information (e.g., visibility) due to fog degrades the performance of image processing applications. Blurred images may cause image processing applications (such as object recognition) to require more processing time, more complex algorithms, and the use of multiple images. Fog can also degrade the performance of geophysical image processing applications (such as remote sensing and surveying applications). For example, in automated driving applications, blurred images captured by camera systems can degrade performance in dangerous critical situations and endanger passengers and / or bystanders.

[0004] Haze removal algorithms are known to utilize light scattering models to recover the original scene radiance from a blurred image. For example, a haze removal algorithm can generate a so-called dark channel prior from a captured blurred image, use the dark channel prior to generate a transmission map, and apply the transmission map to the light scattering model. The dark channel prior is based on the observation that local patches in outdoor images contain pixels with low intensity in at least one color channel. Summary of the Invention

[0005] According to one embodiment of the present invention, a method for enhancing an image includes: capturing an initial image comprising a plurality of pixels; and performing a pixel-by-pixel dehazing operation for each of the plurality of pixels, the operation including: generating a value of a mixed grayscale image based on the color channel of the pixel for each of the plurality of pixels; generating a value of a transmission map based on the mixed grayscale image for each of the plurality of pixels; and generating an output color channel of a processed image based on the value of the transmission map for each of the plurality of pixels, the processed image being an enhancement of the initial image.

[0006] According to one embodiment of the present invention, a method for enhancing an image includes: capturing an initial image comprising a plurality of pixels; generating a mixed grayscale image from the initial image; generating a brightened mixed grayscale image from the brightened initial image; generating a transmissive map for each of the plurality of pixels; calculating an enhancement value based on determining how much haze is present in the initial image; adjusting the transmissive map based on the enhancement value; determining whether the hue of atmospheric light in the initial image is within a possible sky color range of the initial image, the possible sky color range including hue values ​​higher or lower than a non-sky color range; adjusting the transmissive map of each of the plurality of pixels using the enhancement value; adjusting the brightened mixed grayscale image based on determining whether the hue of atmospheric light in the initial image is within a possible sky color range; and generating an enhanced image from the adjusted transmissive map, the adjusted brightened mixed grayscale image, and the initial image after adjusting the transmissive map.

[0007] According to one embodiment of the present invention, an image enhancement device includes: an image sensor configured to capture an initial image; a memory configured to store the captured initial image; and a processor communicating with the memory, the processor being configured to execute instructions stored in the memory, the memory including instructions for performing: capturing an initial image comprising a plurality of pixels; and performing a pixel-by-pixel dehazing operation for each of the plurality of pixels, the execution including: generating a value of a mixed grayscale image from a color channel of the pixel for each of the plurality of pixels; generating a value of a transmission map from the mixed grayscale image for each of the plurality of pixels; and generating an output color channel of a processed image based on the value of the transmission map for each of the plurality of pixels, the processed image being an enhanced version of the initial image. Attached Figure Description

[0008] This patent or application document contains at least one color drawing. A published copy of the patent or patent application containing (one or more) color drawings will be provided by the Patent Office upon request and after payment of the necessary fees.

[0009] To gain a more complete understanding of the invention and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 This is a schematic diagram of an image enhancement device according to an embodiment of this application;

[0011] Figure 2A The illustration shows process steps for enhancing an example color image according to an embodiment of this application, wherein... Figure 2B The illustration shows a magnified, higher-resolution version of the original image. Figure 2C The illustration shows a magnified, higher-resolution diagram of a mixed grayscale image. Figure 2DThe illustration shows a magnified, higher-resolution version of the brightened initial image. Figure 2E The illustration shows a magnified, higher-resolution diagram of a brightened mixed grayscale image. Figure 2F The illustration shows a magnified, higher-resolution schematic of the transmission image. Figure 2G The illustration shows a magnified, higher-resolution schematic of the enhanced transmission pattern. Figure 2H The illustration shows a magnified, higher-resolution diagram of the adjusted, brightened mixed grayscale image. Figure 2I The illustration shows a magnified, higher-resolution view of the mid-scene transmission image. Figure 2J The illustration shows a magnified, higher-resolution image that enhances the image.

[0012] Figure 3 The illustration shows a process flow for enhancing an image according to an embodiment of this application;

[0013] Figure 4 The illustration shows a graphical representation of how pixel-by-pixel dehazing and a simplified light scattering model ensure that blurred images undergo more dehazing than undersaturated images;

[0014] Figure 5 The illustration shows a process flow for calculating atmospheric light in an image according to an embodiment of this application;

[0015] Figure 6 The illustration shows a process flow for generating an intermediate scene transmission map according to an embodiment of this application;

[0016] Figure 7 The illustration shows a graphical representation of how to determine the first, second, or third adjustment factor based on the hue of atmospheric light;

[0017] Figure 8 The illustration shows a model for determining a luminance factor according to an embodiment of this application;

[0018] Figure 9 The diagram illustrates a graphical representation of the pixel-by-pixel method and how the brightening factor determines the required amount of dehazing; and

[0019] Figures 10A to 10D The illustration shows a process flow for dehazing example pixels according to an embodiment of this application, wherein... Figure 10A The illustration shows the capture of the initial image. Figure 10B The figure illustrates the atmospheric light values ​​used to generate the initial image. Figure 10C The diagram illustrates the saturation of atmospheric light as a limit, and Figure 10D The illustration shows the values ​​of example pixels in the initial image to determine the brightness. Detailed Implementation

[0020] Conventional haze removal algorithms utilizing dark channel priors have their own set of advantages and disadvantages. For example, generating the dark channel prior requires large local patches. The larger the size of the local patches, the greater the likelihood that the light scattering model will actually perform as expected. However, as the size of the local patches increases to ensure successful dehazing, processing time and computational cost also increase. This can lead to excessively high computational costs and processing time. Furthermore, algorithms utilizing dark channel priors require edge-preserving filtering steps and / or soft matting steps to further refine their transmission maps to prevent erroneous textures and occlusion artifacts caused by the minimal operators used in dark channel prior-based algorithms.

[0021] A simplified version of the light scattering model, often used to quantify the amount of haze formed in an image, can be represented as:

[0022]

[0023] in It is the intensity of pixels in the captured image. Let A be the scene brightness of the pixel, A be the intensity of atmospheric light in the captured image, and t(x) be the intensity of the pixel in the transmission map.

[0024] Typical methods for image dehazing utilize a combination of a so-called dark channel prior and a simplified light scattering model to remove haze from an image. In other words, the method using the dark channel prior requires the following steps: estimating the dark channel prior based on local patches of pixels; estimating atmospheric light captured in the blurred image; determining the transmission map corresponding to each input pixel based on the dark channel prior and the estimated atmospheric light; thinning the transmission map; and restoring the scene brightness. The scene can be restored by solving Equation 1 as follows:

[0025]

[0026] Where t0 is usually equal to 0.1. Equation 2 is typically a subtractive process, and therefore, the recovered scene brightness may not be as bright as the atmospheric light, which can result in an overly dark image.

[0027] The first drawback of dark channel prior algorithms is that processing time is directly related to the size of local patches. In other words, the smaller the local patches, the shorter the processing time. However, images with low-frequency textures may require larger local patch sizes. In other words, features that dominate patch size with little variation may not satisfy the requirements of dark channel prior algorithms. Therefore, larger patch sizes may need to be considered to ensure sufficient variation within the local patches. Consequently, images with low-frequency textures may require larger local patches, leading to higher computational costs and excessively long processing times.

[0028] Another drawback of the dark channel prior algorithm is that the estimation of atmospheric light is determined as follows: identify a small portion of the brightest pixels in the dark channel prior, and then select pixels within that small portion of pixels from the blurred image.

[0029] However, this can be problematic because the selected pixels may not correspond to atmospheric light. For example, in an image of a road, the brightest pixel might correspond to a white car or a reflective sign, rather than atmospheric light.

[0030] Another drawback of the dark channel prior algorithm is that the minimal operator used may cause clogging artifacts in the dehazed image. In other words, the dehazed image may include clogging artifacts, which can lead to errors in post-processing or decision-making based on the dehazed image.

[0031] Another drawback of the dark channel prior algorithm is that it requires refining the transflection map. Transflection map refining can include a soft matting step. This additional step of refining the transflection map can also significantly increase the computational cost and processing time required for image dehazing.

[0032] Various embodiments of this application disclose pixel-based methods and corresponding devices for image enhancement that do not require an additional soft matting step. More specifically, embodiments of the present invention relate to a method for enhancing images using single-pixel operations, which enhances images with improved contrast / visibility and reduced processing time, while avoiding the artifact blocking inherent in region filtering techniques.

[0033] Figure 1 This is a schematic diagram of an image dehazing device according to an embodiment of this application.

[0034] refer to Figure 1 The image dehazing device 100 may include an image sensor 102 (such as a camera), an image dehazing system 104, a display 118, and a machine vision chip 120.

[0035] The image dehazing system 104 may include a processor 106 coupled to a memory 108.

[0036] The memory 108 may be fixed-line memory and / or random access memory (RAM), read-only memory (ROM), flash memory, hard disk, etc. The memory 108 may be configured to store a mixed grayscale module 112 that can be executed to dehaze images.

[0037] Processor 106 can be configured to access and execute the mixed grayscale module 112 stored in memory 108 to dehaze an image and display it on display 118. Processor 106 may include, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc.

[0038] In various embodiments, processor 106 may also be configured to transmit dehazed images to machine vision chip 120 for further processing related to applications such as object recognition, remote sensing applications, and surveying applications. Alternatively, in some embodiments, processor 106 may be integrated with machine vision chip 120.

[0039] In various embodiments, the processor 106, memory 108, and image sensor 102 may be integrated together, for example, in an application-specific integrated circuit (ASIC) of a camera or another imaging device. In alternative embodiments, the image dehazing device 100 may be connected to or built into an electronic device, such as a personal computer, laptop computer, smartphone, tablet computer, etc.

[0040] Figure 2A The illustration shows the process steps of a pixel-by-pixel method for enhancing an example image according to an embodiment of this application. Figure 2B The illustration shows a magnified, higher-resolution view of the initial image 202. Figure 2C The illustration shows a magnified, higher-resolution view of the mixed grayscale image 204. Figure 2D The illustration shows a magnified, higher-resolution view of the brightened initial image 206. Figure 2E The illustration shows a magnified, higher-resolution diagram of the brightened mixed grayscale image 208. Figure 2F The diagram illustrates a magnified, higher-resolution schematic of the transmission image 210. Figure 2G The illustration shows a magnified, higher-resolution schematic of the enhanced transmission image 212. Figure 2H The illustration shows a magnified, higher-resolution schematic of the adjusted, brightened mixed grayscale image 214. Figure 2I The illustration shows a magnified, higher-resolution schematic of the intermediate scene transmission diagram 216. Figure 2J The illustration shows a magnified, higher-resolution diagram of the enhanced image 218.

[0041] Figure 3The illustration depicts a pixel-by-pixel method for enhancing an image according to an embodiment of this application, corresponding to... Figure 2A The process flow.

[0042] Advantageously, the pixel-wise method generates a mixed grayscale image by blending the color channels of each pixel in the blurred image. In other words, the pixel-wise method generates a mixed grayscale image that substantially matches the dark channel prior without using the minimal operators required for the dark channel prior. This will be shown in more detail below.

[0043] In various embodiments, the pixel-by-pixel method can be applied to applications that process data after a fully blurred image has been determined, or it can be applied to video processing applications where, for example, the image dehazing device 100 (e.g., processing) is embedded in an electronic device's ASIC. In other words, in video applications where pixels are processed as they are read from the image sensor 102, values ​​or statistics collected from previous frames can be applied to the current frame being processed. In other words, as those skilled in the art will understand, a frame repository may or may not be used in conjunction with the pixel-by-pixel method.

[0044] refer to Figure 2A The image receiving module 110 may include instructions for receiving an initial image 202 captured by the image sensor 102 and determining a plurality of pixels (boxes 302) constituting the image. In various embodiments, as described above, the scene captured in the initial image 202 may include haze that causes reduced visibility. Although the embodiments described below describe haze removal (i.e., defogging), these are also applicable to other atmospheric effects that cause reduced visibility. For example, the embodiments can also be applied to images with smoke, fog, snow, rain, mist, etc., due to the use of a simplified light scattering model. In other embodiments, the initial image 202 may not include haze, and the foregoing methods can also be used instead of conventional contrast enhancement techniques, such as local tone mapping.

[0045] In various embodiments, the initial image 202 may be a color image. In various embodiments, each pixel of the plurality of pixels in the initial image 202 may include a vector comprising the intensity of each color channel present in each pixel. For example, each pixel of the plurality of pixels in the initial image 202 may include a red-green-blue (RGB) color channel, a YUV color channel, a cyan-magenta-yellow keyed (CMYK) color channel, etc. For illustrative purposes, the initial image 202 having RGB color channels will be discussed. The color channels present in the pixels of the initial image 202 are not limited to those described in this application.

[0046] Therefore, in various embodiments, the image receiving module 110 may include instructions that, when executed by the processor 106, determine the value of the color channel at each pixel within the image.

[0047] The inventors of this application have recognized that dark channel prior algorithms can produce dehazed images with occlusion artifacts. Dark channel prior algorithms may require the use of local patches to determine the dark channel and a transmission map refinement step, both of which increase processing time. On the other hand, as described above, if smaller local patches are used in the dark channel prior algorithm to save processing time, then low-frequency textures can dominate the local patches. Therefore, the dark channel prior assumption may not hold, and the blurred image may fail to be successfully dehazed.

[0048] Furthermore, as described above, the atmospheric light estimated in the dark channel prior algorithm corresponds to pixels in a blurred image selected from a small subset of pixels, and therefore, the dark channel prior algorithm cannot distinguish between sky regions and bright objects in an image.

[0049] Therefore, the grayscale blending module 112 may include instructions that, when executed in the processor 106, cause the processor 106 to generate a value for the blended grayscale image 204 (box 304) for each of the pixels in the initial image 202, using the color channels of each pixel in the initial image 202. In other words, the blended grayscale image 204 can be generated by blending the color channels of each individual pixel in the initial image 202 together. Depending on the processing capability of the processor 106, this blending of color channels at each pixel can be performed sequentially or in parallel with other pixels in the image. The blended grayscale image 204 can be produced by multiplying the color channels of each individual pixel in the initial image 202. For example, the red, green, and blue channels of each pixel in the initial image 202 can be blended by multiplying them together. Therefore, each of the pixels in the blended grayscale image 204 may include a single value.

[0050] Advantageously, each pixel in the initial image 202 is darkened by blending color channels, and a pixel with lower intensity in at least one color channel of the initial image 202 can be darker than a pixel with a more uniform intensity. One advantage is that the blended grayscale image 204 can have the same effect as generating a dark channel prior, but can be generated using single-pixel operations instead of local patches.

[0051] After generating the mixed grayscale image 204, the mixed grayscale module 112 may include instructions that, when executed in the processor 106, cause the processor 106 to calculate, for each of a plurality of pixels, the value of atmospheric light in the initial image 202 from the mixed grayscale image 204 (box 306). The value of atmospheric light can be calculated based on a small fraction (e.g., the value) of the brightest pixel in the mixed grayscale image 205, the intensity of the color channel of the pixel in the initial image (which corresponds to the brightest pixel in the mixed grayscale image 203), and the number of pixels among the brightest pixels in the mixed grayscale image 205. This will be explained below. Figure 5 A more detailed discussion will follow in the flowchart below.

[0052] The inventors of this application recognized that, for each of the plurality of pixels, since the mixed grayscale image 204 is generated by multiplying the color channels of each pixel of the initial image 202, the initial image 202 may be too dark when generating the mixed grayscale image 204. In other words, each pixel of the mixed grayscale image 204 may be two to three times darker than the value of the darkest color channel of each corresponding pixel in the initial image 202. Therefore, the mixed grayscale module 112 may include other instructions that, when executed in the processor 106, cause the processor 106 to generate, for each of the plurality of pixels, a value of the brightened initial image 206 based on a value of atmospheric light that can be used in subsequent steps to generate a mixed grayscale image that substantially matches the dark channel prior.

[0053] In other words, for each of the multiple pixels, the value of the brightened initial image 206 can be generated based on the value of atmospheric light using the following formula (box 308):

[0054]

[0055] in This represents the value of each pixel in the initial image 206 that has been brightened. Let A represent the value of each pixel in the initial image 202, A represent the value of atmospheric light, and BR represent the global brightening constant.

[0056] Advantageously, the global brightening constant (BR) can be used to increment each color channel of each pixel in the initial image 202, while keeping them less than or equal to one when generating the brightened initial image 206. As described below, one advantage is that the brightened initial image 206 can be used to generate a mixed grayscale image that substantially matches the dark channel prior using single-pixel operations.

[0057] Therefore, the global brightening constant can be any value that increases the brightness of each pixel in the initial image and allows for the generation of a mixed grayscale image from the brightened initial image 206 that substantially matches the dark channel prior. The global brightening constant can be a value between 0.3 and 0.6; alternatively, it can be a value between 0.4 and 0.5. The inventors of this application determined that 0.45 might be the optimal value for the global brightening constant after observing an entire dataset of images dehazed using a pixel-by-pixel method. However, different values ​​within this range can be used with further research.

[0058] Refer again Figure 2A To generate a mixed grayscale image that meets the minimum brightness requirement, the mixed grayscale module 112 may include additional instructions that, when executed in the processor 106, cause the processor 106 to generate the value of the brightened mixed grayscale image 208 for each of the plurality of pixels from the value of the brightened initial image 206 (box 309). In other words, the color channels of each pixel in the brightened initial image 206 can be mixed by multiplying the color channels of each pixel in the individual pixels of the brightened initial image 206 in the same manner as discussed above. Therefore, each of the plurality of pixels in the brightened mixed grayscale image 208 may include a single value.

[0059] Next, the grayscale mixing module 112 may include instructions that, when executed in the processor 106, cause the processor 106 to generate the value of the transmission map 210 for each of the plurality of pixels from the value of the brightened grayscale mixed image 208 (box 310). In other words, the transmission map 210 can be generated by inverting the value of each pixel in the brightened grayscale mixed image 208.

[0060]

[0061] After generating the transmission map 210, the grayscale mixing module 112 may include other instructions that, when executed in the processor 106, cause the processor 106 to generate the values ​​of the intermediate scene transmission map 216 from the values ​​of the transmission map 210 for each of the plurality of pixels (box 312).

[0062] As will be understood by those skilled in the art, distant objects typically correspond to sky regions or other areas of the initial image 202 where only noise (and no useful information) will be recovered by dehazing those areas. Furthermore, as will be understood by those skilled in the art, closer objects in the initial image 202 are less affected by haze and should not undergo as much dehazing as intermediate scene objects that are more affected by haze.

[0063] Therefore, in various embodiments, an intermediate scene transmission map 216 can be generated and structured to focus dehazing on the intermediate scene of the initial image 202 by: generating a value for a brightened transmission map 212 from the value of the transmission map 210 for each of the plurality of pixels; generating a value for an adjusted brightened mixed grayscale image 214 from the brightened mixed grayscale image 208 based on the calculated hue of the atmospheric light; combining the value of the brightened transmission map 212 with the value of the adjusted brightened mixed grayscale image 214; and inverting the combined value of the brightened transmission map 212 and the adjusted brightened mixed grayscale image 214. This will be explained below. Figure 6 We will discuss this in more detail later.

[0064] Next, the grayscale mixing module 112 may include instructions that, when executed in the processor 106, cause the processor 106 to generate an output color channel of a processed image at each pixel based on the values ​​of the transmission map, the processed image being an enhanced version of the initial image (box 314). In one or more embodiments, the grayscale mixing module 112 may include instructions for performing the following: for each of a plurality of pixels, recovering the scene brightness value of the initial image 202 based on each pixel value among atmospheric light, the initial image 202, and the pixel values ​​in the intermediate scene transmission map 216. In various embodiments, the scene brightness of the initial image 202 may be recovered to a simplified version of a light scattering model:

[0065]

[0066] Where T(x) represents the value of the intermediate scene transmission map 216 for each of the multiple pixels.

[0067] Advantageously, because the values ​​of the intermediate scene transmission map 216 are generated based on single-pixel operations and the hue of atmospheric light, no additional transmission map refinement step (e.g., soft matting) is required to smooth the intermediate scene transmission map 216. One advantage is that this reduces the computational cost and processing time for image dehazing.

[0068] The inventors of this application have observed that when restoring scene brightness to obtain a sharper enhanced image 218, filters of different sizes, such as Gaussian filters, can optionally be used when dividing the values ​​of the intermediate scene transmission map 216. For example, in various embodiments, small-sized filters, such as 5×5 Gaussian filters, can be used when restoring scene brightness. In other embodiments, a 5×5 Gaussian filter can be used on the brightened mixed grayscale image 208 before generating the adjusted brightened mixed grayscale image 214. Advantageously, as those skilled in the art will understand, this filter can be implemented inexpensively in terms of computation and has minimal impact on the overall processing time.

[0069] As described above, the advantage of the pixel-by-pixel method is that the simplified version of the light scattering model used with the above steps (Equation 5) ensures that pixels that require more dehazing will actually undergo more dehazing than pixels that do not require as much dehazing.

[0070] Figure 4 The illustration shows a graphical representation of how pixel-by-pixel dehazing and a simplified light scattering model ensure that blurred images undergo more dehazing than undersaturated images.

[0071] like Figure 4 As illustrated in the side-by-side curves 320, the difference between the brightness of the restored scene and the intensity of pixels in the initial image 202 can be greater in the blurred initial image than in the undersaturated initial image.

[0072] In various embodiments, such as Figure 4 As illustrated, the smaller the transmission value (e.g., t(x)) of the pixel in the transmission map 210, the closer the pixel value in the initial image 202 is to atmospheric light. Furthermore, the smaller the transmission value of the pixel in the transmission map 210, the further that transmission value is from its desired recovered scene brightness (J(x)).

[0073] Therefore, as those skilled in the art will understand, the pixel values ​​(e.g., 1-t(x)) of the undersaturated initial image 202 can be closer to the values ​​of atmospheric light than the pixel values ​​in the blurred initial image. In other words, the pixels in the blurred initial image 202 are further away from the desired J(x) value than the pixels in the undersaturated initial image 202, and require additional dehazing.

[0074] In other words, the mixed color channel of an undersaturated pixel will have a smaller transmittance value than that of a blurred pixel.

[0075] Because the values ​​in the intermediate scene transmission map 216 are determined at least in part based on the transmission map 210 (which is generated by inverting the blended pixels), darker blended pixels (e.g., non-zero pixels with values ​​less than 0.5) will be brightened in the transmission map 210 (and vice versa). Therefore, based on a simplified version of the light scattering model (e.g., Equation 5), the higher the values ​​of the non-zero pixels in the transmission map 210 (e.g., the closer they are to one), the fewer pixels will be dehazed (and vice versa). Thus, as... Figure 4 As illustrated in the figure, a simplified version of the scattered light model (e.g., Equation 5) ensures that a blurred image that requires more dehazing will undergo more dehazing than an undersaturated image.

[0076] Figure 5 The diagram illustrates the process for calculating the atmospheric light value of the initial image 202 from the mixed grayscale image 204 for each of the multiple pixels. Figure 5 It can correspond to Figure 3 Box 306.

[0077] As shown in box 402 and referenced Figure 2A As described, for each of the plurality of pixels in the mixed grayscale image 204, the brightest pixel in the mixed grayscale image 205 can be determined. In various embodiments, the brightest pixel in the mixed grayscale image 205 may include a small fraction of the brightest pixels in the mixed grayscale image 204 (e.g., the pixel with the highest intensity). In various embodiments, the brightest pixel in the mixed grayscale image 205 may be a pixel having a value within the top 0.1% to 1% of the values ​​in the mixed grayscale image 204, or alternatively, within the top 1% to 3% in another example.

[0078] In various embodiments, as described above and as understood by those skilled in the art, in video processing applications, pixel processing can be performed while pixels are being read from image sensor 102 (e.g., without the need for a frame repository). Therefore, in video processing applications, to address the need for the entire frame to determine the brightest pixel in the mixed grayscale image 205, the brightest pixel in the mixed grayscale image 205 of the previous frame (f-1) can be accumulated and applied to the current frame (f). Advantageously, this avoids the need for the frame repository to determine the entire image before a value can be determined for a given pixel.

[0079] As illustrated and referenced in box 404 below. Figure 2A As described, for each of the brightest pixels in the mixed grayscale image 205, a value for a pixel in the initial image can be determined, which corresponds to a small subset of the brightest pixels in the mixed grayscale image 203. As will be understood by those skilled in the art, each value of a pixel in the initial image (which corresponds to the brightest pixel in the mixed grayscale image 203) can be a vector comprising the intensity corresponding to each color channel. For example, the value of a pixel in the initial image (which corresponds to the brightest pixel in the mixed grayscale image 203) can include the intensity corresponding to the red, green, and blue channels.

[0080] As illustrated and referenced in box 406 below. Figure 2A As described, the average value of each pixel in the initial image (corresponding to the brightest pixel in the mixed grayscale image 203) can be determined. In various embodiments, calculating the average value among the pixels in the initial image (corresponding to the brightest pixel in the mixed grayscale image 203) can include: individually calculating the sum of the intensities of each color channel among each pixel in the initial image (corresponding to the brightest pixel in the mixed grayscale image 203), and dividing by the number of brightest pixels in the mixed grayscale image 205.

[0081] For example, the sum of the intensities of the red channel, the sum of the intensities of the green channel, and the sum of the intensities of the blue channel in each pixel of the initial image (corresponding to the brightest pixel in the mixed grayscale image 203) can be calculated separately. Then, each sum corresponding to each channel can be divided by the number of pixels included in the brightest subset of pixels. In other words, the value of atmospheric light can be calculated using the following formula:

[0082]

[0083] The inventors of this application have recognized that the brightest pixel in the mixed grayscale image 205 may correspond to an object in the initial image 202 rather than a sky area, and may be overly vivid in color, causing the enhanced image 218 to be discolored. For example, the image may include a bright yellow car, which, if not properly excluded, can easily disrupt the atmospheric correction of the mixed grayscale image. Advantageously, blocks 408 to 412 describe the step of limiting the saturation of atmospheric light within a grayscale range to ensure that the values ​​of atmospheric light correspond to colors similar to atmospheric light, thereby preventing discoloration. In various embodiments, the steps described in blocks 408 to 412 may be optional.

[0084] As illustrated and referenced in box 408 below. Figure 2A The description indicates that the saturation of atmospheric light can be determined. Atmospheric light saturation can be determined by converting atmospheric light into a hue, saturation, value (HSV) representation.

[0085] As illustrated and referenced in box 410 below. Figure 2A As described, the saturation of atmospheric light can be limited to a grayscale range. In various embodiments, atmospheric light can be limited to a grayscale range by limiting the saturation of atmospheric light to a value less than or equal to 0.1.

[0086] As illustrated and referenced in box 412 below. Figure 2A As described, atmospheric light values ​​can be updated based on constrained saturation. In other words, after constraining saturation, the HSV representation can be converted back to RGB representation to produce atmospheric light values.

[0087] In various embodiments, as described above, a value for the intermediate scene transparency map 216 can be generated for each pixel in the pixel set. The value of the intermediate scene transparency map 216 can be generated by combining and inverting the value of the brightened transparency map 212 with the value of the adjusted brightened mixed grayscale image 214 for each of a plurality of pixels.

[0088] Figure 6 The illustration shows the process steps for generating an intermediate scene transmission map 216 for each of a plurality of pixels. Specifically, boxes 502 to 506 illustrate the step of generating a brightened transmission map 212 for each of a plurality of pixels, boxes 508 to 514 illustrate the step of generating an adjusted brightened mixed grayscale image 214 for each of a plurality of pixels, and box 516 illustrates the step of combining the value of the brightened transmission map 212 with the value of the adjusted brightened mixed grayscale image 214 at each of a plurality of pixels and then inverting the result to generate the value of the intermediate scene transmission map 216.

[0089] As illustrated and referenced in box 502 Figure 2A As described, for each of the multiple pixels, the average value of the transmission map 210 can be determined.

[0090] As illustrated and referenced in box 504 below. Figure 2A As described, the brightening factor can be determined based on the average value of transmission diagram 210 and atmospheric light. This will be discussed in the following... Figure 8 We will discuss this in more detail later.

[0091] As illustrated and referenced in box 506 below. Figure 2A As described, for each of a plurality of pixels, the value of the enhanced transmission map 212 can be generated based on a brightness factor. In various embodiments, the value of the enhanced transmission map 212 can be generated by applying the following formula to each pixel in the transmission map 210:

[0092]

[0093] in Let represent the value of each pixel in the transmission image 210, and let b represent the brightness factor. This represents the value of each pixel in the brightened transmission map 212.

[0094] The inventors of this application have observed from the atmospheric light in an image with a small amount of haze that the atmospheric light should be close to the possible values ​​of the sky color in the initial image 202. In other words, an image with atmospheric light having a hue within the possible sky color range of the initial image (i.e., above or below the possible non-sky range) may require less dehazing. Therefore, in various embodiments, an adjusted brightened mixed grayscale image 214 may be generated based on a first adjustment factor, a second adjustment factor, or a third adjustment factor, depending on whether less or more dehazing is required.

[0095] As described above, boxes 508 to 516 illustrate the steps involved in generating the adjusted, brightened, mixed grayscale image 214.

[0096] As illustrated below in box 508, the calculated hue of the atmospheric light can be determined to be above, below, or within the possible non-sky range of the initial image 202. In other words, the hue of the atmospheric light can be determined to be below a hue range that cannot represent the color of the sky in the initial image 202 (e.g., a non-sky range), within a hue range that cannot represent the color of the sky in the initial image 202 (e.g., a non-sky range), or above a hue range that cannot represent the color of the sky in the initial image 202 (e.g., a non-sky range). In various embodiments, if the hue of the atmospheric light is between 0.5 and 0.7, or alternatively between 0.7 and 0.6, then it is within the non-sky range. If the hue is not within this range, then it is within a first range of hue values ​​below the non-sky range or a second range of hue values ​​above the non-sky range, both of which represent possible hues of the sky in the initial image 202.

[0097] If the calculated atmospheric light hue is lower than that of the non-sky range, the method continues to box 510 and determines the first adjustment factor.

[0098] The inventors of this application have recognized that a first adjustment factor with a value between 0.4 and 0.8 is ideal for images with hues below the atmospheric light range that may not be sky range.

[0099] Therefore, the first adjustment factor can be determined by mapping values ​​below the potentially non-sky range to values ​​between 0.4 and 0.8. In other words, the adjustment factor can be determined using the following equation:

[0100]

[0101] Where adjust1 represents the first adjustment factor, and h represents the hue of atmospheric light in radians.

[0102] As illustrated and referenced in box 516 below. Figure 2A As described, after determining the first adjustment factor, for each of the plurality of pixels, the adjusted value of the brightened blended grayscale image 214 can be generated from the value of the brightened blended grayscale image 208 based on the first adjustment factor using the following equation:

[0103]

[0104] in This represents the value of each pixel in the multiple pixels of the adjusted and brightened mixed grayscale image 214.

[0105] On the other hand, referring again to box 508, if the calculated hue of the atmospheric light is outside the sky range, then the method continues to box 512, and a second adjustment factor can be determined.

[0106] The inventors of this application have recognized that if the atmospheric light of the initial image 202 has a hue in a non-sky range (e.g., a hue outside the possible sky range of the initial image 202), then a relatively large amount of haze may exist, and having a second adjustment factor in the range of 0.8 to 1 is ideal.

[0107] Therefore, in various embodiments, the second adjustment factor can be determined using the following equation:

[0108] adjust2 = -3.16 * h 2 +2h+0.6685 Equation 10

[0109] Where adjust2 represents the second adjustment factor.

[0110] Then, referring again to box 516, after determining the second adjustment factor, for each of the multiple pixels, the adjusted value of the brightened blended grayscale image 214 can be generated from the value of the brightened blended grayscale image 208 based on the second adjustment factor using the following equation:

[0111]

[0112] On the other hand, referring again to box 508, if the calculated hue of the atmospheric light is higher than the sky range, then the method continues to box 513, and a third adjustment factor can be determined.

[0113] The inventors of this application have recognized that if the atmospheric light in the initial image 202 has a higher hue than the non-sky range, then a smaller amount of haze can be present, and a third adjustment factor in the range of 0 to 0.8 is ideal. Therefore, in various embodiments, the third adjustment factor can be determined using the following equation:

[0114]

[0115] Where adjust3 represents the third adjustment factor.

[0116] Then, referring again to box 516, after determining the third adjustment factor, for each of the multiple pixels, the adjusted value of the brightened blended grayscale image 214 can be generated from the value of the brightened blended grayscale image 208 based on the third adjustment factor using the following equation:

[0117]

[0118] Advantageously, when generating the values ​​of the adjusted, brightened mixed grayscale image 214, the adjustment factor can be determined based on whether more or less dehazing is required. As understood by those skilled in the art, the higher the pixel values ​​in the intermediate scene transmission map 216 (e.g., the closer they are to one), the fewer of these pixels will be dehazed (as demonstrated by Equation 5). Furthermore, based on Equation 14 shown below, the lower the pixel values ​​in the adjusted, brightened mixed grayscale image 214, the higher the corresponding pixel values ​​in the intermediate scene transmission map 216. Therefore, for the adjusted, brightened mixed grayscale image 214, the more pixel values ​​are reduced, the less dehazing occurs (and vice versa).

[0119] Specifically, a brightened mixed grayscale image adjusted using the first and third adjustment factors can be darker than a brightened mixed grayscale image adjusted using the second adjustment factor. In some embodiments, the darkening caused by the first adjustment factor can be greater than or equal to the darkening caused by the third adjustment factor. In other words, if the calculated hue of the atmospheric light is within the possible sky range (i.e., above or below the non-sky range), then when generating the values ​​of the adjusted brightened mixed grayscale image 214, the pixel values ​​will be reduced more than if the hue were within the non-sky range. Advantageously, this ensures that the pixel values ​​in an image with hues within the possible sky range will be reduced more by the first or third adjustment factor and undergo less dehazing compared to pixels in an image with atmospheric light within the non-sky range.

[0120] In other words, if the sky color of the initial image 202 is detected in atmospheric light, then the initial image 202 must have a small amount of haze. Therefore, the first adjustment factor, the second adjustment factor, and the third adjustment factor ensure that images including a large amount of haze undergo more dehazing (and vice versa).

[0121] As illustrated and referenced in box 516 below. Figure 2A As described, for each of the multiple pixels, the values ​​of the adjusted, brightened mixed grayscale image 214 and the brightened transmission map 212 can be combined and inverted to generate the value of the intermediate scene transmission map 216. The intermediate scene transmission map 216 can be generated using the following formula:

[0122]

[0123] in This represents the pixel value of the intermediate scene transmission map 216.

[0124] As those skilled in the art will understand, since the transmission map 210 can be generated by inverting the pixel values ​​of the brightened mixed grayscale image 208, pixels with values ​​equal to zero in the transmission map 210 represent sky areas or very hazy distant objects, and dehazing them would capture noise rather than scene information. On the other hand, pixels equal to one represent pixels without haze and therefore do not require dehazing. Advantageously, generating the values ​​of the intermediate scene transmission map 216 ensures that pixels in the transmission map 210 with only intermediate range values ​​(e.g., values ​​between zero and one) undergo any dehazing. In other words, pixels with zero values ​​in the transmission map 210 can be pushed to one for the intermediate scene transmission map 216, while pixels equal to one in the transmission map 210 remain equal to one. Therefore, the restored scene brightness of pixels in the transmission map 210 with values ​​equal to zero or one can have values ​​equal to their corresponding pixels in the initial image 202. Thus, dehazing can be performed on pixels in the transmission map 210 with values ​​between zero and one, thereby ensuring that pixels that should not be dehazed are not dehazed.

[0125] The amount of haze present in the initial image 202 changes as the value of the average atmospheric light and the average value of the transmittance map 210 decrease. Advantageously, the brightening factor can indicate the amount of dehazing to be added.

[0126] Figure 7 The illustration shows a graphical representation 600 illustrating how to determine the first, second, or third adjustment factor based on the hue of atmospheric light.

[0127] refer to Figure 7The adjustment factor for generating the adjusted brightened mixed grayscale image 214 can be determined based on the hue of the atmospheric light in the initial image 202. In other words, the adjustment factor used to generate the adjusted brightened mixed grayscale image 214 from the brightened mixed grayscale image 208 can be determined based on whether the hue of the atmospheric light is within the sky range of the initial image 202. As described above, if the hue is higher or lower than the non-sky range of the initial image, then it can be determined whether the hue of the atmospheric light is within the sky range. Specifically, if the hue is lower than the non-sky color range, then the first adjustment factor can be used; if the hue is within the non-sky color range, then the second adjustment factor can be used; and if the hue is higher than the non-sky color range, then the third adjustment factor can be used. In various embodiments, the non-sky color range can include hues between 0.5 and 0.7, or alternatively, hues between 0.7 and 0.6.

[0128] refer to Figure 8 The equation corresponding to the linear trend line 704 for multiple data points 702 can be used to calculate the brightness factor. Each data point 702 can represent an image. Data points 702 can be plotted based on the relationship between their average atmospheric light value and the average pixel intensity value of their corresponding transmittance map. The formula corresponding to the linear trend line 704 can be used to determine the brightness factor. In other words, the brightness factor (e.g., the equation of the linear trend line) is determined as follows:

[0129]

[0130] Where T mean denoted as the average pixel value of the transmission map 210, and b represents the brightness enhancement factor.

[0131] As will be understood by those skilled in the art, the smaller the average pixel value and the smaller the average atmospheric light value of the transmission map 210, the more dehazing is required (e.g., the more blurred the image). Advantageously, the luminance factor is determined based on this relationship, ensuring that the smaller the average pixel value of the transmission map 210, the more image dehazing is required.

[0132] Figure 9 The illustration shows the graphical representation of the pixel-by-pixel method and how the brightening factor indicates the required amount of dehazing.

[0133] refer to Figure 9The first graph 900 illustrates the output values ​​of a pixel-by-pixel method for dehazing an image. In various embodiments, the input may represent the pixel value at the beginning of a step in the pixel-by-pixel method, and the output may represent the pixel value generated at the end of each step. In other words, the first graph 900 illustrates how the pixel value changes from the input image intensity 902 to the intermediate scene transmission map intensity 904, and then from the intermediate scene transmission map intensity 904 to the output image intensity 906. Advantageously, as described above, the closer a pixel value is to the middle range in the input image (e.g., 0.5), the more it will be dehazed. In other words, the distance between the line labeled linear and the line labeled J increases as the line labeled linear approaches 0.5 (and vice versa).

[0134] In addition, the second curve 908 illustrates a graphical representation of the relationship between the brightening factor and the amount of dehazing.

[0135] The second graph 908 illustrates the pixel values ​​of an intermediate scene transparency with an average pixel value of 0.1 (first line 910), an intermediate scene transparency generated with a brightness factor of 0.5 (second line 912), an intermediate scene transparency generated with a brightness factor of 0.6 (third line 914), an intermediate scene transparency generated with a brightness factor of 0.7 (fourth line 916), an intermediate scene transparency generated with a brightness factor of 0.8 (fifth line 918), an intermediate scene transparency generated with a brightness factor of 0.9 (sixth line 920), and an intermediate scene transparency generated with a brightness factor of 1 (seventh line 922).

[0136] Referring to the second curve 902, as the brightening factor decreases, the dehazing amount increases (and vice versa). In other words, the equation (Equation 15) for calculating the brightening factor ensures that the lower the average pixel value of the transmission map 210, the smaller the brightening factor (and vice versa). In other words, the blurrier the image, the smaller the brightening factor.

[0137] Therefore, referring to Equation 7, the smaller the brightening factor, the brighter the brightened transmission map 212 will be (and vice versa). Furthermore, as further demonstrated by Equation 14, because the brightened transmission map 212 is inverted, the brighter the brightened transmission map 212, the darker the intermediate scene transmission map 216 will be (and vice versa). Moreover, as described above, the darker the intermediate scene transmission map 216 (e.g., the lower the values ​​of multiple pixels in the intermediate scene transmission map 216), the more dehazing will occur. Therefore, as... Figure 9 As illustrated in the figure, the more blurred the image, the smaller the brightening factor, which produces a brighter transmission map 210. And because the transmission map is inverted when generating the intermediate scene transmission map 216, this ensures that the more blurred image undergoes more dehazing than the less blurred image.

[0138] Figures 10A to 10D The illustration shows a process flow for dehazing example pixels according to an embodiment of this application, wherein... Figure 10A The illustration shows the capture of the initial image. Figure 10B The figure illustrates the atmospheric light values ​​used to generate the initial image. Figure 10C The diagram illustrates the saturation that limits atmospheric light. Figure 10D The illustration shows the values ​​of example pixels in the initial image to be brightened. Although a single pixel is described, the steps described below can be applied to every pixel of the initial image 202.

[0139] refer to Figure 10A First, an initial image 202 can be captured and the values ​​of example pixels can be determined. For example purposes, the initial image 202 can be a red, green, and blue image (RGB image). The example pixels in the initial image 1002 can have a red channel (R) intensity of 0.5, a green channel (G) intensity of 0.4, and a blue channel (B) intensity of 0.3.

[0140] Then, the following formula can be used to generate the values ​​of sample pixels in the mixed grayscale image 204:

[0141] R*G*B=0.5*0.4*0.3=0.6

[0142] Next, the values ​​of atmospheric light can be calculated. First, a small subset of the brightest pixels in the mixed grayscale image 205 can be identified. For example, the brightest pixels in the mixed grayscale image 205 could be pixels that are among the top 1% of the brightest pixels in the mixed grayscale image 204. Next, the values ​​can be calculated using... Figure 5 The same method described herein is used to calculate the value of atmospheric light by determining the average intensity of each color channel in the color channels between each pixel in the initial image 202 (which corresponds to the brightest pixel in the mixed grayscale image 203) using Equation 6. (See reference...) Figure 10B The RGB representation of atmospheric light with a value of 1004 can include an intensity of 0.8 corresponding to the red channel, 0.8 corresponding to the green channel, and 0.9 corresponding to the blue channel.

[0143] Next, refer to Figure 10C This allows limiting the saturation of atmospheric light to a grayscale level. First, atmospheric light can be converted to a Hue / Saturation Value (HSV) representation of 1006. Then, an updated HSV representation of 1008 can be generated by limiting the saturation to values ​​less than or equal to 0.1. For example... Figure 10CAs illustrated in the diagram, the saturation value is limited to 0.1 for illustrative purposes. Then, based on the updated saturation value, the updated HSV representation can be converted to the updated RGB representation 1010, thereby producing the value of atmospheric light.

[0144] Next, refer to Figure 10D , can be Figure 2A The same method described herein generates the values ​​of the sample pixels in the brightened initial image 1012 by brightening the sample pixels using Equation 3. Figure 10D As illustrated, a brightening constant of 0.45 can be used. For example, the intensity of the red channel of a sample pixel in the initial image 1012 can be calculated using the following formula:

[0145]

[0146] Next, the values ​​of the sample pixels in the transmission map 210 can be generated by blending the color channels of the sample pixels in the brightened initial image 1012 and then inverting the result. In other words, the color channels of the brightened initial image 206 can be blended and inverted as follows:

[0147] t(x)=1-(R 增亮的图像 *G 增亮的图像 *B 增亮的图像 )

[0148] 1 - (0.8 * 0.7 * 0.6) = 0.66

[0149] Then, it can be used Figure 6 The values ​​of example pixels in the brightened transmission map 212 are generated using Equation 7 in the same manner described herein. In other words, the average pixel values ​​and brightening factors of the transmission map can be determined and used to brighten the pixels. Figure 8 The same method described herein uses Equation 15 to determine the brightening factor. For example, the average pixel value of the transmissive map, equal to 0.52, can be used. In other words, the brightening factor can be determined as follows:

[0150]

[0151]

[0152] Then, using the brightening factor, the values ​​of the example pixels in the brightened transmission map 212 can be determined as follows:

[0153] f(x) = t(x) b

[0154] 0.66 0.25 =0.90

[0155] Next, the pixel values ​​in the brightened blended grayscale image 208 can be adjusted based on the hue of atmospheric light to generate example pixel values ​​for the adjusted brightened blended grayscale image 214. Because the hue is outside the sky range, Equation 10 can be used to... Figure 6 The second adjustment factor can then be determined in the same manner as discussed in [the previous section]. Figure 6 The same method described herein is used to generate an adjusted brightened mixed grayscale image 214 by adjusting the intensity of pixels in the brightened mixed grayscale image 208 using Equation 11.

[0156] In other words, the second adjustment factor can be determined as follows:

[0157] adjust2 = -3.16h 2 +2h+0.6685

[0158] adjust2 = -3.16 * 0.5 2 +2*0.5+0.6685=0.91

[0159] Then, using the second adjustment factor, the values ​​of example pixels in the adjusted and brightened mixed grayscale image can be determined as follows:

[0160] (1-t(x))=adjust2*(1-t(x))

[0161] 0.91*(1-0.66)=0.31

[0162] Next, the following formula can be used to... Figure 6 The same method described herein generates example pixel values ​​for the intermediate scene transmission map 216 by combining and inverting the pixel values ​​in the brightened transmission map and the adjusted brightened mixed grayscale image using Equation 14:

[0163] T(x) = 1 - f(t(x)) * g(1 - t(x))

[0164] 1 - 0.90 * 0.31 = 0.72

[0165] Finally, it can be said that... Figure 2A The same method described in [the original text] uses Equation 5 to recover the scene brightness value of the example pixels in the initial image, thereby producing the dehazed pixels. For example... Figure 10C As illustrated in the figure, for illustrative purposes, only the recovered scene brightness of the red channel is shown using the following formula:

[0166]

[0167]

[0168] This calculation can be repeated for the green and blue channels of the example pixel.

[0169] Examples of image processing have been applied to test samples. The examples were used to image and process test samples with bright / dark areas. In the example, it was possible to image a target pattern comprising concentric circles with different brightness levels, and image processing could be applied. Advantageously, when applied to such test samples, the examples produced a clear transition between bright / dark areas and avoided introducing artificial artifacts. In contrast, when a region-based approach is used to perform haze removal preprocessing, the transition between bright / dark areas becomes diffused. Advantageously, the use of the pixel-by-pixel dehazing operation discussed in the embodiments of this disclosure avoids the introduction of such artifacts.

[0170] Various embodiments of this application realize practical applications, such as improving images by removing the effects of fog or haze, and otherwise improving contrast, which can be implemented directly in low-power devices, such as image sensors or low-power processors. In particular, embodiments of this application enable the use of low-power computing processors and sensors to perform the aforementioned operations to enhance images using pixel-based operations. Advantageously, this allows for significant technical improvements in image sensing and processing, as conventional algorithms are computationally expensive and therefore power-intensive or unable to achieve the same level of enhancement as the embodiments. Due to the low computational load of the various embodiments, processing can be performed within the image capture device while the image is being preprocessed.

[0171] Exemplary embodiments of the invention are outlined herein. Other embodiments can also be understood from the entire specification and claims set forth herein.

[0172] Example 1: A method for enhancing an image, the method comprising: capturing an initial image comprising a plurality of pixels; and performing a pixel-by-pixel dehazing operation for each of the plurality of pixels, the operation comprising: generating values ​​of a blended grayscale image based on the color channels of the pixel for each of the plurality of pixels; generating values ​​of a transmission map based on the blended grayscale image for each of the plurality of pixels; and generating output color channels of a processed image, which is an enhanced version of the initial image, based on the values ​​of the transmission map for each of the plurality of pixels.

[0173] Example 2: Following the method of Example 1, where capture and execution are performed at the image sensor chip.

[0174] Example 3: The method of generating a mixed grayscale image from the color channels of pixels, according to one of the examples in Example 1 or 2, involves multiplying the intensity of the color channels of the pixels together for each of the multiple pixels.

[0175] Example 4: According to the method of one of Examples 1 to 3, the pixel-by-pixel dehazing operation further includes: for each of the plurality of pixels, calculating the value of atmospheric light of the initial image from the mixed grayscale image; for each of the plurality of pixels, generating a value of the brightened initial image from the value of atmospheric light; for each of the plurality of pixels, generating a value of the brightened mixed grayscale image from the value of the brightened initial image; and for each of the plurality of pixels, generating a value of the transmissive map from the value of the brightened mixed grayscale image, wherein generating the output color channel of the processed image includes: for each of the plurality of pixels, generating a value of the intermediate scene transmissive map from the value of the transmissive map; and for each of the plurality of pixels, recovering the scene brightness value of the initial image based on the pixel values ​​in the atmospheric light, the initial image, and the intermediate scene transmissive map.

[0176] Example 5: A method according to one of Examples 1 to 4, wherein calculating the atmospheric light value of the initial image from the mixed grayscale image includes: for each of a plurality of pixels, determining the brightest pixel in the mixed grayscale image, wherein the brightest pixel in the mixed grayscale image is a pixel having a value within a small fraction of the values ​​in the mixed grayscale image; at each brightest pixel in the mixed grayscale image, determining the intensity of the color channel of the pixel in the initial image corresponding to the brightest pixel in the mixed grayscale image; and determining the average intensity of each color channel at the pixel in the initial image corresponding to the brightest pixel in the mixed grayscale image.

[0177] Example 6: According to the method of one of Examples 1 to 5, wherein calculating the atmospheric light value of the initial image from the value of the mixed grayscale image further includes: determining the saturation of the atmospheric light based on the value of the atmospheric light; limiting the saturation of the atmospheric light to a grayscale range, wherein limiting the saturation of the atmospheric light to grayscale levels includes: limiting the saturation of the atmospheric light to a value less than or equal to 0.1; and updating the value of the atmospheric light based on the limited saturation.

[0178] Example 7: A method according to one of Examples 1 to 6, wherein generating the value of the brightened initial image based on the value of atmospheric light includes: for each of a plurality of pixels, the intensity of each color channel of the pixel generating the brightened initial image, wherein the intensity of each color channel of the pixel generating the brightened initial image includes: for each of the plurality of pixels, dividing the intensity of each color channel of the pixel in the initial image by the intensity of the corresponding color channel of the atmospheric light; and for each of the plurality of pixels, applying a brightening constant to the result of the division.

[0179] Example 8: The method of one of Examples 1 to 7, wherein generating the transmission map value from the mixed grayscale channel includes: inverting the value of the brightened mixed grayscale image for each of the multiple pixels.

[0180] Example 9: The method of generating the brightened blended grayscale image according to one of Examples 1 to 8 includes: multiplying the intensity of each color channel in the color channel of each pixel in the brightened initial image with each other for each of the multiple pixels.

[0181] Example 10: A method according to one of Examples 1 to 9, wherein generating intermediate scene transmission map values ​​based on transmission map values ​​includes: generating a brightened transmission map value from the transmission map values ​​for each of a plurality of pixels; generating an adjusted brightened mixed grayscale image value from the brightened mixed grayscale image values ​​based on the hue of atmospheric light for each of the plurality of pixels; combining the brightened transmission map value with the adjusted brightened mixed grayscale image value for each of the plurality of pixels; and inverting the combined value of the brightened transmission map and the adjusted brightened mixed grayscale image for each of the plurality of pixels.

[0182] Example 11: A method according to one of Examples 1 to 10, wherein generating the value of the brightened transmissive map includes: determining an average value of the transmissive map for each of a plurality of pixels; determining a brightening factor based on the average value of the transmissive map for each of a plurality of pixels; and determining the value of the brightened transmissive map based on the brightening factor for each of a plurality of pixels.

[0183] Example 12: A method according to one of Examples 1 to 11, wherein generating adjusted values ​​for a brightened mixed grayscale image from values ​​of a brightened mixed grayscale image based on the hue of atmospheric light comprises: determining whether the hue of atmospheric light is within a possible sky range of an initial image, the possible sky range including hue values ​​above or below a non-sky range; determining a first adjustment factor based on determining that the hue of atmospheric light is below a non-sky range; or determining a second adjustment factor based on determining that the hue of atmospheric light is within a non-sky range, the second adjustment factor being different from the first adjustment factor; or determining a third adjustment factor based on determining that the hue of atmospheric light is above a non-sky range, the third adjustment factor being different from both the first and second adjustment factors; and for each of a plurality of pixels, determining values ​​for the adjusted brightened mixed grayscale image based on the first adjustment factor, the second adjustment factor, or the third adjustment factor.

[0184] Example 13: A method based on one of Examples 1 to 12, where restoring scene brightness includes: using the equation Where I(x) represents the color channel value of each pixel in the plurality of pixels, A represents atmospheric light, T(x) represents the intermediate scene transmissive map value of each pixel in the plurality of pixels, and J(x) represents the scene brightness value of the initial image of each pixel in the plurality of pixels.

[0185] Example 14: A method for enhancing an image, the method comprising: capturing an initial image comprising a plurality of pixels; generating a mixed grayscale image based on the initial image; generating a brightened mixed grayscale image from the brightened initial image; generating a transmissive map for each of the plurality of pixels; calculating an enhancement value based on determining how much haze is present in the initial image; adjusting the transmissive map based on the enhancement value; determining whether the hue of atmospheric light in the initial image is within a possible sky color range of the initial image, the possible sky color range including hue values ​​higher or lower than a non-sky color range; adjusting the transmissive map of each of the plurality of pixels using the enhancement value; adjusting the brightened mixed grayscale image based on determining whether the hue of atmospheric light in the initial image is within a possible sky color range; and, after adjusting the transmissive map, generating an enhanced image based on the adjusted transmissive map, the adjusted brightened mixed grayscale image, and the initial image.

[0186] Example 15: The method of Example 14, wherein determining how much haze is present in the initial image is based on the average value of atmospheric light captured in the initial image and the average value of the transmission map.

[0187] Example 16: A method according to one of Examples 14 or 15, wherein generating a blended grayscale image includes: determining the value of the blended grayscale image for each of the multiple pixels from the color channels of each of the multiple pixels by multiplying the intensity of the color channels of each pixel in the multiple pixels.

[0188] Example 17: A method according to one of Examples 14 to 16, wherein generating a brightened blended grayscale image includes: for each of a plurality of pixels, generating a value of a brightened initial image based on atmospheric light captured in an initial image and a global brightening factor; and for each of the plurality of pixels, multiplying the intensity of each color channel in the color channels of each pixel in the brightened initial image together.

[0189] Example 18: A method based on one of Examples 14 to 17, wherein generating a transmission map includes: inverting the value of the brightened mixed grayscale image for each of a plurality of pixels.

[0190] Example 19: A method according to one of Examples 14 to 18, wherein adjusting the brightened mixed grayscale image further includes: determining a first adjustment factor based on determining that the hue of atmospheric light is below the non-sky range color range; or determining a second adjustment factor based on determining that the hue of atmospheric light is within the non-sky range, the second adjustment factor being different from the first adjustment factor; or determining a third adjustment factor based on determining that the hue of atmospheric light is above the non-sky range, the third adjustment factor being different from both the first and second adjustment factors; and adjusting the value of the brightened mixed grayscale image based on the first adjustment factor, the second adjustment factor, or the third adjustment factor for each of a plurality of pixels.

[0191] Example 20: According to the method of one of Examples 14 to 19, generating an enhanced image based on an adjusted transmissive map, an adjusted brightened mixed grayscale image, and an initial image includes: multiplying the adjusted value of the transmissive map with the adjusted value of the brightened mixed grayscale image for each of the plurality of pixels; combining the adjusted value of the transmissive map with the adjusted value of the brightened mixed grayscale image for each of the plurality of pixels; determining an inverted value for each of the plurality of pixels by inverting the combined value of the adjusted transmissive map and the adjusted brightened mixed grayscale image for each of the plurality of pixels; and restoring scene brightness for each of the plurality of pixels based on the inverted value, the intensity of each color channel of the pixel in the initial image, and the intensity of each color channel of atmospheric light captured in the initial image.

[0192] Example 21: An image enhancement device comprising: an image sensor configured to capture an initial image; a memory configured to store the captured initial image; a display; and a processor communicating with the memory and the display, the processor being configured to execute instructions stored in the memory, the memory including instructions for: capturing an initial image comprising a plurality of pixels; and performing a pixel-by-pixel dehazing operation for each of the plurality of pixels, the execution comprising: generating values ​​of a mixed grayscale image from a color channel of the pixel for each of the plurality of pixels; generating values ​​of a transmission map from the mixed grayscale image for each of the plurality of pixels; and generating an output color channel of a processed image based on the values ​​of the transmission map for each of the plurality of pixels, the processed image being an enhanced version of the initial image.

[0193] Example 22: According to the device of Example 21, the instructions for generating values ​​of a mixed grayscale image based on the color channels of pixels include instructions for multiplying the intensities of the color channels of pixels with each other for each of a plurality of pixels.

[0194] Example 23: A device according to one of Examples 21 or 22, wherein the instructions for the pixel-by-pixel dehazing operation further include instructions for performing the following operations: for each of a plurality of pixels, calculating the value of atmospheric light of an initial image from a mixed grayscale image; for each of a plurality of pixels, generating a value of a brightened initial image from the value of atmospheric light; for each of a plurality of pixels, generating a value of a brightened mixed grayscale image from the value of the brightened initial image; and for each of a plurality of pixels, generating a value of a transmissive map from the value of the brightened mixed grayscale image, wherein generating an output color channel of the processed image includes: for each of a plurality of pixels, generating a value of an intermediate scene transmissive map from the value of the transmissive map; and for each of a plurality of pixels, recovering a scene brightness value of the initial image based on the pixel values ​​in the atmospheric light, the initial image, and the intermediate scene transmissive map.

[0195] Example 24: A device according to one of Examples 21 to 23, wherein the instructions for calculating the value of atmospheric light in an initial image from a mixed grayscale image include instructions for: determining, for each of a plurality of pixels, the brightest pixel in the mixed grayscale image, wherein the brightest pixel in the mixed grayscale image is a small fraction of the brightest pixels in the mixed grayscale image; at each brightest pixel in the mixed grayscale image, determining the intensity of a color channel of a pixel in the initial image corresponding to the brightest pixel in the mixed grayscale image; and determining the average intensity of each color channel at the pixel in the initial image corresponding to the brightest pixel in the mixed grayscale image.

[0196] Example 25: A device according to one of Examples 21 to 24, wherein the instructions for calculating the atmospheric light value of an initial image from the values ​​of a mixed grayscale image further include instructions for: calculating the atmospheric light value of the initial image from the values ​​of a mixed grayscale image further includes: determining the saturation of the atmospheric light based on the values ​​of the atmospheric light; limiting the saturation of the atmospheric light to a grayscale range, wherein limiting the saturation of the atmospheric light to a grayscale level includes: limiting the saturation of the atmospheric light to a value less than or equal to 0.1; and updating the values ​​of the atmospheric light based on the limited saturation.

[0197] Example 26: A device according to one of Examples 21 to 25, wherein the instructions for generating the values ​​of the transmission map based on the mixed grayscale channels include instructions for inverting the values ​​of the brightened mixed grayscale image for each of a plurality of pixels.

[0198] Example 27: A device according to one of Examples 21 to 26, wherein the instructions for generating values ​​of a brightened mixed grayscale image include instructions for multiplying the intensity of each color channel in the color channel of each pixel in the brightened initial image with each other for each of a plurality of pixels.

[0199] Example 28: A device according to one of Examples 21 to 27, wherein the instructions for generating intermediate scene transmission map values ​​based on the transmission map values ​​include instructions for: generating a brightened transmission map value from the transmission map values ​​for each of a plurality of pixels; generating an adjusted brightened mixed grayscale image value from the brightened mixed grayscale image values ​​based on the hue of atmospheric light for each of the plurality of pixels; and combining the brightened transmission map value with the adjusted brightened mixed grayscale image value for each of the plurality of pixels, and inverting the brightened transmission map value with the adjusted brightened mixed grayscale image value.

[0200] Example 29: A device according to one of Examples 21 to 28, wherein the instructions for generating the value of the brightened transmissive map include instructions for performing the following operations: determining an average value of the transmissive map for each of a plurality of pixels; determining a brightening factor based on the average value of the transmissive map for each of the plurality of pixels; and determining the value of the brightened transmissive map based on the brightening factor for each of the plurality of pixels.

[0201] Example 30: A device according to one of Examples 21 to 29, wherein the instructions for generating adjusted values ​​of a brightened mixed grayscale image from values ​​of a brightened mixed grayscale image based on the hue of atmospheric light include instructions for: determining a first adjustment factor based on determining that the hue of atmospheric light is below a non-sky color range; or determining a second adjustment factor based on determining that the hue of atmospheric light is within a non-sky color range, the second adjustment factor being different from the first adjustment factor; or determining a third adjustment factor based on determining that the hue of atmospheric light is above a non-sky color range, the third adjustment factor being different from the first and second adjustment factors; and adjusting the values ​​of the brightened mixed grayscale image based on the first adjustment factor, the second adjustment factor, or the third adjustment factor for each of a plurality of pixels.

[0202] Example 31: A method for enhancing an image includes: generating a mixed grayscale image from an initial image captured at an image sensor, the initial image including a plurality of pixels, the initial image including a first region and a second region, the first region including an object to be viewed, the second region including haze to be removed from the initial image, the mixed grayscale image being generated by multiplying the values ​​of the color channels of each of the plurality of pixels; and removing haze from the second region by performing a pixel-by-pixel operation based on the mixed grayscale image.

[0203] Example 32: According to the method of Example 31, the method further includes: calculating the average value of atmospheric light captured in the initial image based on a portion of the brightest pixels in the mixed grayscale image.

[0204] Example 33: According to one of the examples 31 to 32, the method further includes: determining the amount of haze present in a second region of the initial image based at least in part on the average value of atmospheric light captured in the initial image, wherein the average value of atmospheric light captured in the initial image is determined based on a portion of the brightest pixels in the mixed grayscale image.

[0205] Example 34: According to the method of one of Examples 31 to 33, the method further includes: determining the average value of atmospheric light captured in the initial image based on the mixed grayscale image, and limiting the saturation of the average value of atmospheric light in the initial image to a value less than or equal to 0.01.

[0206] Example 35: According to the method of one of Examples 31 to 34, wherein removing haze from a second region of an initial image by performing a pixel-by-pixel operation further includes limiting dehazing to each of a plurality of pixels in a transmission map having a moderate transmission value, wherein the transmission map is generated based on a mixed grayscale image.

[0207] Example 36: The method according to one of the examples 31 to 35, wherein removing haze from a second region of the initial image by performing a pixel-by-pixel operation further includes limiting dehazing based on the hue of the average atmospheric light captured in the initial image.

[0208] While the invention has been described with reference to illustrative embodiments, this specification is not intended to be limiting. Various modifications and combinations of the illustrative embodiments and other embodiments of the invention will be apparent to those skilled in the art upon reference to this specification. Therefore, the appended claims are intended to cover any such modifications or embodiments.

Claims

1. A method for enhancing an image, the method comprising: Capture an initial image comprising multiple pixels; as well as Perform a pixel-by-pixel dehazing operation for each of the plurality of pixels, the operation including: For each of the plurality of pixels, the values ​​of the mixed grayscale image are generated by multiplying the color channels of the pixels in the initial image together. For each of the plurality of pixels, the value of atmospheric light in the initial image is calculated based on a subset of the brightest pixels in the mixed grayscale image; For each of the plurality of pixels, the value of the brightened initial image is generated from the atmospheric light, the value of the pixel in the initial image, and the global brightening constant; For each of the plurality of pixels, the value of the brightened mixed grayscale image is generated by multiplying the color channels of the pixels of the brightened initial image together; For each of the plurality of pixels, the value of the transmission map is generated by inverting the pixel values ​​of the brightened mixed grayscale image, and For each of the plurality of pixels, an output color channel of the processed image is generated, wherein generating the output color channel includes: For each of the plurality of pixels, the value of the intermediate scene transmission map is generated by inverting the value of the brightened transmission map and the value of the adjusted brightened mixed grayscale image; and For each of the plurality of pixels, the scene brightness value of the initial image is recovered based on the pixel values ​​in the atmospheric light, the initial image, and the intermediate scene transmission map. The processed image is an enhanced version of the initial image.

2. The method of claim 1, wherein the capture and the execution are performed at the image sensor chip.

3. The method of claim 1, wherein generating the values ​​of the mixed grayscale image from the color channels of the pixels comprises: For each of the plurality of pixels, the intensity of the pixel's color channels is multiplied together.

4. The method of claim 1, wherein calculating the atmospheric light value of the initial image based on the portion of the brightest pixel of the mixed grayscale image comprises: For each of the plurality of pixels, determine the brightest pixel in the mixed grayscale image, wherein the brightest pixel in the mixed grayscale image is a pixel having a value within a small subset of values ​​in the mixed grayscale image; At each of the brightest pixels in the mixed grayscale image, the intensity of the color channel of a pixel in the initial image is determined, the pixel in the initial image corresponding to the brightest pixel in the mixed grayscale image; as well as Determine the average intensity of each color channel at the pixels in the initial image, where the pixels in the initial image correspond to the brightest pixel in the mixed grayscale image.

5. The method of claim 4, wherein calculating the value of the atmospheric light in the initial image based on the portion of the brightest pixel of the mixed grayscale image further comprises: The saturation of the atmospheric light is determined based on the value of the atmospheric light. Limiting the saturation of the atmospheric light to a grayscale range, wherein limiting the saturation of the atmospheric light to a grayscale level includes limiting the saturation of the atmospheric light to a value less than or equal to 0.1; as well as The atmospheric light value is updated based on the limited saturation.

6. The method of claim 1, wherein generating the values ​​of the brightened initial image from the values ​​of the pixels in the initial image, the atmospheric light, and the global brightening constant comprises: For each of the plurality of pixels, the intensity of each color channel in the color channels of the pixel generating the brightened initial image is determined, wherein the intensity of each color channel in the color channels of the pixel generating the brightened initial image includes: For each of the plurality of pixels, the intensity of each color channel of the pixel in the initial image is divided by the intensity of the corresponding color channel of the atmospheric light; and For each of the plurality of pixels, a brightening constant is applied to the result of the division.

7. The method of claim 1, wherein generating the values ​​of the brightened mixed grayscale image comprises: For each of the plurality of pixels, the intensity of each color channel in the color channel of each pixel in the brightened initial image is multiplied together, and wherein generating the value of the intermediate scene transmission map from the value of the transmission map includes: For each of the plurality of pixels, the value of the brightened transmission map is generated from the value of the transmission map; For each of the plurality of pixels, the adjusted value of the brightened mixed grayscale image is generated from the value of the brightened mixed grayscale image based on the hue of the atmospheric light; For each of the plurality of pixels, the value of the brightened transmission map is combined with the value of the adjusted brightened mixed grayscale image; and For each of the plurality of pixels, the combined value of the brightened transmission image and the adjusted brightened mixed grayscale image is inverted.

8. The method of claim 7, wherein generating the values ​​of the enhanced transmittance map comprises: For each of the plurality of pixels, determine the average value of the transmission map; For each of the plurality of pixels, a brightening factor is determined based on the average value of the transmission map; as well as For each of the plurality of pixels, the value of the enhanced transmission map is determined based on the enhancement factor.

9. The method of claim 7, wherein generating the adjusted brightened mixed grayscale image values ​​from the values ​​of the brightened mixed grayscale image based on the hue of the atmospheric light comprises: Determine whether the hue of the atmospheric light is within the possible sky range of the initial image, the possible sky range including hue values ​​that are higher or lower than the non-sky range; A first adjustment factor is determined based on the determination that the hue of the atmospheric light is lower than that of the non-sky range; or A second adjustment factor is determined based on the determination of the hue of the atmospheric light within the non-sky range, and the second adjustment factor is different from the first adjustment factor; or A third adjustment factor is determined based on the determination that the hue of the atmospheric light is higher than the non-sky range, and the third adjustment factor is different from the first adjustment factor and the second adjustment factor; as well as For each of the plurality of pixels, the value of the adjusted and brightened mixed grayscale image is determined based on the first adjustment factor, the second adjustment factor, or the third adjustment factor.

10. The method of claim 1, wherein restoring scene brightness comprises: Using equations ,in Let A represent the color channel value of each of the plurality of pixels, and let A represent the atmospheric light value in the initial image. This represents the value of the intermediate scene transmission map for each of the plurality of pixels, and Each of the plurality of pixels represents the scene brightness value of the initial image.

11. A method for enhancing an image, the method comprising: A mixed grayscale image is generated from an initial image captured at an image sensor, the initial image comprising multiple pixels, the initial image comprising a first region and a second region, the first region comprising an object to be viewed, and the second region comprising haze to be removed from the initial image, the mixed grayscale image being generated by multiplying the values ​​of the color channels of each of the multiple pixels; as well as Based on the mixed grayscale image, the haze is removed from the second region by performing a pixel-by-pixel operation, the operation including: The average value of atmospheric light captured in the initial image is calculated based on a subset of the brightest pixels in the mixed grayscale image; For each of the plurality of pixels, the value of the brightened initial image is generated from the average value of the atmospheric light, the value of the pixel in the initial image, and the global brightening constant; For each of the plurality of pixels, the value of the brightened mixed grayscale image is generated by multiplying the color channels of the pixels of the brightened initial image together; For each of the plurality of pixels, the value of the transmission map is generated by inverting the value of the pixel in the brightened mixed grayscale image; For each of the plurality of pixels, an intermediate scene transmission map value is generated by combining and inverting the values ​​of the brightened transmission map and the adjusted brightened mixed grayscale image; the brightened transmission map is generated from the transmission map; and For each of the plurality of pixels, the scene brightness value of the initial image is recovered based on the average value of the atmospheric light, the pixel value in the initial image and the intermediate scene transmission map, thereby removing the haze from the second region.

12. The method of claim 11, wherein the execution further comprises: The amount of haze present in the second region of the initial image is determined at least in part based on the average value of atmospheric light captured in the initial image, and the brightening factor is adjusted according to the amount of haze. And the enhanced transmission map is generated from the transmission map according to the adjusted enhancement factor.

13. The method of claim 11, wherein the execution further comprises: The saturation of the atmospheric light, determined based on the average value of the atmospheric light in the initial image, is limited to a value less than or equal to 0.1; And update the average value of the atmospheric light based on the limited saturation.

14. The method of claim 11, wherein the value of the intermediate scene transmission map is generated by combining and inverting the value of the brightened transmission map and the value of the adjusted brightened mixed grayscale image, such that dehazing is limited to each of a plurality of pixels in the transmission map having a moderate transmission value.

15. The method of claim 11, wherein the execution further comprises: An adjustment factor is determined based on the hue of the average atmospheric light captured in the initial image to limit dehazing, wherein the adjustment factor is used to determine the value of the adjusted brightened mixed grayscale image from the value of the brightened mixed grayscale image.

16. An image enhancement device, comprising: An image sensor is configured to capture an initial image; A memory is configured to store the captured initial image; as well as A processor, communicating with the memory, is configured to execute instructions stored in the memory, the memory including instructions for: Capture an initial image comprising multiple pixels; as well as Perform a pixel-by-pixel dehazing operation for each of the plurality of pixels, the operation including: For each of the plurality of pixels, the values ​​of the mixed grayscale image are generated by multiplying the color channels of the pixels in the initial image together. For each of the plurality of pixels, the value of atmospheric light in the initial image is calculated based on a subset of the brightest pixels in the mixed grayscale image; For each of the plurality of pixels, the value of the brightened initial image is generated from the atmospheric light, the value of the pixel in the initial image, and the global brightening constant; For each of the plurality of pixels, the value of the brightened mixed grayscale image is generated by multiplying the color channels of the pixels of the brightened initial image together; For each of the plurality of pixels, the value of the transmission map is generated by inverting the pixel values ​​of the brightened mixed grayscale image, and For each of the plurality of pixels, an output color channel of the processed image is generated, wherein generating the output color channel includes: For each of the plurality of pixels, the value of the intermediate scene transmission map is generated by inverting the value of the brightened transmission map and the value of the adjusted brightened mixed grayscale image; and For each of the plurality of pixels, the scene brightness value of the initial image is recovered based on the pixel values ​​in the atmospheric light, the initial image, and the intermediate scene transmission map. The processed image is an enhanced version of the initial image.

17. The apparatus of claim 16, wherein the instructions for generating values ​​of the mixed grayscale image based on the color channels of the pixels include instructions for multiplying the intensities of the color channels of the pixels together for each of the plurality of pixels.

Images