Correction of color-tinted pixels captured in low-light conditions

Abstract

Aspects of the present disclosure relate to color correction in an image processing pipeline. An example method may include receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels, determining that the first image data corresponds to an original image captured in a dark environment, generating second image data by performing one or more tone mapping operations on the first image data, the second image data corresponding to current luminance data and current chrominance data for each of the plurality of pixels, and generating output image data. For each of the plurality of pixels, the output data may include an output luminance value of a corresponding pixel of the current luminance data, and a chrominance value of a corresponding pixel of selected data from the reference chrominance data and the current chrominance data, the selection being based at least in part on the reference chrominance data and the current chrominance data.

Description

Technical Field

[0001] The present disclosure relates generally to systems and methods for image capture, and particularly to pipeline processing of captured images. Background Art

[0002] Abnormal black levels in an image sensor, such as a complementary metal oxide semiconductor (CMOS) image sensor, may cause a visible reddish tint to appear in black areas of an image. Color mismatches in the image sensor may be amplified by an image signal processor (ISP). For example, digital gain or dynamic gain compression (DRC) performed by the ISP may amplify the color mismatch, resulting in such a reddish tint. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts that are further described below in the Detailed Description in a simplified form. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The systems, methods, and devices of the present disclosure each have multiple innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

[0004] One innovative aspect of the subject matter described in the present disclosure can be implemented as a method for color correction. The method can be performed by a suitable device and can include: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data corresponds to an original image captured in a dark environment; generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and generating output image data to include, for each of the plurality of pixels: an output luminance value of a corresponding pixel of the current luminance data; and a chrominance value of the corresponding pixel from a selected one of the reference chrominance data and the current chrominance data, the selection being based at least in part on the reference chrominance data and the current chrominance data.

[0005] In some aspects, the first image data and the second image data are in YUV format, the reference chromaticity data includes reference U data and reference V data, and the current chromaticity data includes current U data and current V data. In some aspects, for each of the plurality of pixels, when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as being color tinted, the output image data includes the reference chromaticity data instead of the current chromaticity data. In some aspects, detecting that a pixel in the first image data is gray includes: determining that an absolute difference between the reference U data and a midpoint U value is less than a reference threshold, and determining that an absolute difference between the reference V data and a midpoint V value is less than a reference threshold. In some aspects, detecting that a pixel in the second image data is color tinted includes: determining that the corresponding absolute differences between the current U data and the midpoint U value and between the current V data and the midpoint V value are both greater than a current minimum value and less than a current maximum value.

[0006] In some aspects, generating the second image data further comprises performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. In some aspects, determining that the first image data corresponds to an original image captured in a dark environment is determined by determining that an automatic exposure control (AEC) gain associated with the original image is greater than a threshold AEC gain.

[0007] Another innovative aspect of the subject matter described in the present disclosure can be implemented in an apparatus including a memory and one or more processors. The one or more processors are configured to: receive first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determine that the first image data corresponds to an original image captured in a dark environment; generate second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and generate output image data to include, for each of the plurality of pixels: an output luminance value of a corresponding pixel of the current luminance data; and a chrominance value of the corresponding pixel from a selected one of the reference chrominance data and the current chrominance data, the selection being based at least in part on the reference chrominance data and the current chrominance data.

[0008] In some aspects, the first image data and the second image data are in YUV format, the reference chromaticity data includes reference U data and reference V data, and the current chromaticity data includes current U data and current V data. In some aspects, for each of the plurality of pixels, when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is color-tinted, the output image data includes the reference chromaticity data instead of the current chromaticity data. In some aspects, detecting that the pixel in the first image data is gray includes: determining that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold, and determining that the absolute difference between the reference V data and the midpoint V value is less than a reference threshold. In some aspects, detecting that the pixel in the second image data is color-tinted includes determining that the corresponding absolute differences between the current U data and the midpoint U value and between the current V data and the midpoint V value are both greater than a current minimum value and less than a current maximum value.

[0009] In some aspects, generating the second image data further comprises performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. In some aspects, the one or more processors are configured to determine that the first image data corresponds to an original image captured in a dark environment by determining that an automatic exposure control (AEC) gain associated with the original image is greater than a threshold AEC gain.

[0010] Another innovative aspect of the subject matter described in the present disclosure can be implemented in a non-transitory computer-readable storage medium storing instructions executed by one or more processors of an image processing device. Execution of the instructions causes the image processing device to perform operations, including: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data corresponds to an original image captured in a dark environment; generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and generating output image data to include, for each of the plurality of pixels: an output luminance value of a corresponding pixel of the current luminance data; and a chrominance value of the corresponding pixel from a selected one of the reference chrominance data and the current chrominance data, the selection being based at least in part on the reference chrominance data and the current chrominance data.

[0011] In some aspects, the first image data and the second image data are in YUV format, the reference chromaticity data includes reference U data and reference V data, and the current chromaticity data includes current U data and current V data. In some aspects, for each of a plurality of pixels, when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as being color-tinted, the output image data includes the reference chromaticity data instead of the current chromaticity data. In some aspects, detecting that the pixel in the first image data is gray includes: determining that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold, and determining that the absolute difference between the reference V data and the midpoint V value is less than the reference threshold. In some aspects, detecting that the pixel in the second image data is color-tinted includes determining that the corresponding absolute differences between the current U data and the midpoint U value and between the current V data and the midpoint V value are both greater than a current minimum value and less than a current maximum value.

[0012] In some aspects, generating the second image data further comprises performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. In some aspects, determining that the first image data corresponds to an original image captured in a dark environment is determined by determining that an automatic exposure control (AEC) gain associated with the original image is greater than a threshold AEC gain.

[0013] Another innovative aspect of the subject matter described in this disclosure can be implemented in an image processing pipeline. The image processing pipeline may include: a raw image processing unit for receiving a raw image and performing one or more image processing operations on the raw image; a color correction matrix (CCM) unit for performing one or more CCM operations on the processed raw image; a denoising unit for performing one or more denoising operations on the output of the CCM unit to generate first image data corresponding to reference luminance data and reference chrominance data of each of a plurality of pixels; a tone mapping unit for generating second image data by performing one or more tone mapping operations on the first image data, the second image data corresponding to current luminance data and current chrominance data of each of the plurality of pixels; and a color correction unit, which is configured to: receive the first image data and the second image data; determine that the first image data corresponds to a raw image captured in a dark environment; and generate output image data, the output image data including, for each of the plurality of pixels: an output luminance value of a corresponding pixel of the current luminance data; and a chrominance value of the corresponding pixel from a selected one of the reference chrominance data and the current chrominance data, the selection being based at least in part on the reference chrominance data and the current chrominance data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Aspects of the disclosure are illustrated by way of example, and not by way of limitation, in the accompanying drawings and in which like references refer to similar elements.

[0015] Figure 1 is a block diagram of an example device configured to perform color correction in accordance with some implementations.

[0016] Figure 2 An example process flow for detecting and correcting anomalous colorized pixels in a captured image is shown in accordance with some implementations.

[0017] Figure 3 An illustrative flow chart showing example operations for processing image data in a color correction stage is shown in accordance with some implementations.

[0018] Figure 4 is an illustrative flow chart depicting example operations for color correction in an image processing pipeline, according to some implementations. DETAILED DESCRIPTION

[0019] Aspects of the present disclosure may be used to detect and correct abnormal color-tinted pixels in images captured under low-light conditions. Abnormal color-tinted pixels in an image may be caused by the output of an image sensor indicating little or no light (which may be intended to be represented by a dark black). When processing the output from the image sensor, the color tinting may be amplified by a digital gain or DRC performed by an ISP. Such abnormal color tinting may result in black pixels of an image including a visible reddish tint. Conventional techniques for correcting such color-tinted pixels may have a high false positive rate when detecting color-tinted pixels, resulting in color degradation in one or more areas of the image that do not include color-tinted pixels. Conventional techniques for correcting such color-tinted pixels may additionally or alternatively have a high false negative rate when detecting color-tinted pixels, resulting in a final image including pixels with uncorrected reddish tints. Conventional techniques typically process raw image data. Since the use of black level correction may not effectively suppress the reddish tint in black pixels, and the raw image data of red and green colors may be similar, the final image may include one or more black portions with alternating reddish and greenish color-tinted pixels.

[0020] Methods and devices of example implementations of the present disclosure can detect and correct pixels that exhibit such undesirable color tints of pixels of an image captured in a low-light environment. For example, a device can detect one or more pixels that are converted from gray pixels to color-tinted pixels at a stage in an image processing pipeline. When such pixels are detected, the color tint can be removed by replacing each such pixel with a corresponding pixel from an earlier stage in the image processing pipeline. For example, the color tint may be caused or amplified by one or more tone mapping operations in the image processing pipeline. Replacing the color-tinted pixels with corresponding gray pixels from an earlier stage may include replacing tone-mapped color-tinted pixels with corresponding pixels from a stage in the image processing pipeline prior to the one or more tone mapping operations.

[0021] While conventional color correction techniques may operate on raw image data, some exemplary embodiments of the present disclosure may operate on image data in a YUV color space, where Y refers to brightness, U refers to blue projected chromaticity, and V refers to red projected chromaticity. Other implementations may operate in other similar brightness / chromaticity color spaces. Using such a color space allows example implementations to focus color correction directly on chromaticity without affecting the brightness of the output pixels.

[0022] In the following description, many specific details, such as examples of specific components, circuits and processes are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific naming is set forth to provide a thorough understanding of the present disclosure. However, it is obvious to those skilled in the art that these specific details may not be necessary for practicing the teachings disclosed herein. In other cases, well-known circuits and devices are shown in block diagram form to avoid blurring the teachings of the present disclosure. Some parts of the following detailed description are presented according to other symbolic representations of data bit operations in processes, logic blocks, processing and computer memories. In the present disclosure, processes, logic blocks, processes, etc. are considered to be self-consistent steps or instruction sequences that lead to desired results. These steps are those steps that require physical manipulation of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared and otherwise manipulated in a computer system.

[0023] It should be remembered, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically indicated clearly from the following discussion, it should be understood that throughout this application, discussions using terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "solve," etc., refer to the behavior and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the computer system's registers and memories into other data similarly represented as physical quantities in the computer system's memories or registers or other such information storage, transmission, or display devices.

[0024] In the figure, a single block may be described as performing one or more functions; however, in actual practice, one or more functions performed by the block may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described below according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing deviations from the scope of the present disclosure. In addition, the example device may include components other than the components shown, including well-known components, such as processors, memories, etc.

[0025] Aspects of the present disclosure are applicable to any suitable electronic device configured or capable of capturing images or video (e.g., security systems having one or more cameras, smartphones, tablets, laptops, digital video and / or still cameras, web cameras, etc.). Although described below with respect to devices having or coupled to one camera, aspects of the present disclosure are applicable to devices having any number of cameras and are therefore not limited to devices having one camera. Aspects of the present disclosure are applicable to cameras configured to capture still images as well as to capture video, and may be implemented in devices having or coupled to cameras of different capabilities.

[0026] The term "device" is not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects.

[0027] Figure 1 1 is a block diagram of an example device 100 configured to perform color correction according to some embodiments of the present disclosure. The example device 100 may include or be coupled to a camera 102, a processor 104, a memory 106 storing instructions 108, and a controller 110. In some embodiments, the device 100 may include (or be coupled to) a display 114, and a plurality of input / output (I / O) components 116. The device 100 may also include a power supply 118, which may be coupled to the device 100 or integrated into the device 100. The device 100 may include additional features or components not shown. For example, a wireless interface for a wireless communication device may be included, which may include a plurality of transceivers and a baseband processor. In another example, the device 100 may include or be coupled to an additional camera other than the camera 102.

[0028] The camera 102 may be any suitable camera capable of capturing still images (e.g., individually captured image frames) and / or capturing video (e.g., continuously captured image frames). The camera 102 may include a single image sensor or may be a dual camera module or any other suitable module having one or more image sensors, such as one or more CMOS image sensors.

[0029] The memory 106 may be a non-transitory or non-transitory computer-readable medium that stores computer-executable instructions 108 to perform all or part of one or more operations described in the present disclosure. The processor 104 may be one or more suitable processors capable of executing scripts or instructions (e.g., instructions 108) of one or more software programs stored in the memory 106. In some aspects, the processor 104 may be one or more general-purpose processors that execute the instructions 108 to cause the device 100 to perform any number of functions or operations. In additional or alternative aspects, the processor 104 may include an integrated circuit or other hardware to perform functions or operations without the use of software.

[0030] Although in Figure 1 104, the processor 104, memory 106, controller 110, optional display 114, and optional I / O components 116 may be coupled to each other in various arrangements. For example, the processor 104, memory 106, controller 110, optional display 114, and / or optional I / O components 116 may be coupled to each other via one or more local buses (not shown for simplicity).

[0031] The display 114 may be any suitable display or screen that allows user interaction and / or presents items (e.g., captured images, videos, or preview images) for viewing by the user. In some aspects, the display 114 may be a touch-sensitive display. The I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (e.g., commands) from a user and provide output to the user. For example, the I / O component 116 may include (but is not limited to) a graphical user interface, a keyboard, a mouse, a microphone, and speakers, etc. The display 114 and / or the I / O component 116 may provide a preview image to the user and / or receive user input for adjusting one or more settings of the camera 102.

[0032] The controller 110 may include one or more controllers. The one or more controllers may be configured to control the camera 102. The controller 110 may include an image signal processor (ISP) 112, which may be one or more image signal processors to process captured image frames or videos provided by the camera 102. In some aspects, the controller 110 or ISP 112 may execute instructions from a memory (such as instructions 108 from the memory 106 or instructions stored in a separate memory coupled to the ISP 112). In other aspects, the controller 110 or ISP 112 may include specific hardware. The controller 110 or ISP 112 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions. For example, the ISP 112 may include or otherwise be configured to implement one or more stages of an image processing pipeline.

[0033] Figure 2 An example process flow 200 for detecting and correcting incorrectly colored pixels in a captured image is shown in accordance with some embodiments of the present disclosure. In some aspects, the process flow 200 may include: Figure 1 The image processing pipeline implemented or controlled by the device 100 is shown. Although the process flow 200 may be described as being performed by the device 100, the process flow 200 may also be performed by any other suitable image capture device.

[0034] about Figure 2, raw image data may be an input to the processing flow 200. Such raw image data may be received, for example, from one or more image sensors included in or coupled to the device 100, or from an analog front end of the device 100. In the raw image processing stage 202, the device 100 may perform one or more operations on the raw image data (e.g., one or more operations for black level correction, lens shading correction (LSC), auto focus, auto exposure, auto white balance, etc.). After the image data is processed in the raw image processing stage 202, the device 100 may process the image data in a color correction matrix (CCM) stage 204. The processed raw image data may be converted into a red green blue (RGB) color space and then proceed to the CCM stage 204. Alternatively, the device 100 may convert the data into an RGB color space in the CCM stage 204. In the CCM stage 204, the device 100 may perform one or more operations on the image data in the RGB color space (e.g., one or more operations for color correction in the RGB color space). In some examples, the device 100 may transform the RGB pixel values ​​of the image data into a defined and known color space represented by a color correction matrix. After the device 100 processes the image data in the CCM stage 204, the device 100 processes the image data in the denoising stage 206, performing one or more denoising operations on the image data. In some implementations, the denoising stage 206 may process the image data in a YUV color space, for example, the device 100 may convert the image data from the RGB color space to a YUV color space before or in the denoising stage 206. The output image data from the denoising stage 206 may be referred to as reference image data. For example, the output YUV image data representing the image data in the YUV color space may include reference Y data representing the brightness of the image data and reference U data and reference V data representing the chrominance of the image data.

[0035] After the denoising stage 206, the process flow 200 is divided into multiple paths (e.g. Figure 2). Discussing first the upper process 220, the device 100 processes the image data in a local tone mapping (LTM) stage 208. In the LTM stage 208, the device 100 may perform one or more tone mapping operations on the image data. In some aspects, the LTM stage 208 may operate on the image data in an RGB color space. Therefore, before or as part of the LTM stage 208, the device 100 may convert the image data into an RGB color space. Tone mapping may adjust the brightness of different regions of the image data or combine different exposures to increase local contrast between different regions of the scene. In contrast to global or spatially uniform tone mapping, local tone mapping may adjust the brightness of a given pixel based on local features of the image (such as the vicinity). After the LTM stage 208, the device 100 may process the image data in a gamma compression stage 210. In the gamma compression stage 210, the device 100 may perform one or more operations to increase the exposure of underexposed portions of the image data while reducing the exposure of overexposed portions of the image data. In some aspects, device 100 can apply a gamma function. For example, device 100 can apply a gamma filter, such as Where 0<γ<1 and A>0. When applying the gamma function, the device 100 converts [0,A -1 / γ The input brightness Y in the domain of in The output brightness (Y out ). In the example gamma filter, γ can be adjusted to adjust the contrast of the image, with lower values ​​corresponding to lower contrast and increased exposure in underexposed portions of the image. A can be set to less than 1 to ensure that the exposure of one or more portions of the image is reduced enough to prevent them from being overexposed.

[0036] After the gamma compression stage 210, the device 100 may process the image data in a sharpness stage 212, which may perform one or more sharpness enhancement operations on the image data, such as an edge enhancement operation, a deblurring filtering operation, or the like. In some aspects, the device 100 may process the image data in a YUV color space in the sharpness stage 212. Thus, before or as part of the sharpness stage 212, the device 100 may convert the image data into a YUV color space. The output image data of the sharpness stage 212 may be referred to as current image data. For example, the output YUV image data representing the image data in the YUV color space may include current Y data representing the brightness of the image data and current U data and current V data representing the chrominance of the image data. The current Y data may be provided to a combiner 218. The current U data and the current V data may be provided to a color correction stage 216.

[0037] In the color correction stage 216, the device 100 can selectively output reference U data and reference V data or current U data and current V data for each pixel of the image data. As described above, for images captured in low light or dark environments, some gray pixels may become color-colored pixels during image processing, which may cause a reddish color to appear in black areas of the image. This may usually be caused by a tone mapping operation. As discussed in more detail below, the color correction stage 216 can detect when the captured image appears to be captured in a sufficiently dark environment. For such an image, the device 100 can detect when the gray pixel becomes a color-colored pixel (e.g., by comparing the reference U and V data with the corresponding current U and V data). For such a pixel, the device 100 can replace the current U and V data with the corresponding reference U and V data. In this way, the image processing steps that cause abnormal color can be undone. The color correction stage 216 can output the corrected U data and the corrected V data to the combiner 218. The combiner 218 combines the current brightness data with the corrected U data and the corrected V data to generate corresponding Y, U, and V data for output image data. Therefore, the output image data reflects the current luminance (Y) data and the corrected chrominance (U and V) data.

[0038] Figure 3 An illustrative flowchart is shown that illustrates example operations 300 for processing image data in a color correction phase according to some embodiments of the present disclosure. In some aspects, operations 300 may be performed by device 100 at Figure 2 The color correction stage 216 of the image processing system may be performed. The reference U and V data from the denoising stage 206 and the current U and V data from the sharpness stage 212 may be used to generate corrected U and V data for the image. As described above, the reference U and V data may correspond to image data before tone mapping has been performed. The current U and V data may correspond to image data after tone mapping has been performed.

[0039] Abnormal color tints often appear in images captured in dark environments. Therefore, the device 100 makes a threshold determination 302 as to whether the image data meets a threshold darkness level. An example threshold darkness level may include an automatic exposure correction gain threshold. For example, the device 100 may determine whether the total automatic exposure correction gain of the captured image exceeds the automatic exposure correction gain threshold. As described above, such automatic exposure correction may occur as part of the raw image processing stage 202. Another example threshold darkness level may include a camera flash enable threshold. For example, the device 100 may determine whether the environment is dark enough to cause the camera flash to be abandoned. Another example threshold darkness level may be a light sensor threshold. For example, the device 100 may determine whether a light sensor included in or coupled to the device 100 senses less than a threshold amount of light. If the threshold darkness level is not met, the operation 300 may output the current U data and the current V data as corrected U data and corrected V data, and the process may end.

[0040] If a threshold darkness level is met, the image is a candidate for replacing the current U and V data of one or more pixels with reference U and V data. Operation 300 can proceed to the chrominance replacement step 310, where the device 100 can perform a series of determinations for each pixel of the current and reference U and V data. For each pixel, the device 100 can determine in block 311 whether the pixel is gray in the reference U and V data. In some aspects, the determination can include determining that the absolute difference between the reference U data and the midpoint (“midpoint U value”) within the possible range of U values is less than a reference threshold and that the absolute difference between the reference V data and the midpoint (“midpoint V value”) within the possible range of V values is also less than the reference threshold. Thus, when the U value and V value of a pixel are respectively close enough to the midpoint U value and midpoint V value, the pixel is determined to be gray. Thus, a pixel is gray when (|REF U - MID U| < REF_TH) and (|REF V - MID V| < REF_TH), where REF U is the reference U data, REF V is the reference V data, MID U is the midpoint U value, and MID V is the midpoint V value. For example, if the possible range of U and V values is between 0 and 256, the midpoint U and V values are each 128. Determining whether a pixel is gray can include determining whether (|REF U - 128| < REF_TH) and (|REF V - 128| < REF_TH). In some embodiments, the reference threshold can be a single-digit reference threshold, and the ranges of U and V are between 0 and 256. In one example, the reference threshold can be 4. In some embodiments, determining that a pixel in the reference data is gray can also include determining that the absolute difference between the reference U data and the reference V data is less than the reference threshold, i.e., determining |REF U - REF V| < REF_TH. If each of these absolute differences is less than the reference threshold, the pixel is considered gray. If the pixel is not considered gray in the reference U and V data, the current U and current V data are retained in the corrected U and V data of the pixel. If the pixel is gray in the reference U and V data, the device 100 can determine in block 312 whether the pixel is color-shaded in the current U and V data.

[0041] Note that in some embodiments, the reference threshold can vary at least in part based on the automatic exposure control (AEC) gain and / or the dynamic range compression (DRC) gain. For example, for a larger AEC gain or DRC gain, a higher reference threshold can be used compared to when the AEC gain or DRC gain is lower. The AEC gain and DRC gain can depend on the characteristics of the image sensor used by the device 100. Thus, for different image sensors under the same lighting conditions, the reference threshold can vary under the same lighting conditions.

[0042] At block 312, determining whether the pixel is colored in the current U data and the current V data may include determining that each of the current U data and the current V data has at least a minimum difference and does not exceed a maximum difference with the midpoint U value and the midpoint V value, respectively, i.e., determining that the absolute difference between the current U data and the midpoint U value and the absolute difference between the current V data and the midpoint V value are both between the current minimum value and less than the current maximum value. Therefore, when |CUR U - MID U|∈[CUR MIN ,CUR MAX ] and |CUR V-MID V|∈[CUR MIN ,CUR MAX ], it can be determined that the pixels in the current U data and the current V data are colored, where CUR U and CUR V are the current U data and the current V data, respectively, and CUR MIN is the current minimum value, CUR MAX is the current maximum value. The current minimum value may be greater than or equal to the reference threshold value. In some embodiments, the current minimum value may be the same as the reference threshold value, and may be 4, for example, when the U and V data range is between 0 and 256. In some embodiments, the current maximum value may be selected as a multiple of the current minimum value, such as three times the current minimum value. In one example, when the U and V data range is between 0 and 256, the current maximum value may be 12. If the pixel is not determined to be color-colored in the current U data and the current V data, the current U data and the current V data may be saved in the corrected U data and the corrected V data. If the pixel is determined to be color-colored, the corrected U data and the corrected V data of the pixel may include reference U data and reference V data instead of the current U data and the current V data. Each pixel in the current and reference U and V data is processed using the chrominance replacement step 310 to generate corrected U data and corrected V data for an image that meets the threshold darkness level.

[0043] Operation 300 may thus generate corrected U data and corrected V data, which may be output to combiner 218 to be combined into U data and V data in the output of process flow 200, for example.

[0044] Note that although operation 300 describes generating corrected U data and corrected V data on a per-pixel basis, in some other implementations, corrected U data and corrected V data may be determined on a per-window or per-region basis. For example, an image corresponding to reference and current U and V data may be divided into a plurality of windows or regions, and for each window or region, the corrected U and V data may include reference U and V data or current U and V data. In some embodiments, a representative value may be selected for each of the reference U and V data and the current U and V data. Such a representative value may be a predetermined pixel of the region or window, such as a center pixel, or a pixel in the upper left corner, etc., or may be an average value of the pixels in the window or region, such as a mean or median. In some examples, the representative value may be performed Figure 3 The chroma replacement step 310 is used to determine whether the corrected U and V data for the window or region should include the values ​​of the reference U and V data or the current U and V data.

[0045] In addition, note that while operation 300 describes replacing the anomalous color-colored pixels in the current U data and the current V data with corresponding pixels of the reference U data and the reference V data, in some other implementations, the anomalous color-colored pixels in the current U data and the current V data may be replaced with U data and V data of predetermined grayscale pixel values. For example, at block 312, in response to determining that the pixel is color-colored in the current U data and the current V data, the chromatic replacement step 310 may replace the current U data and the current V data with U and V data of predetermined grayscale pixel values. In some aspects, such predetermined grayscale pixel values ​​may have U and V data corresponding to a midpoint U value and a midpoint V value.

[0046] Figure 4 is an illustrative flow chart depicting example operations 400 for color correction in an image processing pipeline according to some embodiments of the present disclosure. Operation 400 may be performed by any suitable device including or coupled to an image processing pipeline. In some embodiments, operation 400 may be performed by Figure 1The device 100 shown in FIG. 4 is executed. In block 402, the device 100 receives first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels. In block 404, the device 100 determines that the first image data corresponds to an original image captured in a dark environment. For example, the determination may be based at least in part on an automatic exposure control (AEC) gain associated with the original image being greater than a threshold AEC gain. In block 406, the device 100 generates second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of a plurality of pixels. In block 408, the device 100 generates output image data for each of a plurality of pixels. For each of a plurality of pixels, generating output data may include generating output data to include an output luminance value (408A) of a corresponding pixel of the current luminance data, and including a chrominance value of the corresponding pixel selected from the reference chrominance data and the current chrominance data, wherein the selection is based at least in part on the reference chrominance data and the current chrominance data (408B). In some aspects, when the corresponding pixel of the first image data is determined to be gray and the corresponding pixel of the second image data is determined to be colored, the reference chromaticity data can be selected. In some aspects, when the corresponding pixel of the first image data is not determined to be gray or the corresponding pixel of the second image data is not determined to be colored, the current chromaticity data is selected.

[0047] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules or components may also be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium (e.g., a processor-readable medium) including instructions 108. Figure 1 The memory 106 in the example device 100 of the present invention is implemented, when the instruction 108 is executed by the processor 104 (or the controller 110 or the ISP 112), the device 100 performs one or more of the above methods. The non-transitory processor-readable storage medium may form part of a computer program product, which may include packaging materials.

[0048] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. These techniques may additionally or alternatively be implemented at least in part through a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0049] The various illustrative logical blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors, e.g. Figure 1 The processor 104 or ISP 112 in the example device 100 of . Such processors may include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. As used herein, the term "processor" may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functions described herein may be provided within a dedicated software module or hardware module configured as described herein. In addition, these techniques may be fully implemented in one or more circuits or logic elements. A general-purpose processor may be a microprocessor and / or any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0050] Although the present disclosure shows illustrative aspects, it should be noted that various changes and modifications may be made herein without departing from the scope of the appended claims. For example, one or more processing steps in the process flow 200 may be omitted, or one or more additional processing steps may be added. In addition, unless otherwise explicitly stated, the functions, steps, or actions of the method claims according to the aspects described herein need not be performed in any particular order. For example, the steps of the described example operations, if performed by the device 100, the controller 110, the processor 104, and / or the ISP 112, may be performed in any order and at any frequency. In addition, although an element may be described or claimed in the singular, the plural form is considered unless a limitation to the singular is explicitly stated. Therefore, the present disclosure is not limited to the examples shown, and any unit for performing the functions described herein is included in the aspects of the present disclosure.

Claims

1. A method for color correction in an image processing pipeline, comprising: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data satisfies a threshold darkness level; generating second image data by performing one or more tone mapping operations on the first image data, the second image data corresponding to current luminance data and current chrominance data for each pixel of the plurality of pixels; as well as Generating output image data to each pixel in the plurality of pixels comprises: The output brightness value of the pixel corresponding to the current brightness data; and A chromaticity value of the corresponding pixel from a selected one of the reference chromaticity data and the current chromaticity data, the selection being based at least in part on the reference chromaticity data and the current chromaticity data, wherein the reference chromaticity data is selected when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as being colored.

2. The method of claim 1, wherein: The first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data.

3. The method of claim 2, wherein: Detecting that a pixel in the first image data is gray includes: It is determined that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold.

4. The method of claim 3, wherein: Detecting that a pixel in the first image data is gray also includes determining that an absolute difference between the reference U data and the reference V data is less than the reference threshold.

5. The method of claim 3, wherein: Detecting that a pixel in the first image data is gray further includes determining that an absolute difference between the reference V data and a midpoint V value is less than the reference threshold.

6. The method of claim 5, wherein: The midpoint U value and the midpoint V value are both 128, and wherein the corresponding ranges of the U value and the V value in the YUV format are between 0 and 256.

7. The method of claim 2, wherein: Detecting that a pixel in the second image data is colored includes: Determine that the absolute difference between the current U data and the midpoint U value is greater than or equal to the current minimum value and less than or equal to the current maximum value; and Determine that the absolute difference between the current V data and the midpoint V value is greater than or equal to the current minimum value and less than or equal to the current maximum value; Wherein, the current minimum value is the minimum absolute difference between the current U data and the midpoint U value or between the current V data and the midpoint V value, and wherein, the current maximum value is the maximum absolute difference between the current U data and the midpoint U value or between the current V data and the midpoint V value.

8. The method of claim 1, wherein: Determining that the first image data satisfies the threshold darkness level includes determining that an automatic exposure control (AEC) gain associated with a raw image captured in a dark environment is greater than a threshold AEC gain.

9. The method of claim 1, wherein: Generating the second image data further includes performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data.

10. A device comprising a memory and one or more processors, wherein: The one or more processors are configured to: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data satisfies a threshold darkness level; generating second image data by performing one or more tone mapping operations on the first image data, the second image data corresponding to current luminance data and current chrominance data for each pixel of the plurality of pixels; as well as Generating output image data to each pixel in the plurality of pixels comprises: The output brightness value of the pixel corresponding to the current brightness data; and A chromaticity value of the corresponding pixel from a selected one of the reference chromaticity data and the current chromaticity data, the selection being based at least in part on the reference chromaticity data and the current chromaticity data, wherein the reference chromaticity data is selected when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as being colored.

11. The device of claim 10, wherein: The first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data.

12. The device of claim 11, wherein: The one or more processors are configured to detect that a pixel in the first image data is gray by: It is determined that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold.

13. The device of claim 12, wherein: The one or more processors are further configured to detect that a pixel in the first image data is gray by determining that an absolute difference between the reference U data and the reference V data is less than the reference threshold.

14. The device of claim 12, wherein: The one or more processors are further configured to detect that a pixel in the first image is gray by determining that an absolute difference between the reference V data and a midpoint V value is less than the reference threshold.

15. The device of claim 14, wherein: The midpoint U value and the midpoint V value are both 128, and wherein the corresponding ranges of the U value and the V value in the YUV format are between 0 and 256.

16. The device of claim 11, wherein: The one or more processors are configured to detect that a pixel in the second image data is colored by: Determine that the absolute difference between the current U data and the midpoint U value is greater than or equal to the current minimum value and less than or equal to the current maximum value; as well as Determine that the absolute difference between the current V data and the midpoint V value is greater than or equal to the current minimum value and less than or equal to the current maximum value; Wherein, the current minimum value is the minimum absolute difference between the current U data and the midpoint U value or between the current V data and the midpoint V value, and wherein, the current maximum value is the maximum absolute difference between the current U data and the midpoint U value or between the current V data and the midpoint V value.

17. The device of claim 10, wherein: The one or more processors are configured to determine that the first image data satisfies the threshold darkness level by determining that an automatic exposure control (AEC) gain associated with a raw image captured in a dark environment is greater than a threshold AEC gain.

18. The device of claim 10, wherein: The one or more processors are configured to generate the second image data based on performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data.

19. A non-transitory computer-readable storage medium storing instructions, which, when executed by one or more processors of an image processing device, cause the image processing device to perform operations comprising: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data satisfies a threshold darkness level; generating second image data by performing one or more tone mapping operations on the first image data, the second image data corresponding to current luminance data and current chrominance data for each pixel of the plurality of pixels; as well as Generating output image data to each pixel in the plurality of pixels comprises: The output brightness value of the pixel corresponding to the current brightness data; and A chromaticity value of the corresponding pixel from a selected one of the reference chromaticity data and the current chromaticity data, the selection being based at least in part on the reference chromaticity data and the current chromaticity data, wherein the reference chromaticity data is selected when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as being colored.

20. The non-transitory computer-readable storage medium of claim 19, wherein: The first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data.

21. The non-transitory computer-readable storage medium of claim 20, wherein: Execution of the instructions for detecting that pixels in the first image data are gray further causes the image processing device to perform operations including: Determining that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold; and It is determined that the absolute difference between the reference V data and the midpoint V value is less than the reference threshold.

22. The non-transitory computer-readable storage medium of claim 21, wherein: Execution of the instructions for detecting that a pixel in the first image data is gray further causes the image processing device to determine that an absolute difference between the reference U data and the reference V data is less than the reference threshold.

23. The non-transitory computer-readable storage medium of claim 22, wherein: The midpoint U value and the midpoint V value are both 128, and wherein the corresponding ranges of the U value and the V value in the YUV format are between 0 and 256.

24. The non-transitory computer-readable storage medium of claim 20, wherein: Execution of the instructions for detecting that pixels in the second image data are colorized causes the image processing device to perform operations further comprising: Determine that the absolute difference between the current U data and the midpoint U value is greater than or equal to the current minimum value and less than or equal to the current maximum value; as well as Determine that the absolute difference between the current V data and the midpoint V value is greater than or equal to the current minimum value and less than or equal to the current maximum value; Wherein, the current minimum value is the minimum absolute difference between the current U data and the midpoint U value or between the current V data and the midpoint V value, and wherein, the current maximum value is the maximum absolute difference between the current U data and the midpoint U value or between the current V data and the midpoint V value.

25. The non-transitory computer-readable storage medium of claim 19, wherein: Execution of the instructions for determining that the first image data satisfies the threshold darkness level causes the image processing device to perform operations further comprising determining that an automatic exposure control (AEC) gain associated with a raw image captured in a dark environment is greater than a threshold AEC gain.

26. The non-transitory computer-readable storage medium of claim 19, wherein: Execution of the instructions for generating the second image data causes the image processing device to perform operations further comprising: performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data.

27. An image processing pipeline comprising: An original image processing unit, configured to receive an original image and perform one or more image processing operations on the original image; a color correction matrix (CCM) unit for performing one or more CCM operations on the processed raw image; a denoising unit configured to perform one or more denoising operations on an output of the CCM unit to generate first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; a tone mapping unit for generating second image data by performing one or more tone mapping operations on the first image data, the second image data corresponding to current luminance data and current chrominance data of each pixel of the plurality of pixels; as well as A color correction unit is configured to: receiving the first image data and the second image data; determining that the first image data satisfies a threshold darkness level; as well as Generating output image data to each pixel in the plurality of pixels comprises: The output brightness value of the pixel corresponding to the current brightness data; and A chromaticity value of the corresponding pixel from a selected one of the reference chromaticity data and the current chromaticity data, the selection being based at least in part on the reference chromaticity data and the current chromaticity data, wherein the reference chromaticity data is selected when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as being colored.

Images