Correction of color-hued pixels captured in low-light conditions

Abstract

Aspects of the present disclosure relate to color correction in image processing pipelines. An exemplary method may include: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data corresponds to a raw image captured in a dark environment; generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and generating output image data. For each of the plurality of pixels, the output data may include an output luminance value of the pixel corresponding to the current luminance data, and chrominance values ​​of the corresponding pixel from selected data among the reference chrominance data and the current chrominance data, the selection being at least partially based on the reference chrominance data and the current chrominance data.

Description

Technology Field

[0001] The present disclosure generally relates to systems and methods for image capture, and specifically to pipeline processing of captured images. Background Technology

[0002] Abnormal black levels in image sensors, such as complementary metal-oxide semiconductor (CMOS) image sensors, can result in visible reddish tints in the black areas of an image. Color mismatch in image sensors can also be amplified by the image signal processor (ISP). For example, digital gain or dynamic gain compression (DRC) performed by the ISP can amplify color mismatch, leading to such reddish tints. means of solving the problem

[0003] This overview is provided to introduce, in a simplified form, the selection of concepts further described below in the detailed description. This overview is not intended to identify the core or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Each of the systems, methods, and devices of this disclosure has several innovative aspects, and no single aspect among these aspects is solely responsible for the preferred attributes disclosed herein.

[0004] One innovative aspect of the subject matter described in the present disclosure may be implemented as a method for color correction. The method may be performed by a suitable device and may include the steps of: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data corresponds to a raw image captured in a dark environment; generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and generating output image data for each of the plurality of pixels such that the output luminance value of the pixel corresponding to the current luminance data and the chrominance values ​​of the corresponding pixel are selected from the reference chrominance data and the current chrominance data, wherein the selection is at least partially based on the reference chrominance data and the current chrominance data.

[0005] In some embodiments, the first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data. In some embodiments, for each pixel of a plurality of pixels, if the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as colored, the output image data includes the reference chrominance data instead of the current chrominance data. In some embodiments, detecting that a pixel in the first image data is gray includes determining that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold, and determining that the absolute difference between the reference V data and the midpoint V value is less than a reference threshold. In some embodiments, detecting that a pixel in the second image data is colored includes determining that individual absolute differences between the current U data and the midpoint U value, and between the current V data and the midpoint V value, are each greater than the current minimum value and smaller than the current maximum value.

[0006] In some embodiments, generating the second image data further comprises performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. In some embodiments, the first image data is determined to correspond to a raw image captured in a dark environment by determining that the automatic exposure control (AEC) gain associated with the raw image is greater than a threshold AEC gain.

[0007] Other innovative aspects of the subject matter described in the present disclosure may be implemented in an apparatus comprising a memory and one or more processors. The one or more processors receive first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels, determine that the first image data corresponds to a raw image captured in a dark environment, and generate second image data by performing one or more tone mapping operations on the first image data, wherein the second image data is generated such that the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels, and for each of the plurality of pixels, generate output image data such that the output luminance value of the pixel corresponding to the current luminance data and the chrominance values ​​of the pixel corresponding to the selected data among the reference chrominance data and the current chrominance data, wherein the selection is at least partially based on the reference chrominance data and the current chrominance data.

[0008] In some embodiments, the first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data. In some embodiments, for each pixel of a plurality of pixels, if the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is colored, the output image data includes the reference chrominance data instead of the current chrominance data. In some embodiments, detecting that a pixel in the first image data is gray includes determining that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold, and determining that the absolute difference between the reference V data and the midpoint V value is less than a reference threshold. In some embodiments, detecting that a pixel in the second image data is colored includes determining that the individual absolute differences between the current U data and the midpoint U value, and between the current V data and the midpoint V value, are each greater than the current minimum value and less than the current maximum value.

[0009] In some embodiments, generating the second image data further comprises performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. In some embodiments, one or more processors are configured to determine that the first image data corresponds to a raw image captured in a dark environment by determining that the automatic exposure control (AEC) gain associated with the raw image is greater than a threshold AEC gain.

[0010] Other innovative aspects of the subject matter described in this disclosure may be implemented in a non-transient computer-readable storage medium that stores instructions for execution by one or more processors of an image processing device. The execution of the commands causes an image processing device to perform operations, the operations comprising receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels, determining that the first image data corresponds to a raw image captured in a dark environment, and generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of a plurality of pixels, and generating output image data for each of a plurality of pixels such that the output luminance value of the pixel corresponding to the current luminance data and the chrominance values ​​of the pixel corresponding to the selected data among the reference chrominance data and the current chrominance data are generated, wherein the selection is based at least partially on the reference chrominance data and the current chrominance data.

[0011] In some embodiments, the first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data. In some embodiments, for each pixel of a plurality of pixels, if the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as colored, the output image data includes the reference chrominance data instead of the current chrominance data. In some embodiments, detecting that a pixel in the first image data is gray includes determining that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold, and determining that the absolute difference between the reference V data and the midpoint V value is less than a reference threshold. In some embodiments, detecting that a pixel in the second image data is colored includes determining that individual absolute differences between the current U data and the midpoint U value, and between the current V data and the midpoint V value, are each greater than the current minimum value and smaller than the current maximum value.

[0012] In some embodiments, generating the second image data further comprises performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. In some embodiments, the first image data is determined to correspond to a raw image captured in a dark environment by determining that the automatic exposure control (AEC) gain associated with the raw image is greater than a threshold AEC gain.

[0013] Other innovative aspects of the subject matter described in this disclosure may be implemented in an image processing pipeline. An image processing pipeline comprises: a raw image processing means for receiving a raw image and performing one or more image processing operations on the raw image; a CCM means for performing one or more color correction matrix (CCM) operations on the processed raw image; a noise removal means for performing one or more noise removal operations on the output of the CCM means to generate first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; a tone mapping means for generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of a plurality of pixels; and the tone mapping means for receiving the first image data and the second image data, determining that the first image data corresponds to a raw image captured in a dark environment, and for each pixel of a plurality of pixels, the output luminance value of the pixel corresponding to the current luminance data, and the chrominance of the corresponding pixel from selected data among the reference chrominance data and the current chrominance data A color correction means configured to generate output image data including values, wherein the selection is based at least partially on reference chrominance data and current chrominance data. Brief explanation of the drawing

[0014] The embodiments of the present disclosure are illustrated in the accompanying drawings as examples rather than limitations, and in the drawings, the same reference numerals refer to similar elements. FIG. 1 is a block diagram of an exemplary device configured to perform color correction according to some implementations. FIG. 2 illustrates an exemplary processing flow for detecting and correcting abnormally colored pixels in a captured image according to some implementations. FIG. 3 illustrates an exemplary flowchart showing an exemplary operation for processing image data in a color correction stage according to some implementations. FIG. 4 is an exemplary flowchart illustrating exemplary operations for color correction in an image processing pipeline according to some implementations. Specific details for implementing the invention

[0015] Aspects of the present disclosure may be used to detect and correct abnormally colored pixels in images captured under low-light conditions. Abnormally colored pixels in an image may result from outputs from an image sensor indicating little to no light (which may be intended to be represented by deep black). The color tone may be amplified by digital gain or DRC performed by an ISP in processing the outputs from the image sensor. Such abnormal color tones may result in black pixels in an image containing visible red tones. Conventional techniques for correcting such colored pixels may have a high false positive rate in detecting colored pixels, which may result in color degradation in one or more regions of the image that do not contain colored pixels. Additionally, or alternatively, conventional techniques for correcting such colored pixels may have a high false negative rate in detecting colored pixels, which may result in a final image containing pixels with uncorrected red tones. Conventional techniques typically process raw image data. Since using black level correction may not effectively suppress red tones in black pixels, and the raw image data for red and green colors may be similar, the final image may contain one or more black regions with alternating red and green toned pixels.

[0016] The methods and apparatus of exemplary embodiments of the present disclosure may detect and correct pixels exhibiting such undesirable color tones of pixels in an image captured in low-light environments. For example, the apparatus may detect one or more pixels that are converted from gray pixels to colored pixels at a stage in an image processing pipeline. When such pixels are detected, the color tone may be removed by replacing each of such pixels with a corresponding pixel from a previous stage in the image processing pipeline. For example, the color tone may be caused or amplified by one or more tone mapping operations in the image processing pipeline. Replacing a colored pixel with a corresponding gray pixel from a previous stage may include replacing the tone-mapped colored pixel with a corresponding pixel from a stage in the image processing pipeline prior to one or more tone mapping operations.

[0017] While conventional color tone correction techniques may operate on raw image data, some exemplary embodiments of the present disclosure may operate on image data in the YUV color space, where Y refers to luminance, U refers to blue projection chrominance, and V refers to red projection chrominance. Other embodiments may operate in other similar luminance / chroma color spaces. Using such color spaces allows the exemplary embodiments to focus color tone correction directly on chrominance without affecting the luminance of the output pixels.

[0018] In the following description, numerous specific details, such as examples of specific components, circuits, and processors, are described to provide a thorough understanding of the present disclosure. The term “coupled” as used herein means directly connected or connected through one or more interposed components or circuits. Additionally, in the following description and for the purposes of explanation, specific nomenclature is described to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to carry out the teachings disclosed herein. In some instances, widely known circuits and devices are illustrated in block diagram form to avoid obscuring the teachings of the present disclosure. Some parts of the following detailed descriptions are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within computer memory. In the present disclosure, procedures, logic blocks, processes, etc., are recognized as self-consistent sequences of steps or instructions leading to a desired result. Steps are those that require physical manipulation of physical quantities. Although not essential, these quantities typically take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, or otherwise manipulated in a computer system.

[0019] However, it should be kept in mind that all these terms and similar terms are to be associated with appropriate physical quantities and are merely convenient labels applied to these quantities. As is evident from the following discussions, unless specifically stated otherwise, it is recognized that throughout this Appendix, discussions utilizing terms such as “accessing,” “receiving,” “transmitting,” “using,” “selecting,” “determining,” “normalizing,” “multiplying,” “averaging,” “monitoring,” “comparing,” “applying,” “updating,” “measuring,” “deriving,” “solving,” etc. refer to actions and processes of a computer system or similar electronic computing device, which manipulates and converts data represented as physical (electronic) quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the memories or registers of the computer system or such other information storage, transmission, or display devices.

[0020] In the drawings, a single block may be described as performing a function or functions; however, in actual practice, the function or functions performed by the block may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or a combination of hardware and software. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps are generally described below in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the design constraints imposed on the overall system and the specific application. Those skilled in the art may implement the described functionality in various ways for each specific application, but such decisions regarding implementation should not be interpreted as causing a deviation from the scope of this disclosure. Additionally, exemplary devices may include components other than those illustrated, including widely known components such as processors, memory, etc.

[0021] The embodiments of the present disclosure are applicable to any suitable electronic device configured to capture or capable of capturing images or video (such as security systems having one or more cameras, smartphones, tablets, laptop computers, digital video and / or still cameras, web cameras, etc.). Although described below with respect to a device having or coupled with a single camera, the embodiments of the present disclosure are applicable to devices having any number of cameras and are therefore not limited to devices having a single camera. The embodiments of the present disclosure are applicable to cameras configured to capture video as well as still images and may be implemented in devices having or coupled with cameras of different capabilities.

[0022] The term “device” is not limited to one or a specific number of physical objects (such as a smartphone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more parts that may implement at least some parts of the present disclosure. Although the following description and examples use the term “device” to describe various aspects of the present disclosure, the term “device” is not limited to a specific configuration, type, or number of objects.

[0023] FIG. 1 is a block diagram of an exemplary device (100) configured to perform color correction according to some embodiments of the present disclosure. The exemplary device (100) may include or be coupled thereto a camera (102), a processor (104), a memory (106) for storing commands (108), and a controller (110). In some embodiments, the device (100) may include a display (114) and a plurality of input / output (I / O) components (116) (or may be coupled thereto). The device (100) may also include a power supply (118) which may be coupled to or integrated with the device (100). The device (100) may include additional features or components not illustrated. For example, a wireless interface that may include a plurality of transceivers and a baseband processor may be included for the wireless communication device. In another example, the device (100) may include additional cameras other than the camera (102) or be coupled thereto.

[0024] The camera (102) may be any suitable camera capable of capturing still images (such as individual captured image frames) and / or capturing video (such as a series of captured image frames). The camera (102) may include a single image sensor, a dual camera module, or any other suitable module having one or more image sensors, such as one or more CMOS image sensors.

[0025] Memory (106) may be a non-transient or non-transient computer-readable medium storing computer-executable instructions (108) for performing all or part of one or more operations described in this disclosure. Processor (104) may be one or more suitable processors capable of executing scripts or instructions of one or more software programs (such as instructions (108)) stored in memory (106). In some embodiments, processor (104) may be one or more general-purpose processors that execute instructions (108) to enable the device (100) to perform any number of functions or operations. In additional or alternative embodiments, processor (104) may include integrated circuits or other hardware for performing functions or operations without the use of software.

[0026] In the example of FIG. 1, the processor (104) is shown as being coupled to each other, but the processor (104), memory (106), controller (110), optional display (114), and optional I / O components (116) may be coupled to each other in various arrangements. For example, the processor (104), memory (106), controller (110), optional display (114), and / or optional I / O components (116) may be coupled to each other through one or more local buses (not shown for simplification).

[0027] The display (114) may be any suitable display or screen for allowing user interaction and / or presenting items (such as captured images, videos, or preview images) for viewing by the user. In some embodiments, the display (114) may be a touch-sensitive display. The I / O components (116) may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from the user and providing output to the user. For example, the I / O components (116) may include (but are not limited to) a graphical user interface, keyboard, mouse, microphone, and speakers, etc. The display (114) and / or the I / O components (116) may receive user input to provide a preview image to the user and / or adjust one or more settings of the camera (102).

[0028] The controller (110) may include one or more controllers. One or more controllers may be configured to control the camera (102). The controller (110) may include an image signal processor (ISP) (112) which may be one or more image signal processors for processing captured image frames or video provided by the camera (102). In some embodiments, the controller (110) or the ISP (112) may execute instructions from memory (such as instructions (108) from memory (106) or instructions stored in a separate memory coupled to the ISP (112). In other embodiments, the controller (110) or the ISP (112) may include specific hardware. The controller (110) or the ISP (112) may, alternatively or additionally, include a combination of specific hardware and the ability to execute software instructions. For example, the ISP (112) may be configured to include or otherwise implement one or more stages of an image processing pipeline.

[0029] FIG. 2 illustrates an exemplary processing flow (200) for detecting and correcting inaccurately colored pixels in a captured image according to some embodiments of the present disclosure. In some embodiments, the processing flow (200) may be included in an image processing pipeline implemented or controlled by the device (100) illustrated in FIG. 1. Although the processing flow (200) may be described as being performed by the device (100), the processing flow (200) may also be performed by any other suitable image capture device.

[0030] With respect to FIG. 2, raw image data may be input to the processing flow (200). Such raw image data may be received, for example, from one or more image sensors included in or coupled to the device (100), or from the analog front end of the device (100). In the raw image processing stage (202), the device (100) may perform one or more operations on the raw image data (such as one or more operations for black level correction, lens shading correction (LSC), autofocus, auto exposure, auto white balance, etc.). After processing the image data in the raw image processing stage (202), the device (100) may process the image data in the color correction matrix (CCM) stage (204). The processed raw image data may be converted to a red-green-blue (RGB) color space before proceeding to the CCM stage (204). Alternatively, the device (100) may convert the data to the RGB color space in the CCM stage (204). In the CCM stage (204), the device (100) may perform one or more operations on the image data in the RGB color space (such as one or more operations for color correction in the RGB color space). In some examples, the device (100) may convert the RGB pixel values ​​of the image data, represented by a color correction matrix, into a defined and known color space. After the device (100) processes the image data in the CCM stage (204), the device (100) processes the image data in a noise removal stage (206) that performs one or more noise removal operations on the image data.In some implementations, the noise removal stage (206) may process image data in the YUV color space, and, for example, the device (100) may convert image data from the RGB color space to the YUV color space before or at the noise removal stage (206). The output image data from the noise removal stage (206) may be referred to as reference image data. For example, the output YUV image data representing image data in the YUV color space may include reference Y data representing the luminance of the image data, and reference U data and reference V data representing the chrominance of the image data.

[0031] After the noise removal stage (206), the processing flow (200) is divided into multiple paths (as shown in FIG. 2, the upper flow (220) and lower flow (230)). First, regarding the upper flow (220), the device (100) processes the image data in the local tone mapping (LTM) stage (208). In the LTM stage (208), the device (100) may perform one or more tone mapping operations on the image data. In some embodiments, the LTM stage (208) may operate on the image data in the RGB color space. Thus, prior to or as part of the LTM stage (208), the device (100) may convert the image data into the RGB color space. Tone mapping may adjust the brightness of different regions of the image data or combine different exposures to increase local contrast between different regions of the scene. Local tone mapping, in contrast to globally or spatially uniform tone mapping, may adjust the brightness of a given pixel based locally, such as near features of the image. After the LTM stage (208), the device (100) may process the image data in a gamma compression stage (210). In the gamma compression stage (210), the device (100) may perform one or more operations to increase the exposure of underexposed parts of the image data while decreasing the exposure of overexposed parts of the image data. In some embodiments, the device (100) may apply a gamma function. For example, the device (100) A gamma filter such as can also be applied, where, and A > 0. When applying the gamma function, the device (100) is Input luminance (Y) in the domain in) output luminance (Y) in the range [0,1] out Maps to ). In an exemplary gamma filter, It may be adjusted to control the contrast of the image, and lower values ​​correspond to increased exposure and lower contrast for underexposed parts of the image. A may be set to less than 1 to ensure that the exposure of one or more parts of the image is sufficiently reduced to prevent overexposure.

[0032] After the gamma compression stage (210), the device (100) may process image data in a sharpness stage (212), which may perform one or more sharpness enhancement operations on the image data, such as edge enhancement, deblurring filtering, etc. In some embodiments, the device (100) may process image data in the YUV color space in the sharpness stage (212). Accordingly, prior to or as part of the sharpness stage (212), the device (100) may convert the image data into the YUV color space. The output image data of the sharpness stage (212) may be referred to as the current image data. For example, the output YUV image data representing the image data in the YUV color space may include the current Y data representing the luminance of the image data, the current U data representing the chrominance of the image data, and the current V data. The current Y data may be provided to the combiner (218). Current U data and current V data may also be provided to a color correction stage (216).

[0033] In the color correction stage (216), the device (100) may optionally output reference U data and reference V data or current U data and current V data for each pixel of the image data. As mentioned above, for images captured in low-light or dark environments, some gray pixels may be converted into colored pixels during image processing, which may result in a reddish tint visible in the black areas of the image. This may often be caused by tone mapping operations. As discussed in more detail below, the color correction stage (216) may detect when the captured image appears to have been captured in a sufficiently dark environment. For such images, the device (100) may detect when gray pixels are converted into colored pixels (e.g., by comparing reference U and V data with the corresponding current U and V data). For such pixels, the device (100) may replace the current U and V data with the corresponding reference U and V data. In this way, image processing steps that result in abnormal color tones may be canceled. The color correction stage (216) may output the corrected U data and the corrected V data to the combiner (218). The combiner (218) combines the current luminance data with the corrected U data and the corrected V data to generate individual Y, U, and V data for the output image data. Thus, the output image data reflects the current luminance (Y) data and the corrected chrominance (U and V) data.

[0034] FIG. 3 illustrates an exemplary flowchart illustrating an exemplary operation (300) for processing image data in a color correction stage according to some embodiments of the present disclosure. In some embodiments, the operation (300) may be performed by the device (100) in the color correction stage (216) of FIG. 2. Reference U and V data from the noise removal stage (206) and current U and V data from the sharpness stage (212) may be used to generate corrected U and V data for the image. As discussed above, the reference U and V data may correspond to image data before tone mapping is performed. The current U and V data may correspond to image data after tone mapping is performed.

[0035] Abnormal color tones typically occur in images captured in dark environments. Accordingly, the device (100) determines whether the image data meets a threshold darkness level (302). An exemplary threshold darkness level may include an auto-exposure correction gain threshold. For example, the device (100) may determine whether the total auto-exposure correction gain for the captured image exceeds the auto-exposure correction gain threshold. As discussed above, such auto-exposure correction may occur as part of the raw image processing stage (202). Another exemplary threshold darkness level may include a camera flash enablement threshold. For example, the device (100) may determine whether the environment is sufficiently dark for the camera flash to be enabled. Another exemplary threshold darkness level may be a light sensor threshold. For example, the device (100) may determine whether a light sensor included in or coupled to the device (100) detects less than a threshold amount of light. If the threshold darkness level is not satisfied, the operation (300) may output the current U data and current V data as corrected U data and corrected V data, and the process may terminate.

[0036] When the threshold darkness level is satisfied, the image is a candidate image for replacing the current U and V data of one or more pixels with reference U and V data. The operation (300) may proceed to a chrominance replacement step (310), whereby the device (100) may perform a series of decisions for each pixel of the current and reference U and V data. For each pixel, the device (100) may, in block (311), determine whether the pixel is gray in the reference U and V data. In some embodiments, this determination may include determining that the absolute difference between the reference U data and the midpoint ("midpoint U value") in the range of possible values ​​of U is less than the reference threshold, and that the absolute difference between the reference V data and the midpoint ("midpoint V value") in the range of possible values ​​of V is also less than the reference threshold. Therefore, a pixel is determined to be gray when its U and V values ​​are sufficiently close to the midpoint U value and midpoint V value, respectively. Thus, a pixel is gray when (|REF U - MID U| < REF_TH) and (|REF V - MID V| < REF_TH), where REF U is reference U data and REF V is reference V data, MID U is midpoint U value and MID V is midpoint V value. For example, if the possible range of U and V values ​​is between 0 and 256, the midpoint U and V values ​​are 128, respectively. Determining whether a pixel is gray may include determining whether (|REF U - 128| < REF_TH) and (|REF V - 128| < REF_TH). In some implementations, the reference threshold may be a single-digit reference threshold for U and V having ranges between 0 and 256. In one example, the reference threshold may be 4.In some implementations, determining that a pixel in reference data is gray may also include determining that the absolute difference between reference U data and reference V data is less than a reference threshold, i.e., determining that |REF U - REF V| < REF_TH. If each of these absolute differences is less than the reference threshold, the pixel is considered gray. If the pixel is not considered gray in reference U and V data, the current U and current V data are maintained at the corrected U and V data for that pixel. If the pixel is gray in reference U and V data, the device (100) may determine, in block (312), whether the pixel is colored in the current U and V data.

[0037] Note that in some implementations, the reference threshold may vary based at least partially on the Automatic Exposure Control (AEC) gain and / or Dynamic Range Compression (DRC) gain. For example, for larger AEC gains or DRC gains, a higher reference threshold may be used compared to when the AEC gain or DRC gain is lower. The AEC gain and DRC gain may also depend on the characteristics of the image sensor used by the device (100). Consequently, the reference threshold may vary for different image sensors under the same lighting conditions.

[0038] In block (312), determining whether a pixel is color-hugged in the current U data and current V data may include determining that each of the current U data and current V data has at least a minimum difference and a maximum difference from the midpoint U value and midpoint V value, respectively, that is, determining that the absolute difference between the current U data and the midpoint U value and the absolute difference between the current V data and the midpoint V value are each less than the current minimum value and the current maximum value. Thus, the pixel in the current U data and current V data is and In the case where it may be determined to be a color tone, where CUR U and CUR V are the current U data and current V data, respectively, and CUR MIN is the current minimum value, and CUR MAXis the current maximum value. The current minimum value may be greater than or equal to the reference threshold. In some implementations, the current minimum value may be equal to the reference threshold, for example, 4 when the U and V data are in the range between 0 and 256. In some implementations, the current maximum value may be selected to be a multiple of the current minimum value, such as 3 times the current minimum value. In one example, the current maximum value may be 12 when the U and V data are in the range between 0 and 256. If it is not determined that the pixel is colored in the current U data and current V data, the current U data and current V data may be retained in the corrected U data and corrected V data. If it is determined that the pixel is colored, the corrected U data and corrected V data for the pixel may include the reference U data and reference V data rather than the current U data and current V data. Therefore, processing each pixel in the current and reference U and V data using the chrominance replacement step (310) generates corrected U data and corrected V data for images that satisfy a threshold darkness level.

[0039] Accordingly, the operation (300) may generate corrected U data and corrected V data, which may be output to a combiner (218) for integration as U data and V data at the output of the processing flow (200), for example.

[0040] It should be noted that while the operation (300) describes generating corrected U data and corrected V data on a pixel-by-pixel basis, in some other implementations, the corrected U data and corrected V data may be determined on a window-by-window or area-by-area basis. For example, an image corresponding to reference and current U and V data may be divided into multiple windows or areas, and for each window or area, the corrected U and V data may include reference U and V data or current U and V data. In some implementations, a representative value may be selected for each of the reference U and V data and current U and V data. Such a representative value may be a predetermined pixel of the area or window, such as a center pixel or the top left pixel, or an average value such as the average or median value of the pixels in the window or area. In some examples, the chrominance replacement step (310) of FIG. 3 may be performed on the representative value to determine whether the corrected U and V data for the window or area should include values ​​of the reference U and V data or current U and V data.

[0041] Furthermore, it should be noted that while the operation (300) describes replacing abnormally colored pixels in the current U data and current V data with corresponding pixels in the reference U data and reference V data, in some other implementations, abnormally colored pixels in the current U data and current V data may be replaced with U and V data corresponding to predetermined gray pixel values. For example, in block (312), in response to determining that a pixel is colored in the current U data and current V data, the chrominance replacement step (310) may replace the current U data and current V data with U and V data corresponding to predetermined gray pixel values. In some embodiments, such predetermined gray pixel values ​​may have U and V data corresponding to midpoint U values ​​and midpoint V values.

[0042] FIG. 4 is an exemplary flowchart illustrating an exemplary operation (400) for color correction in an image processing pipeline according to some embodiments of the present disclosure. The operation (400) may be performed by any suitable device comprising or coupled to the image processing pipeline. In some embodiments, the operation (400) may be performed by the device (100) illustrated in FIG. 1. In block 402, the device (100) receives first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels. In block 404, the device (100) determines that the first image data corresponds to a raw image captured in a dark environment. For example, this determination may be based at least partially on the fact that the automatic exposure control (AEC) gain associated with the raw image is greater than a threshold AEC gain. In block 406, the device (100) generates second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels. In block 408, the device (100) generates output image data for each of the plurality of pixels. For each of the plurality of pixels, generating output data may include generating output data to include the output luminance value of the pixel corresponding to the current luminance data (408A), and generating output data to include the chrominance values ​​of the corresponding pixel from selected data among reference chrominance data and current chrominance data, wherein the selection is at least partially based on the reference chrominance data and current chrominance data (408B).In some embodiments, reference chrominance data may be selected when the corresponding pixel of the first image data is determined to be gray and the corresponding pixel of the second image data is determined to be colored. In some embodiments, current chrominance data is selected when the corresponding pixel of the first image data is not determined to be gray or the corresponding pixel of the second image data is not determined to be colored.

[0043] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules or components may also be implemented together in an integrated logic device or separately as distinct but interoperable logic devices. When implemented in software, the techniques may be realized at least partially by a non-transient processor-readable storage medium (such as memory (106) in the exemplary device (100) of FIG. 1) comprising instructions (108) that cause the device (100) to perform one or more of the methods described above when executed by a processor (104) (or controller (110) or ISP (112)). The non-transient processor-readable storage medium may form part of a computer program product, which may include packaging materials.

[0044] Non-transient processor-readable storage media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Additionally or alternatively, the techniques may be realized at least partially by a processor-readable communication medium that stores or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0045] The various exemplary logic blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors, such as the processor (104) or ISP (112) in the exemplary device (100) of FIG. 1. Such processor(s) may include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field programmable gate arrays (FPGAs), or other equivalent integrated or separate logic circuits. The term “processor” as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. Additionally, in some embodiments, the functionality described herein may be provided within dedicated software modules or hardware modules configured as described herein. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements. A general-purpose processor may be a microprocessor and / or any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other configuration.

[0046] While the present disclosure presents exemplary embodiments, it should be noted that various changes and modifications may be made in this specification without departing from the scope of the appended claims. For example, one or more processing steps in the processing flow (200) may be omitted, or one or more additional processing steps may be added. Additionally, the functions, steps, or actions of the method claims according to the embodiments described in this specification do not need to be performed in any specific order unless otherwise explicitly stated. For example, when performed by the device (100), controller (110), processor (104), and / or ISP (112), the steps of the described exemplary operations may be performed in any order and at any frequency. Furthermore, although the elements may be described or claimed in the singular, the plural is considered unless the limitation to the singular is explicitly stated. Accordingly, the present disclosure is not limited to the examples illustrated, and any means of performing the functionality described in this specification are included in the embodiments of the present disclosure.

Claims

Claim 1 A method for color correction in an image processing pipeline, comprising: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data corresponds to a raw image captured in a dark environment; generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and for each pixel of the plurality of pixels: an output luminance value of the pixel corresponding to the current luminance data; A method for color correction in an image processing pipeline, comprising the step of generating output image data to include chrominance values ​​of corresponding pixels from selected data among the reference chrominance data and the current chrominance data, wherein the selection is at least partially based on the reference chrominance data and the current chrominance data, and for each pixel of the plurality of pixels, if the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as colored, the output image data includes the reference chrominance data instead of the current chrominance data. Claim 2 delete Claim 3 A method for color correction in an image processing pipeline according to claim 1, wherein the first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data. Claim 4 A method for color correction in an image processing pipeline according to claim 3, wherein detecting that a pixel in the first image data is gray includes determining that the absolute difference between the reference U data and the midpoint U value is smaller than a reference threshold. Claim 5 A method for color correction in an image processing pipeline according to claim 4, wherein detecting that a pixel in the first image data is gray further comprises determining that the absolute difference between the reference U data and the reference V data is smaller than the reference threshold. Claim 6 A method for color correction in an image processing pipeline according to claim 4, wherein detecting that a pixel in the first image data is gray further comprises determining that the absolute difference between the reference V data and the midpoint V value is smaller than the reference threshold. Claim 7 A method for color correction in an image processing pipeline according to claim 6, wherein the midpoint U value and the midpoint V value are each 128, and the individual ranges of the U and V values ​​in the YUV format are between 0 and 256. Claim 8 A method for color correction in an image processing pipeline according to claim 3, wherein detecting that a pixel in the second image data is colored comprises determining that the absolute difference between the current U data and the midpoint U value is greater than or equal to the current minimum value and less than or equal to the current maximum value; and determining that the absolute difference between the current V data and the midpoint V value is greater than or equal to the current minimum value and less than or equal to the current maximum value. Claim 9 A method for color correction in an image processing pipeline according to claim 1, wherein the step of determining that the first image data corresponds to a raw image captured in a dark environment includes the step of determining that the automatic exposure control (AEC) gain associated with the raw image is greater than a threshold AEC gain. Claim 10 A method for color correction in an image processing pipeline, wherein the step of generating the second image data further comprises the step of performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. Claim 11 A device comprising a memory and one or more processors, wherein the one or more processors receive first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determine that the first image data corresponds to a raw image captured in a dark environment; generate second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and for each pixel of the plurality of pixels: an output luminance value of the pixel corresponding to the current luminance data; An apparatus configured to generate output image data including chrominance values ​​of corresponding pixels from selected data among the reference chrominance data and the current chrominance data, wherein the selection is at least partially based on the reference chrominance data and the current chrominance data, and for each pixel of the plurality of pixels, if the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as colored, the output image data includes the reference chrominance data instead of the current chrominance data. Claim 12 delete Claim 13 In claim 11, the device wherein the first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data. Claim 14 In claim 13, the device is configured such that one or more processors detect that a pixel in the first image data is gray by determining that the absolute difference between the reference U data and the midpoint U value is smaller than a reference threshold. Claim 15 In claim 14, the device is configured such that the one or more processors additionally detect that a pixel in the first image data is gray by determining that the absolute difference between the reference U data and the reference V data is smaller than the reference threshold. Claim 16 In claim 14, the device is configured such that the one or more processors additionally detect that a pixel in the first image data is gray by determining that the absolute difference between the reference V data and the midpoint V value is smaller than the reference threshold. Claim 17 A device according to claim 16, wherein the midpoint U value and the midpoint V value are each 128, and the individual ranges of the U and V values ​​in the YUV format are between 0 and 256. Claim 18 An apparatus according to claim 13, wherein the one or more processors are configured to detect that a pixel in the second image data is colored by determining that the absolute difference between the current U data and the midpoint U value is greater than or equal to the current minimum value and less than or equal to the current maximum value; and determining that the absolute difference between the current V data and the midpoint V value is greater than or equal to the current minimum value and less than or equal to the current maximum value. Claim 19 In claim 11, the apparatus is configured such that the one or more processors determine that the first image data corresponds to the raw image captured in the dark environment by determining that the automatic exposure control (AEC) gain associated with the raw image is greater than a threshold AEC gain. Claim 20 In claim 11, the apparatus is configured such that the one or more processors generate the second image data based on performing one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. Claim 21 A non-transient computer-readable storage medium for storing instructions, wherein the instructions, when executed by one or more processors of an image processing device, cause the image processing device to perform operations, the operations comprising: receiving first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; determining that the first image data corresponds to a raw image captured in a dark environment; generating second image data by performing one or more tone mapping operations on the first image data, wherein the second image data corresponds to current luminance data and current chrominance data for each of the plurality of pixels; and for each pixel of the plurality of pixels: an output luminance value of the pixel corresponding to the current luminance data; A non-transient computer-readable storage medium comprising generating output image data to include chrominance values ​​of corresponding pixels from selected data among the reference chrominance data and the current chrominance data, wherein the selection is at least partially based on the reference chrominance data and the current chrominance data, and for each pixel of the plurality of pixels, if the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as colored, the output image data comprises the reference chrominance data instead of the current chrominance data. Claim 22 delete Claim 23 A non-transient computer-readable storage medium according to claim 21, wherein the first image data and the second image data are in YUV format, the reference chrominance data includes reference U data and reference V data, and the current chrominance data includes current U data and current V data. Claim 24 A non-transient computer-readable storage medium, wherein the execution of the commands for detecting that a pixel in the first image data is gray further comprises: causing the image processing device to perform operations including determining that the absolute difference between the reference U data and the midpoint U value is less than a reference threshold; and determining that the absolute difference between the reference V data and the midpoint V value is less than a reference threshold. Claim 25 In claim 24, the execution of the commands for detecting that a pixel in the first image data is gray further enables the image processing device to determine that the absolute difference between the reference U data and the reference V data is smaller than the reference threshold, a non-transient computer-readable storage medium. Claim 26 A non-transient computer-readable storage medium according to claim 25, wherein the midpoint U value and the midpoint V value are each 128, and the individual ranges of the U and V values ​​in the YUV format are between 0 and 256. Claim 27 A non-transient computer-readable storage medium according to claim 23, wherein the execution of the commands for detecting that a pixel in the second image data is colored further comprises the image processing device determining that the absolute difference between the current U data and the midpoint U value is greater than or equal to the current minimum value and less than or equal to the current maximum value; and determining that the absolute difference between the current V data and the midpoint V value is greater than or equal to the current minimum value and less than or equal to the current maximum value. Claim 28 A non-transient computer-readable storage medium according to claim 21, wherein the execution of the commands for determining that the first image data corresponds to a raw image captured in a dark environment further comprises the image processing device performing operations including determining that an automatic exposure control (AEC) gain associated with the raw image is greater than a threshold AEC gain. Claim 29 A non-transient computer-readable storage medium according to claim 21, wherein the execution of the commands for generating the second image data further comprises the image processing device performing operations including one or more gamma correction operations and one or more sharpness enhancement operations on the first image data. Claim 30 As an image processing pipeline, raw image processing means for receiving a raw image and performing one or more image processing operations on said raw image; CCM means for performing one or more color correction matrix (CCM) operations on said processed raw image; noise removal means for performing one or more noise removal operations on the output of said CCM means to generate first image data corresponding to reference luminance data and reference chrominance data for each of a plurality of pixels; tone mapping means for generating second image data by performing one or more tone mapping operations on said first image data, wherein said second image data corresponds to current luminance data and current chrominance data for each of said plurality of pixels; and as a color correction means, receiving said first image data and said second image data; determining that said first image data corresponds to a raw image captured in a dark environment; and for each of said plurality of pixels: an output luminance value of the pixel corresponding to said current luminance data; An image processing pipeline comprising a color correction means configured to generate output image data including chrominance values ​​of corresponding pixels from selected data among the reference chrominance data and the current chrominance data, wherein the selection is at least partially based on the reference chrominance data and the current chrominance data, and for each pixel of the plurality of pixels, when the corresponding pixel of the first image data is detected as gray and the corresponding pixel of the second image data is detected as colored, the output image data includes the reference chrominance data instead of the current chrominance data.

Images