Device, method and system for highlight recovery of Bayer images

Abstract

The present invention discloses a device, method and system for highlight recovery of Bayer images, and relates to the field of image processing technology. The device includes a highlight control module connected to a white balance gain control module, and is configured to: for each frame of image, obtain the white balance gain of each color channel, compare the white balance gain of each color channel to determine the maximum gain value; and obtain a predetermined RAW domain data bit width range, and calculate the maximum value of the highlight area according to the highest bit width value and the maximum gain value; and obtain a preset highlight area minimum value and then determine a mapping curve in combination with the highlight area maximum value, and map the color channel data output by the white balance gain control module after multiplying the white balance gain according to the mapping curve to obtain the color channel data after highlight compression. The present invention can correct the problem of detail loss and color cast caused by overexposure of the highlight area after white balance processing, and has the characteristics of being simple and easy to implement, flexible to debug and having few side effects.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a device, method and system for restoring highlights of a Bayer image. Background Art

[0002] RAW images are the raw data (RAW means "unprocessed") generated by an image sensor (e.g., a CMOS or CCD image sensor) converting the captured light signal into a digital signal. These images are lossless and contain the original color information of the object. The color filter array (CFA) is a filter array layer overlying the image sensor, used to capture color information. Conventional photosensors can only sense light intensity and cannot distinguish between wavelengths. Therefore, image sensors use a color filter array to obtain color information at each pixel. The most common Bayer filter provides information about the intensity of light in the red, green, and blue wavelength regions. The image sensor uses the Bayer filter to obtain this information, then uses a color restoration algorithm to infer the pixel's color value. The resulting image is called an RGB Bayer image or Bayer image. For example, in a digital camera, a Bayer image is the original image stored internally in the camera and typically has the suffix .RAW. Typically, an imaging system has an image processing pipeline (ISP pipeline, or Image Signal Processing pipeline), which can perform various image processing operations on RAW format images to generate full-color images that can be displayed on a display device (such as a monitor) for viewing.

[0003] In the ISP pipeline algorithm, a series of problems with color restoration and tone processing can be solved through automatic white balance (AWB) processing. The purpose of white balance is to restore the object's inherent color from the color presented under different light sources, so as to achieve the effect of reducing or eliminating color cast. Since the ISP pipeline involves a large amount of data and strict real-time requirements, the ISP (image signal processor) is usually implemented using a hardware solution. When using dedicated hardware to process the ISP pipeline, in order to save hardware resources, the storage of each data usually has a bit width limit. For example, the RAW data bit width range will be configured according to the hardware resources in the ISP Pipeline algorithm.

[0004] The white balance algorithm is a global algorithm. It multiplies the three color channel data (including R channel, G channel and B channel) output by the sensor by a gain coefficient. These three gain coefficients are defined as white balance gain (AWB gain). After the white balance gain is applied to the image, the image obtains a normal white balance - that is, the image is compensated by the white balance gain to obtain its display effect under standard white light. However, for the brighter areas in the image, after white balance processing, the multiplication of the white balance gain value may cause the RAW data bit width range defined by the ISP Pipeline to be exceeded. For data that exceeds the defined bit width range, the usual processing method is clipping (or cropping), which truncates the data that exceeds the range to the defined value of the highest bit width. For example, if the maximum bit width of the configured image source data is 12 bits, the corresponding image brightness range is [0, 4095]. When the brightness value of a highlight area exceeds 4095 after white balance gain processing, the brightness value exceeding 4095 will be truncated to 4095 (the maximum brightness value that can be used). See Figure 1 As shown, the highlight area has a uniform brightness value, which will lead to loss of highlight area details (overexposure) and image color cast. For example, the truncation process may cause the sky in the image to appear purple and lose details such as the original clouds.

[0005] To address the loss of detail in highlight areas of an image, existing technologies have proposed highlight restoration techniques. This technique corrects highlight cropping by restoring color information in cropped areas of an image. The following three methods are commonly used in existing highlight restoration techniques.

[0006] The first method uses a highlight removal method based on a two-color reflection model. This model assumes that an image is composed of diffuse and specular reflection components. Under non-uniform lighting conditions, objects often have areas of intense reflection, known as highlights, on their surfaces. These areas can be considered specular reflection components. Therefore, methods such as spatial or color space analysis are used to remove specular reflection components, thereby obtaining an image without highlight regions.

[0007] The second method is tone mapping. Tone mapping compresses the brightness of an HDR (High-Dynamic Range) image and maps it to the display range of an LDR (Low-Dynamic Range) display device, while maintaining important information such as detail and color from the original HDR image. This algorithm compresses image brightness, including adjusting the entire brightness range of the image, while preserving image detail and color.

[0008] The third method is the color propagation highlight restoration method. To perform color propagation highlight restoration, the color values ​​of pixels that have not been truncated and are adjacent to the truncated area are propagated to the truncated area. Therefore, the color values ​​of the untruncated pixels are intended to "bleed over" to correct the truncated area. The color propagation highlight restoration technology is described in patent application CN202080066899.0 for highlight restoration in the image processing pipeline.

[0009] However, the first two methods are both highlight suppression algorithms for the entire image. These algorithms are highly complex and require long computation times, making them difficult to apply to RAW-domain white balance processing modules. Furthermore, because highlight suppression algorithms involve the entire image, they affect the overall image processing results, making it impossible to debug only the highlight areas, resulting in limited debugging flexibility. The third method requires the image processor to retain multiple pixel values ​​in memory. For each cropped region, the raster scan sequence of the image pixel data may require caching the entire cropped region in SRAM to restore the hue of the cropped pixels in that region. This undoubtedly increases the buffer size, increasing the size, cost, and complexity of the hardware.

[0010] In summary, when restoring / correcting the loss of highlight area details and color cast caused by the clipping operation after white balance processing, how to provide a highlight detail recovery device with low complexity, easy implementation, more flexible debugging and fewer side effects for overexposure in the highlight area based on the data bit width limitation is a technical problem that urgently needs to be solved. Summary of the Invention

[0011] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a device, method, and system for highlight recovery of Bayer images. The highlight recovery device can correct detail loss and color cast problems caused by overexposure of highlight areas after white balance processing. It can better maintain the details of the original highlight areas, make color transitions smoother, and significantly improve the color cast phenomenon in the highlight areas. At the same time, combined with the white balance gain, it can adaptively adjust the brightness range of the overexposed areas of each frame of the image, retaining the same-year compression of the maximum brightness value of the original image. The present invention is simple and easy to implement, flexible to debug, and has few side effects. It is particularly suitable for white balance processing of RAW domain data.

[0012] To achieve the above objectives, the present invention provides the following technical solutions:

[0013] A device for restoring highlight of a Bayer image includes a highlight control module, wherein an input end of the highlight control module is connected to an output end of a white balance gain control module, and a Bayer image having three color channels (R, G, and B) is input into the highlight control module after white balance gain processing by the white balance gain control module;

[0014] The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and

[0015] Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and,

[0016] Get the preset minimum value of the highlight area lum min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , the mapping curve is as follows:

[0017]

[0018] G output =G awb ;

[0019]

[0020] in,

[0021] Furthermore, the minimum value of the highlight area lum min Set by the user.

[0022] Furthermore, the bit width range of the RAW domain data is defined by the ISP Pipeline, and the maximum bit width of the RAW domain data is 8 bits, 10 bits, 12 bits, 14 bits, or 16 bits.

[0023] Furthermore, for each frame of image, after obtaining the white balance gain of the multi-channel of the frame of image, the corresponding highlight area maximum value lum is adaptively updated synchronously according to the white balance gain. max .

[0024] The present invention also provides another highlight restoration device for a Bayer image, comprising a clipping module and a highlight control module connected in parallel, wherein the input ends of the clipping module and the highlight control module are both connected to the output end of the white balance gain control module;

[0025] The highlight restoration device has a truncation mode and a highlight control mode. In the truncation mode, a Bayer image having three color channels of R, G, and B is input to the truncation module after being subjected to white balance gain processing by a white balance gain control module. In the highlight control mode, a Bayer image having three color channels of R, G, and B is input to the highlight control module after being subjected to white balance gain processing by a white balance gain control module.

[0026] The clipping module is used to obtain the image source data bit width value of the Bayer image, and perform a clipping operation on the color channel values ​​that exceed the bit width range after the white balance gain processing according to the aforementioned image source data bit width value;

[0027] The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and

[0028] Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and,

[0029] Get the preset minimum value of the highlight area lum min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awbAfter mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , the mapping curve is as follows:

[0030]

[0031] G output =G awb ;

[0032]

[0033] in,

[0034] Furthermore, the minimum value of the highlight area lum min Set by the user.

[0035] The present invention also provides a highlight restoration method for a Bayer image, comprising the following steps:

[0036] Acquire image data information after being processed and output by the white balance gain control module and multiplied by the white balance gain;

[0037] Perform highlight recovery operation for each frame image, wherein the highlight recovery operation is as follows: obtain the white balance gain of the three color channels R, G and B, compare the white balance gain of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and

[0038] Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and,

[0039] Get the preset minimum value of the highlight area lum min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and Bawb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , the mapping curve is as follows:

[0040]

[0041] G output =G awb ;

[0042]

[0043] in,

[0044] Furthermore, the minimum value of the highlight area lum min Set by the user.

[0045] The present invention also provides an image processing system, comprising an ISPPipeline having a plurality of pipeline processing modules, each pipeline processing module being configured to process input image data to generate updated image data and then output the updated image data to the next-stage pipeline processing module;

[0046] The pipeline processing module includes a white balance gain control module and a highlight control module. The input end of the highlight control module is connected to the output end of the white balance gain control module. The Bayer image with three color channels of R, G and B is input into the highlight control module after white balance gain processing by the white balance gain control module.

[0047] The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represents the white balance gains of the three color channels R, G, and B respectively, and MAX(.) represents a function that returns the maximum value; and

[0048] Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and,

[0049] Get the preset minimum value of the highlight area lum min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , the mapping curve is as follows:

[0050]

[0051] G output =G awb ;

[0052]

[0053] in,

[0054] Furthermore, it also includes a truncation module, wherein the truncation module and the highlight control module are arranged in parallel, and the input ends of the truncation module and the highlight control module are both connected to the output end of the white balance gain control module;

[0055] At this time, the image processing system is configured to have a cropping mode and a highlight control mode. In the cropping mode, the Bayer image with three color channels of R, G, and B is input to the cropping module after being subjected to white balance gain processing by the white balance gain control module; in the highlight control mode, the Bayer image with three color channels of R, G, and B is input to the highlight control module after being subjected to white balance gain processing by the white balance gain control module;

[0056] The clipping module is used to obtain the image source data bit width value of the Bayer image, and perform a clipping operation on the color channel values ​​that exceed the bit width range after white balance gain processing according to the aforementioned image source data bit width value.

[0057] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art, as an example:

[0058] On the one hand, the highlight restoration device can correct the loss of detail and color cast caused by overexposure in highlight areas after white balancing. It can effectively preserve the original highlight details and make color transitions smoother and more natural, significantly improving the color cast in highlight areas. The method is simple to implement, flexible to debug, and has few side effects.

[0059] On the other hand, combined with white balance gain, the brightness range of the overexposed area of ​​each frame of the input image can be adaptively adjusted to retain the same-year compression as the maximum brightness value of the original image.

[0060] The technical solution provided by this invention effectively restores details in the original image in highlight areas, significantly improving the purple tint in highlights. Furthermore, the computational method is simple, and image data outside the custom-adjusted highlight brightness range remains unchanged, without affecting details of normal brightness. The processed image data achieves a natural color transition without side effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a comparison diagram of the brightness area mapping curves before and after clipping processing in the prior art.

[0062] Figure 2 This is a connection example diagram of the highlight control module in the highlight recovery device provided by the present invention.

[0063] Figure 3 This is a logic block diagram of the information processing of the highlight control module provided by the present invention.

[0064] Figure 4 This is a comparison diagram of the brightness area mapping curve after processing by the highlight control module provided by the present invention and the brightness area mapping curve after the truncation processing operation.

[0065] Figure 5 This is an example diagram of the display effect after highlight control processing provided by an embodiment of the present invention.

[0066] Figure 6 This is an information processing logic block diagram of the image processing pipeline (ISP Pipeline) provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following is a further detailed description of the Bayer image highlight recovery device, method, and system disclosed in the present invention in conjunction with the accompanying drawings and specific embodiments. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered in isolation, and they can be combined with each other to achieve better technical effects. In the drawings of the following embodiments, the same reference numerals appearing in each drawing represent the same features or components, which can be applied to different embodiments. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0068] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not intended to limit the conditions under which the invention can be implemented. Any structural modification, change in proportional relationship, or adjustment of size should fall within the scope of the technical content disclosed in the invention without affecting the efficacy and purpose of the invention. The scope of the preferred embodiments of the present invention includes alternative implementations, in which the functions can be performed in a non-described or discussed order, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the art of the art to which the embodiments of the present invention belong.

[0069] Techniques, methods, and apparatus known to persons of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0070] Example

[0071] A highlight restoration device for a Bayer image comprises a highlight control module, wherein the input end of the highlight control module is connected to the output end of an AWB Gain control (or automatic white balance control) module. Figure 2 The Bayer image with three color channels of R, G and B is input into the aforementioned highlight control module after being subjected to white balance gain processing by the white balance gain control module.

[0072] The highlight control module is used to correct the overexposure of the highlight area after the white balance processing. Specifically, it is to obtain the white balance gain (AWB gain), compare the gains of the three channels R, G, and B, and determine the maximum gain value; and calculate the mapping curve, and map the mapping curve on the result data after the white balance processing to obtain the final data after highlight control (highlight compression). The mapping curve proposed by the present invention involves parameters such as the minimum value of the highlight area and the maximum value of the highlight area. The parameter highlight minimum valuelum min It can be preset by the system or the user, and the maximum value of the highlight area is adaptively calculated by the white balance gain information. max The maximum brightness value of the original image is retained as much as possible.

[0073] For details, see Figure 3As shown, the highlight control module is configured to perform the following operations: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represents the white balance gain of the three color channels R, G and B respectively, MAX(·) represents the function that returns the maximum value; and, obtains the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and, get the preset minimum value of the highlight area lum min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output .

[0074] The mapping curve is as follows:

[0075]

[0076] G output =G awb ;

[0077]

[0078] in,

[0079] In a preferred embodiment, the minimum value of the highlight area lum min It can be set by the user, so that the user can flexibly set the starting value for highlight control according to actual needs.

[0080] The bit width range of the RAW domain data is defined by the ISP Pipeline, and the maximum bit width of the RAW domain data can be 8 bits, 10 bits, 12 bits, 14 bits, or 16 bits.

[0081] For each frame of image, after obtaining the white balance gain of the multi-channel of the frame image, the corresponding highlight area maximum value lum is adaptively updated according to the white balance gain. max .

[0082] The following describes the technical solution of the present invention in detail by taking the RAW domain data bit width range defined by ISP Pipeline as 12 bits as an example.

[0083] At this time, the RAW domain data range is [0, 4095], that is, a=4095.

[0084] Minimum value of highlight area lum min The value range of lum is as follows: min ∈[0, 4095].

[0085] Maximum value of highlight area lum max The calculation formula is as follows: max =Max gain *4095.

[0086] The calculation formula for the power n in the mapping curve is as follows:

[0087] The calculation formula of the mapping curve is as follows:

[0088]

[0089] G output =G awb ;

[0090]

[0091] Among them, R awb , G awb and B awb Represents the three color channel data after the image source data is multiplied by the white balance gain. The calculation formula is as follows:

[0092] R awb =R input *R gain ;

[0093] G awb =G input *G gain ;

[0094] B awb =B input *B gain ;

[0095] Among them, R awb , G awb and B awbRepresents the R, G, and B color channel data of the image source data input to the AWB Gain control module.

[0096] That is to say, for the R color channel data R after multiplying the white balance gain awb , when R awb Greater than the set lum min When the value is For R color channel data R awb Processing (R awb As input value, R output As the output value), the final highlighted compressed R color channel data R output When R awb Less than or equal to lum min When the value is set, there is no need to use the above mapping curve for mapping. awb , G output The value is equal to G awb For the B color channel data B after multiplying the white balance gain awb , when B awb Greater than the set lum min When the value is For B color channel data B awb Processing (B awb As input value, B output As the output value), the final highlighted compressed B color channel data B output When B awb Less than or equal to lum min When the value is lum, there is no need to use the above mapping curve for mapping. min Mapping the value as the starting point of the mapping curve can effectively reduce side effects such as excessive and unnatural colors.

[0097] In the above scheme, the minimum value of the highlighted area lum min The setting can be flexibly adjusted by the user; the maximum value of the parameter highlight area lum max It can be calculated by adaptive white balance gain, and lum max , retain the maximum brightness value of the original image as much as possible; for each frame of the image, the white balance gain can be updated, and the corresponding lum max value, realizes adaptive adjustment of highlight range per frame. Figure 4As shown in the figure, the highlight area mapping curve comparison after the highlight recovery of the image after white balance gain is performed using the above technical solution is illustrated. The dotted line represents the data after highlight control using the highlight recovery device of the present invention. Compared with the prior art, which directly truncates the overflowed 4095 data (see Figure 1 ) data, the present invention can restore the details of the highlight area, the color transition is more natural, and the color cast phenomenon of the highlight area is improved. Figure 5 Example of sky image shown.

[0098] Another embodiment of the present invention provides a method for restoring highlights of a Bayer image, comprising the following steps.

[0099] S100 , obtaining image data information multiplied by the white balance gain after being processed and output by the white balance gain control module.

[0100] S200 , performing a highlight recovery operation on each frame of image.

[0101] The steps of performing the highlight recovery operation in step S200 may be specifically as follows:

[0102] S210, obtaining the white balance gains of the three color channels R, G and B, and comparing the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represents the white balance gains of the three color channels R, G, and B respectively, and MAX(·) represents a function that returns the maximum value.

[0103] S220, obtaining a predetermined RAW domain data bit width range, and calculating the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a.

[0104] S230, obtaining the preset minimum value lum of the highlight area min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained.output , G output and B output .

[0105] The mapping curve is as follows:

[0106]

[0107] G output =G awb ;

[0108]

[0109] in,

[0110] In this embodiment, preferably, the minimum value of the highlight area lum min Set by the user.

[0111] For other technical features, please refer to the description of the previous embodiment and will not be repeated here.

[0112] Another embodiment of the present invention provides another device for restoring highlights of a Bayer image.

[0113] The highlight recovery device of the Bayer image comprises a clipping module and a highlight control module connected in parallel, wherein the input ends of the clipping module and the highlight control module are both connected to the output end of the white balance gain control module.

[0114] The highlight recovery device has a truncation mode and a highlight control mode. In the truncation mode, the Bayer image with three color channels of R, G and B is input into the aforementioned truncation module after being subjected to white balance gain processing by the white balance gain control module; in the highlight control mode, the Bayer image with three color channels of R, G and B is input into the aforementioned highlight control module after being subjected to white balance gain processing by the white balance gain control module.

[0115] The clipping module is used to obtain the image source data bit width value of the Bayer image, and perform a clipping operation on the color channel values ​​that exceed the bit width range after white balance gain processing according to the aforementioned image source data bit width value.

[0116] By way of example and not limitation, for example, if the source image data of a Bayer image has a bit width of 12 bits, the corresponding image brightness range is [0, 4095]. In cropping mode, the source image data, after being multiplied by the white balance gain, enters the cropping module, which crops values ​​outside the bit width range. This cropping module is not limited by the predetermined RAW data bit width range.

[0117] The white balance gain control module performs white balance processing on the input image source data. The calculation formula of the image source data multiplied by the white balance gain is as follows:

[0118] R awb =R input *R gain ;

[0119] G awb =G input *G gain ;

[0120] B awb =B input *B gain °

[0121] The interception module obtains the input three color channel data R after multiplying the white balance gain awb , G awb and B awb Finally, the color channel data exceeding 4095 is truncated to obtain the three color channel data R after truncation. output , G output and B output , the calculation formula is as follows:

[0122] R output =CLIP[R awb ,0,4095];

[0123] G output =CLIP[G awb ,0,4095];

[0124] B output =CLIP[B awb , 0, 4095].

[0125] Among them, CLIP represents the interception function. awb ,0,4095] means R awb The value is controlled between 0 and 4095. awb When it is greater than 4095, the part greater than 4095 is intercepted, even if R awb Equal to 4095. CLIP[G awb ,0,4095] means G awb The value is controlled between 0 and 4095. awb When it is greater than 4095, the part greater than 4095 is intercepted, even if G awb Equal to 4095. CLIP[B awb ,0,4095] means B awb The value of B is controlled between 0 and 4095. awbWhen it is greater than 4095, the part greater than 4095 is intercepted, even if B awb Equal to 4095. After processing by the truncation module, values ​​that overflow 4095 will be truncated.

[0126] The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and

[0127] Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and,

[0128] Get the preset minimum value of the highlight area lum min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , the mapping curve is as follows:

[0129]

[0130] G output =G awb ;

[0131]

[0132] in,

[0133] For other technical features, please refer to the description of the previous embodiment and will not be repeated here.

[0134] See also Figure 6FIG. 1 is another embodiment of the present invention, which also provides an image processing system.

[0135] The image processing system includes an ISP Pipeline (image processing pipeline) having multiple pipeline processing modules, each pipeline processing module is configured to process input image data to generate updated image data and then output it to the next-level pipeline processing module.

[0136] The pipeline processing module includes a white balance gain control module and a highlight control module. The input end of the highlight control module is connected to the output end of the white balance gain control module. The Bayer image with three color channels of R, G and B is input into the aforementioned highlight control module after white balance gain processing by the white balance gain control module.

[0137] The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain , G gain , B gain ), where R gain , G gain , B gain Represents the white balance gain of the three color channels R, G and B respectively, MAX(·) represents the function that returns the maximum value; and, obtains the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and, get the preset minimum value of the highlight area lum min ,lum min ∈[0, a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output .

[0138] The mapping curve is as follows:

[0139]

[0140] G output =G awb ;

[0141]

[0142] in,

[0143] Preferably, the pipeline processing module may further include a truncation module, the truncation module and the highlight control module are arranged in parallel, and the input ends of the truncation module and the highlight control module are both connected to the output end of the white balance gain control module.

[0144] At this time, the image processing system is configured to have a cropping mode and a highlight control mode. In the cropping mode, the Bayer image with three color channels of R, G and B is input into the aforementioned cropping module after being subjected to white balance gain processing by the white balance gain control module; in the highlight control mode, the Bayer image with three color channels of R, G and B is input into the aforementioned highlight control module after being subjected to white balance gain processing by the white balance gain control module.

[0145] The clipping module is used to obtain the image source data bit width value of the Bayer image, and perform a clipping operation on the color channel values ​​that exceed the bit width range after white balance gain processing according to the aforementioned image source data bit width value.

[0146] For other technical features, please refer to the description of the previous embodiment and will not be repeated here.

[0147] In the above description, the disclosure of the present invention is not intended to limit itself to these aspects. Rather, within the scope of the intended protection of the present disclosure, the components can be selectively and operationally combined in any number. In addition, terms such as "including", "encompassing" and "having" should be interpreted as inclusive or open by default, rather than exclusive or closed, unless they are explicitly defined to the contrary. All technical, scientific or other terms have the meaning understood by those skilled in the art unless they are defined to the contrary. Common terms found in dictionaries should not be interpreted too idealistically or too impractically in the context of relevant technical documents, unless the present disclosure explicitly defines them as such. Any changes and modifications made by a person of ordinary skill in the field of the present invention based on the above disclosure are within the scope of protection of the claims.

Claims

1. A device for restoring highlights of a Bayer image, characterized by: The highlight control module includes an input end connected to an output end of a white balance gain control module, and a Bayer image having three color channels of R, G, and B is input into the highlight control module after being subjected to white balance gain processing by the white balance gain control module; The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain ,G gain ,B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and, Get the preset minimum value of the highlight area lum min ,lum min ∈[0,a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , The mapping curve is as follows: G output =G awb ; in, 2. The highlight restoration device for a Bayer image according to claim 1, wherein: The minimum value of the highlight area lum min Set by the user.

3. The highlight recovery device for a Bayer image according to claim 1, wherein: The bit width range of the RAW domain data is defined by the ISP Pipeline, and the maximum bit width of the RAW domain data is 8 bits, 10 bits, 12 bits, 14 bits, or 16 bits.

4. The highlight restoration device for a Bayer image according to claim 1, wherein: For each frame of image, after obtaining the white balance gain of the multi-channel of the frame image, the corresponding highlight area maximum value lum is adaptively updated according to the white balance gain. max .

5. A device for restoring highlights of a Bayer image, characterized in that: It includes a clipping module and a highlight control module connected in parallel, wherein the input ends of the clipping module and the highlight control module are both connected to the output end of the white balance gain control module; The highlight restoration device has a truncation mode and a highlight control mode. In the truncation mode, a Bayer image having three color channels of R, G, and B is input to the truncation module after being subjected to white balance gain processing by a white balance gain control module. In the highlight control mode, a Bayer image having three color channels of R, G, and B is input to the highlight control module after being subjected to white balance gain processing by a white balance gain control module. The clipping module is used to obtain the image source data bit width value of the Bayer image, and perform a clipping operation on the color channel values ​​that exceed the bit width range after the white balance gain processing according to the aforementioned image source data bit width value; The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain ,G gain ,B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and, Get the preset minimum value of the highlight area lum min ,lum min ∈[0,a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , The mapping curve is as follows: G output =G awb ; in, 6. The highlight restoration device for a Bayer image according to claim 5, characterized in that: The minimum value of the highlight area lum min Set by the user.

7. A highlight restoration method for a Bayer image, characterized in that Including steps: Acquire image data information after being processed and output by the white balance gain control module and multiplied by the white balance gain; Perform highlight recovery operation for each frame image, wherein the highlight recovery operation is as follows: obtain the white balance gain of the three color channels R, G and B, compare the white balance gain of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain ,G gain ,B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and, Get the preset minimum value of the highlight area lum min ,lum min ∈[0,a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , the mapping curve is as follows: G output =G awb ; in, 8. The highlight restoration method of a Bayer image according to claim 7, wherein: The minimum value of the highlight area lum min Set by the user.

9. An image processing system comprising an ISP pipeline having a plurality of pipeline processing modules, each pipeline processing module being configured to process input image data to generate updated image data and then output the updated image data to the next pipeline processing module, characterized in that: The pipeline processing module includes a white balance gain control module and a highlight control module. The input end of the highlight control module is connected to the output end of the white balance gain control module. The Bayer image with three color channels of R, G and B is input into the highlight control module after white balance gain processing by the white balance gain control module. The highlight control module is configured to: for each frame of image, obtain the white balance gains of the three color channels R, G and B, compare the white balance gains of the three color channels R, G and B to determine the maximum gain value Max gain , let Max gain =MAX(R gain ,G gain ,B gain ), where R gain , G gain , B gain Represent the white balance gains of the three color channels R, G, and B respectively, MAX(·) represents the function that returns the maximum value; and Get the predetermined RAW domain data bit width range, according to the brightness value a corresponding to the highest bit width value and the aforementioned maximum gain value Max gain Calculate the maximum value of the highlight area lum max , let lum max =Max gain *a; and, Get the preset minimum value of the highlight area lum min ,lum min ∈[0,a], the three color channel data R output by the white balance gain control module after multiplying the white balance gain awb , G awb and B awb After mapping according to the corresponding mapping curve, the three color channel data R after highlight compression are obtained. output , G output and B output , The mapping curve is as follows: G output =G awb ; in, 10. The image processing system according to claim 9, wherein: It also includes a truncation module, wherein the truncation module and the highlight control module are arranged in parallel, and the input ends of the truncation module and the highlight control module are both connected to the output end of the white balance gain control module; At this time, the image processing system is configured to have a cropping mode and a highlight control mode. In the cropping mode, the Bayer image with three color channels of R, G, and B is input to the cropping module after being subjected to white balance gain processing by the white balance gain control module; in the highlight control mode, the Bayer image with three color channels of R, G, and B is input to the highlight control module after being subjected to white balance gain processing by the white balance gain control module; The clipping module is used to obtain the image source data bit width value of the Bayer image, and perform a clipping operation on the color channel values ​​that exceed the bit width range after white balance gain processing according to the aforementioned image source data bit width value.

Images