Image compensation method, device and equipment and readable medium

Abstract

The invention provides an image compensation method, device and equipment and a readable medium, and belongs to the technical field of signal processing. The method comprises the following steps: acquiring original image data acquired by an RGB-IR image sensor; determining full-size image data corresponding to an infrared channel and full-size image data corresponding to each color channel after infrared separation according to the original image data; determining a dynamic range gain coefficient corresponding to a compensation area according to the original image data and the full-size image data of the target channel; wherein the full-size image data of the target channel comprises at least one of the following: the full-size image data of the infrared channel and the full-size image data of each color channel after infrared separation; and according to the dynamic range gain coefficient corresponding to the compensation area, compensating the full-size image data after infrared separation of each color channel of the compensation area. The method is used for improving the dynamic range of the image after infrared separation and improving the image quality.

Description

Technical Field

[0001] The present disclosure relates to the field of signal processing technology, and in particular to an image compensation method, device, equipment and readable medium. Background Art

[0002] The image quality of the traditional RGB (R, red; G, green; B, blue) image sensor is low in low or very low illumination. Therefore, in some applications, RGB-IR image sensors are used to collect data. IR (Infrared) outputs RGB color images when the illumination is sufficient, and outputs IR images when the illumination is low or very low.

[0003] The RGB-IR image sensor is based on the traditional Bayer color filter array, which modifies some pixels into IR pixels. The IR pixels only allow infrared light to pass through. Figure 1 The commonly used pixel arrangement of RGB-IR image sensors is given. Figure 1 The middle left picture is in 4×4 format. Figure 1 The middle right image is in 2×2 format. Compared with the traditional Bayer format image sensor, the RGB-IR image sensor has the ability to output not only RGB color images, but also IR black and white images. Therefore, RGB-IR technology is widely used in automotive, security, monitoring, face recognition, and liveness detection.

[0004] RGB-IR technology is used to process the raw data collected by the RGB-IR image sensor. The RGB-IR image sensor can be divided into 4x4 and 2x2 formats.

[0005] In RGB-IR technology, such as Figure 2 As shown in the figure, for the 4x4 color filter array in RGB-IR, the B channel is restored to the R channel, and the IR channel is restored to the B channel to obtain the traditional 2x2 bayer format; for the 2x2 color filter array in RGB-IR, the IR channel is restored to the G channel to obtain the traditional 2x2 bayer format. When the illumination is sufficient, if you want to restore reasonable colors, you need to separate the IR components in the three channels of R, G, and B. However, after separating the IR components, the dynamic range of the image will be reduced, making the image darker as a whole, and abnormal colors will appear around the overexposed area. Therefore, it is necessary to restore the dynamic range of the image to improve the image quality. Summary of the invention

[0006] Embodiments of the present disclosure provide an image compensation method, apparatus, device, and readable medium.

[0007] A first aspect of the embodiments of the present disclosure provides an image compensation method, including:

[0008] Obtain the original image data collected by the RGB-IR image sensor;

[0009] Determine, based on the original image data, full-size image data corresponding to the infrared channel and full-size image data corresponding to each color channel after infrared separation;

[0010] Determine a dynamic range gain coefficient corresponding to the compensation area according to the original image data and the full-size image data of the target channel; wherein the full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel, and the full-size image data of each color channel after infrared separation;

[0011] The full-size image data after infrared separation of each color channel of the compensation area is compensated according to the dynamic range gain coefficient corresponding to the compensation area.

[0012] A second aspect of the present disclosure provides an image compensation device, including:

[0013] An acquisition module is used to acquire the original image data collected by the RGB-IR image sensor;

[0014] An infrared separation module, used to determine, based on the original image data, full-size image data corresponding to the infrared channel and full-size image data corresponding to each color channel after infrared separation;

[0015] A coefficient determination module, used to determine the dynamic range gain coefficient corresponding to the compensation area according to the original image data and the full-size image data of the target channel; wherein the full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel, and the full-size image data of each color channel after infrared separation;

[0016] The compensation module is used to compensate the full-size image data of each color channel of the compensation area after infrared separation according to the dynamic range gain coefficient corresponding to the compensation area.

[0017] A third aspect of the present disclosure provides an electronic device, including:

[0018] at least one processor;

[0019] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect;

[0020] At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

[0021] A fourth aspect of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to the first aspect is implemented.

[0022] The embodiments of the present disclosure have the following advantages:

[0023] The original image data collected by the RGB-IR image sensor is used to determine the full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel after infrared separation, and the dynamic range gain coefficient corresponding to the compensation area is determined according to the original image data and the full-size image data of the target channel. The full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel and the full-size image data of each color channel after infrared separation. Then, according to the dynamic range gain coefficient corresponding to the compensation area, the full-size image data of each color channel after infrared separation of the compensation area is compensated, so that the dynamic range of the image after infrared separation can be guaranteed through compensation, the image quality can be guaranteed, and the problem of image brightness reduction caused by infrared separation can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of pixel arrangement of a traditional RGB-IR image sensor;

[0025] Figure 2 It is a schematic diagram of the format restoration process in the traditional RGB-IR technology;

[0026] Figure 3 A schematic diagram of a flow chart of an image compensation method provided in an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of a process for determining a dynamic range gain coefficient of a compensation region in a first embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram of a process for determining a dynamic range gain coefficient of a compensation region in the second method provided in an embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of a process for determining a dynamic range gain coefficient based on an IR channel provided in an embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of a compensation area division and dynamic range gain coefficient determination process provided in an embodiment of the present disclosure;

[0031] Figure 8 It is a structural schematic diagram of an image compensation device provided in an embodiment of the present disclosure;

[0032] Fig. 9is a schematic diagram of an example structure of an image compensation device provided in an embodiment of the present disclosure;

[0033] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0035] As used in this disclosure, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0036] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure.As used in the present disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0037] When the terms “comprising” and / or “made of…” are used in the present disclosure, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0038] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0039] The embodiment of the present disclosure provides an image compensation method, which can be applied to any electronic device, and is used to compensate the dynamic range of the image corresponding to the color channels (i.e., R channel, G channel, and B channel) of the RGB-IR image sensor in the image signal processor. The image compensation method is applicable to an image signal processor (ISP) or an image processor (IP) including an RGB-IR processing module, and is applicable to the post-processing of a monocular RGB-IR image sensor, and is applicable to scenes where the IR component in visible light needs to be separated.

[0040] Figure 3 The figure is a schematic flow chart of the image compensation method provided by the embodiment of the present disclosure, and the method mainly comprises the following steps:

[0041] Step 301: Acquire raw image data collected by an RGB-IR image sensor.

[0042] The original image data refers to the raw image data collected by the RGB-IR image sensor, including but not limited to raw image data in 4x4 and 2x2 formats.

[0043] Step 302: Determine, based on the original image data, the full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel after infrared separation.

[0044] The full-size image data corresponding to each color channel after infrared separation includes the full-size image data corresponding to each of the three channels R, G and B after infrared separation.

[0045] In some embodiments, determining the full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel after infrared separation based on the original image data includes: obtaining the full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel based on the original image data; performing infrared separation on the full-size image data corresponding to each color channel using the full-size image data corresponding to the infrared channel and the separation coefficient corresponding to each color channel to obtain the full-size image data after separation of each color channel.

[0046] In an exemplary embodiment, full-size image data corresponding to the infrared channel and full-size image data corresponding to each color channel are obtained based on the original image data, including: using each pixel value of the target channel in the original image data to interpolate the pixel position of the non-target channel to obtain the full-size image data corresponding to the target channel, and the target channel is any one of the infrared channel and each color channel. The full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel are obtained in the same manner. Among them, the interpolation method includes but is not limited to bilinear interpolation (bilinear), bicubic interpolation (bicubic), etc., and the specific interpolation method used is not limited here.

[0047] In an exemplary embodiment, infrared separation is performed on the full-size image data corresponding to the infrared channel and the separation coefficients corresponding to each color channel to obtain the full-size image data after separation of each color channel, including: performing the following process on each color channel respectively, multiplying each pixel value in the full-size image data corresponding to the infrared channel by the separation coefficient corresponding to the color channel to obtain the separated image data corresponding to the color channel; subtracting the full-size image data corresponding to the color channel from the separated image data corresponding to the color channel to obtain the full-size image data after separation of the color channel.

[0048] The separation coefficient corresponding to each color channel is obtained by pre-calibration, and the specific method for obtaining the separation coefficient is not limited here.

[0049] Step 303, determining the dynamic range gain coefficient corresponding to the compensation area based on the original image data and the full-size image data of the target channel; wherein the full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel, and the full-size image data of each color channel after infrared separation.

[0050] In some embodiments, the compensation area is any one of the pixel blocks obtained by dividing the pixel array of the RGB-IR image sensor, and the size of the compensation area is N×M, where N and M are both integers greater than 1;

[0051] or,

[0052] The compensation area is the entire pixel array of the RGB-IR image sensor.

[0053] Wherein, when the compensation area is any one of the pixel blocks obtained by segmenting the pixel array of the RGB-IR image sensor, the dynamic range gain coefficient corresponding to each of the compensation areas is determined respectively. In this paper, only the determination process of the dynamic range gain coefficient of one compensation area is taken as an example for explanation, and the dynamic range gain coefficients of other compensation areas are executed with reference to this process.

[0054] In some embodiments, the full-size image data of the target channel includes: the full-size image data of the infrared channel, and the full-size image data of each color channel after infrared separation.

[0055] Specifically, determining the dynamic range gain coefficient corresponding to the compensation area according to the original image data and the full-size image data of the target channel includes: determining the gain coefficient corresponding to each color channel of the compensation area according to the original image data and the full-size image data corresponding to each color channel after infrared separation; determining the dynamic range gain coefficient corresponding to the compensation area as: any one of the maximum value, the minimum value and the mean value of the gain coefficient corresponding to each color channel of the compensation area.

[0056] In some embodiments, the gain coefficients corresponding to the color channels of the compensation area are determined respectively according to the original image data and the full-size image data corresponding to the infrared separation of each color channel, including but not limited to the following methods:

[0057] Method 1

[0058] For any of the color channels in the compensation area, the following process is performed:

[0059] Determine the pixel mean value in the compensation area in the full-size image data after infrared separation of the color channel as the first mean value; determine the pixel mean value of the color channel in the compensation area in the original image data as the second mean value; determine the ratio of the first mean value to the second mean value as the gain coefficient corresponding to the color channel of the compensation area.

[0060] In an exemplary embodiment, Figure 4 The figure shows a schematic diagram of the process of determining the dynamic range gain coefficient of the compensation area in the first method, and the process includes:

[0061] First, the pixel averages of the full-size image data of the three channels R, G and B after infrared separation of the corresponding compensation area are counted, that is, the first averages, which are represented as R_avg1, G_avg1, and B_avg1 respectively;

[0062] Then, the pixel averages of the three channels R, G and B of the original image data (ie, raw image data) of the corresponding compensation area are counted, that is, the second averages, which are represented as R_avg2, G_avg2, and B_avg2 respectively;

[0063] After that, the gain coefficients of the three channels R, G and B are calculated according to the following formula:

[0064] R_gain1=R_avg1 / R_avg2

[0065] G_gain1=G_avg1 / G_avg2

[0066] B_gain1=B_avg1 / B_avg2

[0067] Among them, R_gain1 represents the gain coefficient of the R channel, G_gain1 represents the gain coefficient of the G channel, and B_gain1 represents the gain coefficient of the B channel;

[0068] Finally, the dynamic range compensation gain coefficient corresponding to the compensation area is calculated (expressed as gain1). Here, gain1 can be selected from the maximum value, minimum value, and average value of R_gain1, G_gain1, and B_gain1. The calculation formulas are as follows:

[0069] gain1=max(R_gain1,G_gain1,B_gain1)

[0070] gain1=min(R_gain1,G_gain1,B_gain1)

[0071] gain1=(R_gain1+G_gain1+B_gain1) / 3

[0072] Wherein, when the compensation area is the entire pixel array of the RGB-IR image sensor, the gain coefficient is calculated using the original image data and the entire full-size image data of each channel; when the compensation area is any one of a plurality of pixel array blocks obtained by dividing the pixel array of the RGB-IR image sensor, the gain coefficient is calculated using the original image data corresponding to the compensation area and the full-size image data of each channel in the compensation area. For example, when the compensation area is the N×M pixel array block in the upper left corner of the pixel array, the gain coefficient is calculated using the image data of the N×M block in the upper left corner of the original image data and the image data of the N×M block in the upper left corner of the full-size image data corresponding to each channel.

[0073] Method 2

[0074] For any of the color channels in the compensation area, the following process is performed:

[0075] Determine the pixel mean corresponding to the unsaturated area in the compensation area in the full-size channel data after infrared separation of the color channel as the third mean; determine the pixel mean of the unsaturated area of ​​the color channel in the compensation area in the original image data as the fourth mean; and use the ratio of the third mean to the fourth mean as the gain coefficient corresponding to the color channel in the compensation area.

[0076] The process of obtaining the unsaturated region includes: comparing each pixel value of the image corresponding to the compensation region with the saturation threshold value, and determining the saturated region and the unsaturated region of the image corresponding to the compensation region according to the comparison result. For example, when the pixel value is greater than or equal to the saturation threshold value, it is determined that the corresponding pixel belongs to the saturated region; when the pixel value is less than the saturation threshold value, it is determined that the corresponding pixel belongs to the unsaturated region.

[0077] In an exemplary embodiment, Figure 5 The figure shows a schematic diagram of the process of determining the dynamic range gain coefficient of the compensation area in the second method, and the process includes:

[0078] First, determine the unsaturated areas in the full-size image data of the R, G, and B channels after infrared separation corresponding to the compensation area, and count the pixel averages of the unsaturated areas of the R, G, and B channels after infrared separation corresponding to the compensation area, that is, the third averages, which are represented as R_avg3, G_avg3, and B_avg3 respectively;

[0079] Then, the unsaturated regions of the three channels R, G and B in the original image data (i.e., raw image data) corresponding to the compensation area are determined, and the pixel averages of the unsaturated regions of the three channels R, G and B in the original image data (i.e., raw image data) corresponding to the compensation area are counted, which are expressed as R_avg4, G_avg4, and B_avg4 respectively;

[0080] Next, calculate the gain coefficients of the three channels R, G, and B respectively according to the following formula:

[0081] R_gain2=R_avg3 / R_avg4

[0082] G_gain2=G_avg3 / G_avg4

[0083] B_gain2=B_avg3 / B_avg4

[0084] Among them, R_gain2 represents the gain coefficient of the R channel, G_gain2 represents the gain coefficient of the G channel, and B_gain2 represents the gain coefficient of the B channel;

[0085] Finally, the dynamic range compensation gain coefficient corresponding to the compensation area is calculated (expressed as gain2). Here, gain2 can be selected from the maximum, minimum, and average values ​​of R_gain2, G_gain2, and B_gain2. The calculation formulas are as follows:

[0086] gain2=max(R_gain2,G_gain2,B_gain2)

[0087] gain2=min(R_gain2,G_gain2,B_gain2)

[0088] gain2=(R_gain2+G_gain2+B_gain2) / 3

[0089] Wherein, when the compensation area is the entire pixel array of the RGB-IR image sensor, the gain coefficient is calculated using the original image data and the entire full-size image data of each channel; when the compensation area is any one of a plurality of pixel array blocks obtained by dividing the pixel array of the RGB-IR image sensor, the gain coefficient is calculated using the original image data corresponding to the compensation area and the full-size image data of each channel in the compensation area. For example, when the compensation area is the N×M pixel array block in the upper left corner of the pixel array, the gain coefficient is calculated using the image data of the N×M block in the upper left corner of the original image data and the image data of the N×M block in the upper left corner of the full-size image data corresponding to each channel.

[0090] Method 3

[0091] According to the full-scale image data after infrared separation of each color channel, the saturated area corresponding to each color channel in the compensation area is determined; the union of the saturated areas corresponding to each color channel in the compensation area is used as the target saturated area; the area corresponding to the target saturated area is removed from the compensation area of ​​the original image data to obtain the unsaturated area in the compensation area of ​​the original image data; the target saturated areas are removed from the compensation area of ​​the full-scale image data after infrared separation of each color channel to obtain the unsaturated areas in the compensation area corresponding to each color channel; according to the unsaturated area in the compensation area of ​​the original image data and the unsaturated area in the compensation area corresponding to each color channel, the gain coefficient corresponding to each color channel in the compensation area is determined respectively.

[0092] In some embodiments corresponding to the third mode, determining the gain coefficient corresponding to each color channel of the compensation area according to the unsaturated area in the compensation area of ​​the original image data and the unsaturated area in the compensation area corresponding to each color channel, respectively, includes:

[0093] For any of the color channels in the compensation area, the following process is performed:

[0094] Determine a pixel mean of an unsaturated area of ​​the compensation area corresponding to the color channel as a fifth mean; determine a pixel mean of an unsaturated area of ​​the compensation area of ​​the original image data corresponding to the color channel as a sixth mean; and use a ratio of the fifth mean to the sixth mean as a gain coefficient corresponding to the color channel of the compensation area.

[0095] In an exemplary embodiment, a schematic diagram of a process for determining a dynamic range gain coefficient of a compensation region in method 3 includes:

[0096] First, determine the saturated areas of the full-size image data of the R, G, and B channels after infrared separation corresponding to the compensation area, and count the saturated areas of the R, G, and B channels after infrared separation corresponding to the compensation area, which are expressed as R_sat1, G_sat1, and B_sat1, respectively, and perform a union operation on the three to obtain the saturated area, which is expressed as:

[0097] sat_area=R_sat1∪G_sat1∪B_sat1

[0098] Furthermore, the saturated areas are removed from the full-size image data of the R, G, and B channels after infrared separation corresponding to the compensation area, and the unsaturated areas of the R, G, and B channels corresponding to the compensation area (denoted as non_sat_area) are obtained, which can be expressed as:

[0099] non_sat_area = 1 - sat_area

[0100] Then, the pixel averages of the three channels R, G, and B in the non_sat_area after infrared separation of the corresponding compensation area are counted, that is, the fifth averages, which are represented as R_avg5, G_avg5, and B_avg5 respectively;

[0101] Afterwards, the pixel averages of the three channels R, G and B in the original image data (ie, raw image data) corresponding to the compensation area in the non_sat_area are determined, namely, the sixth averages, which are represented as R_avg6, G_avg6, and B_avg6 respectively;

[0102] Next, calculate the gain coefficients of the three channels R, G, and B respectively according to the following formula:

[0103] R_gain3=R_avg5 / R_avg6

[0104] G_gain3=G_avg5 / G_avg6

[0105] B_gain3=B_avg5 / B_avg6

[0106] Among them, R_gain3 represents the gain coefficient of the R channel, G_gain3 represents the gain coefficient of the G channel, and B_gain3 represents the gain coefficient of the B channel;

[0107] Finally, the dynamic range compensation gain coefficient corresponding to the compensation area is calculated (expressed as gain3). Here, gain3 can be selected from the maximum value, minimum value, and mean value of R_gain3, G_gain3, and B_gain3. The calculation formulas are as follows:

[0108] gain3=max(R_gain3,G_gain3,B_gain3)

[0109] gain3=min(R_gain3,G_gain3,B_gain3)

[0110] gain3=(R_gain3+G_gain3+B_gain3) / 3

[0111] Wherein, when the compensation area is the entire pixel array of the RGB-IR image sensor, the gain coefficient is calculated using the original image data and the entire full-size image data of each channel; when the compensation area is any one of a plurality of pixel array blocks obtained by dividing the pixel array of the RGB-IR image sensor, the gain coefficient is calculated using the original image data corresponding to the compensation area and the full-size image data of each channel in the compensation area. For example, when the compensation area is the N×M pixel array block in the upper left corner of the pixel array, the gain coefficient is calculated using the image data of the N×M block in the upper left corner of the original image data and the image data of the N×M block in the upper left corner of the full-size image data corresponding to each channel.

[0112] In some embodiments, the full-size image data of the target channel includes: full-size image data corresponding to the infrared channel.

[0113] Specifically, the dynamic range gain coefficient corresponding to the compensation area is determined based on the original image data and the full-size image data of the target channel, including: obtaining the bit width value corresponding to the original image data; obtaining a reference value corresponding to the bit width value; determining the pixel mean in the compensation area in the full-size image data corresponding to the infrared channel as the seventh mean; determining the difference between the reference value and the seventh mean, and taking the ratio of the reference value to the difference as the dynamic range gain coefficient corresponding to the compensation area.

[0114] Exemplarily, the reference numerical value corresponding to the bit width value is the maximum value that can be represented by the bit width value. For example, when the bit width value is 8, the maximum value that can be represented is 255.

[0115] In an exemplary embodiment, Figure 6 The figure shows a schematic diagram of the process of determining the dynamic range gain coefficient based on the IR channel, which mainly includes the following processes:

[0116] Step 601, obtaining full-size image data of the IR channel;

[0117] Step 602, calculating the pixel average of the full-size image data of the infrared channel corresponding to the compensation area, expressed as IR_avg1;

[0118] Step 603, calculate the dynamic range compensation gain coefficient according to the following formula:

[0119] gain4=white_level / (white_level-IR_avg1)

[0120] White_level is related to the bit width of the input image and is calculated as follows:

[0121] white_level=2 bitwidth -1

[0122] Among them, gain4 represents the dynamic range compensation gain coefficient corresponding to the compensation area, bitwidth represents the bit width value, and white_level represents the maximum value that can be represented by the bit width value, that is, the reference value.

[0123] Step 304 , compensating the full-size image data of each color channel of the compensation area after infrared separation according to the dynamic range gain coefficient corresponding to the compensation area.

[0124] In some embodiments, when the compensation area is any one of the pixel blocks obtained by segmenting the pixel array of the RGB-IR image sensor, the dynamic range gain coefficient corresponding to each of the compensation areas is determined respectively, and step 304 is performed for each compensation area respectively, so as to obtain complete full-size image data after infrared separation of each color channel after compensation.

[0125] In some embodiments, compensating the full-size image data of each color channel of the compensation area after infrared separation according to the dynamic range gain coefficient corresponding to the compensation area includes:

[0126] For any of the color channels in the compensation area, the following process is performed:

[0127] Each pixel value belonging to the compensation area in the full-size image data after infrared separation of the color channel is multiplied by the dynamic range gain coefficient corresponding to the compensation area to obtain the image data of the compensation area after the color channel is compensated.

[0128] In some embodiments, after performing the step of compensating the full-size image data after infrared separation of each color channel of the compensation area according to the dynamic range gain coefficient corresponding to the compensation area, the method further includes: down-sampling the compensated full-size image data after infrared separation of each color channel to obtain image data in Bayer format. The down-sampling method corresponds to the Bayer format, so that the image data in the Bayer format is obtained after down-sampling.

[0129] In an exemplary embodiment, Figure 7 The figure shows the division of the compensation area and the schematic diagram of the dynamic range gain coefficient determination process, which mainly includes the following processes:

[0130] First, the full-size image data of each channel of R, G, B, and IR after infrared separation and the input raw image data are divided into n×m pixel blocks of the same size. The size of the pixel block is N×M, and one pixel block is a compensation area.

[0131] Then, a corresponding dynamic range compensation gain coefficient is calculated for each block. The calculation method may refer to the determination process of the dynamic range compensation gain coefficient of the compensation area described above.

[0132] In some embodiments, since the pixel array is divided into a plurality of compensation areas, artifacts may be introduced into the image obtained by gain compensation. The artifacts may be removed by performing linear interpolation on the compensated image.

[0133] In the disclosed embodiment, the original image data collected by the RGB-IR image sensor is used to determine the full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel after infrared separation, and the dynamic range gain coefficient corresponding to the compensation area is determined based on the original image data and the full-size image data of the target channel. The full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel and the full-size image data of each color channel after infrared separation. Then, based on the dynamic range gain coefficient corresponding to the compensation area, the full-size image data of each color channel after infrared separation of the compensation area is compensated, so that the dynamic range of the image after infrared separation can be guaranteed through compensation, the image quality can be guaranteed, and the problem of image brightness reduction caused by infrared separation can be avoided.

[0134] Through dynamic range compensation, on the one hand, the overall dynamic range of the image can be effectively restored, while the brightness of the image can be improved and the expression of details in the dark area of ​​the image can be enhanced; on the other hand, the color expression ability of the transition zone between the saturated area and the unsaturated area in the image can be improved, and the appearance of erroneous colors can be suppressed.

[0135] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this disclosure. Adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this disclosure.

[0136] An image compensation device is provided in an embodiment of the present disclosure. The specific implementation of the device can refer to the relevant description of the method embodiment, which will not be repeated here. Figure 8 The schematic diagram of the structure of the device is shown, which mainly includes:

[0137] An acquisition module 801 is used to acquire raw image data collected by the RGB-IR image sensor;

[0138] Infrared separation module 802, used to determine the full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel after infrared separation according to the original image data;

[0139] The coefficient determination module 803 is used to determine the dynamic range gain coefficient corresponding to the compensation area according to the original image data and the full-size image data of the target channel; wherein the full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel, and the full-size image data of each color channel after infrared separation;

[0140] The compensation module 804 is used to compensate the full-size image data of each color channel of the compensation area after infrared separation according to the dynamic range gain coefficient corresponding to the compensation area.

[0141] In an exemplary embodiment, Fig. 9 The figure shows an example structural diagram of an image compensation device, which mainly includes:

[0142] An input unit, used for receiving raw image data in a 4x4 format or a 2x2 format output by the RGB-IR image sensor; the input unit belongs to the acquisition module 801;

[0143] An interpolation unit, used for interpolating the raw image data to obtain full-size image data of the infrared channel and each color channel;

[0144] The IR separation unit is used to perform infrared separation on the full-size image data of each color channel using the full-size image data of the infrared channel to obtain the full-size image data of each color channel after infrared separation;

[0145] The interpolation unit and the IR separation unit belong to the infrared separation module 802;

[0146] A dynamic range compensation unit, including a coefficient determination module 803 and a compensation module 804, is used to determine a dynamic range gain coefficient and perform dynamic range compensation on the full-size image data after infrared separation of each color channel;

[0147] The down-sampling unit is used to down-sample the full-size image data after infrared separation of each color channel after compensation to obtain image data in Bayer format.

[0148] The functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the method embodiments. The specific implementation and technical effects thereof can be referred to the description of the method embodiments above, and will not be described again here for the sake of brevity.

[0149] It should be noted that all modules involved in this embodiment are logic modules. In practical applications, a logic unit may be a physical unit, or a part of a physical unit, or may be implemented as a combination of multiple physical units. In addition, in order to highlight the innovative part of the present disclosure, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by the present disclosure, but this does not mean that there are no other units in this embodiment.

[0150] Reference Fig.10 , an embodiment of the present disclosure provides an electronic device, comprising:

[0151] at least one processor 1001;

[0152] A memory 1002 having at least one program stored thereon, and when the at least one program is executed by the at least one processor, the at least one processor implements the above method;

[0153] At least one I / O interface 1003 is connected between the processor and the memory and is configured to implement information exchange between the processor and the memory.

[0154] Among them, the processor 1001 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 1002 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 1003 is connected between the processor 1001 and the memory 1002, and can realize information interaction between the processor 1001 and the memory 1002, including but not limited to a data bus (Bus), etc.

[0155] In some embodiments, the processor 1001 , the memory 1002 , and the I / O interface 1003 are connected to each other through a bus, and further connected to other components of the computing device.

[0156] This embodiment further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in this embodiment is implemented. To avoid repeated description, the specific steps of the method are not repeated here.

[0157] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods applied for above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0158] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0159] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0160] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. An image compensation method, characterized in that: include: Obtain the original image data collected by the RGB-IR image sensor; Determine, based on the original image data, full-size image data corresponding to the infrared channel and full-size image data corresponding to each color channel after infrared separation; Determine a dynamic range gain coefficient corresponding to the compensation area according to the original image data and the full-size image data of the target channel; wherein the full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel, and the full-size image data of each color channel after infrared separation; The full-size image data after infrared separation of each color channel of the compensation area is compensated according to the dynamic range gain coefficient corresponding to the compensation area.

2. The method according to claim 1, characterized in that The compensation area is any one of the pixel blocks obtained by dividing the pixel array of the RGB-IR image sensor, and the size of the compensation area is N×M, where N and M are both integers greater than 1; or, The compensation area is the entire pixel array of the RGB-IR image sensor.

3. The method according to claim 1, characterized in that The determining, according to the original image data and the full-size image data of the target channel, a dynamic range gain coefficient corresponding to the compensation area comprises: Determine the gain coefficients corresponding to the color channels of the compensation area respectively according to the original image data and the full-size image data corresponding to the infrared separation of the color channels; The dynamic range gain coefficient corresponding to the compensation area is determined as: any one of a maximum value, a minimum value and a mean value of the gain coefficients corresponding to the color channels of the compensation area.

4. The method according to claim 3, characterized in that Determining the gain coefficients corresponding to the color channels of the compensation area respectively according to the original image data and the full-size image data corresponding to the infrared separation of the color channels includes: For any of the color channels in the compensation area, the following process is performed: Determine a pixel mean value in the compensation area of ​​the full-size image data after infrared separation of the color channel as a first mean value; Determine a pixel mean value of the color channel in the compensation area in the original image data as a second mean value; A ratio of the first mean value to the second mean value is determined as a gain coefficient corresponding to the color channel of the compensation area.

5. The method according to claim 3, characterized in that: Determining the gain coefficients corresponding to the color channels of the compensation area respectively according to the original image data and the full-size image data corresponding to the infrared separation of the color channels includes: For any of the color channels in the compensation area, the following process is performed: Determine a pixel mean value corresponding to an unsaturated area in the compensation area in the full-size channel data after infrared separation of the color channel as a third mean value; Determine a pixel mean value of an unsaturated region of the color channel in the compensation region in the original image data as a fourth mean value; The ratio of the third mean value to the fourth mean value is used as a gain coefficient corresponding to the color channel of the compensation area.

6. The method according to claim 3, characterized in that: Determining the gain coefficients corresponding to the color channels of the compensation area respectively according to the original image data and the full-size image data corresponding to the infrared separation of the color channels includes: Determine the saturation area corresponding to each color channel in the compensation area according to the full-size image data after infrared separation of each color channel; Taking the union of the saturated areas corresponding to the color channels in the compensation area as the target saturated area; removing an area corresponding to the target saturated area from the compensation area of ​​the original image data to obtain an unsaturated area within the compensation area of ​​the original image data; Removing the target saturated areas from the compensation areas of the full-scale image data after infrared separation of each color channel, respectively, to obtain unsaturated areas in the compensation areas corresponding to each color channel; According to the unsaturated area in the compensation area of ​​the original image data and the unsaturated area in the compensation area corresponding to each of the color channels, the gain coefficients corresponding to each of the color channels in the compensation area are determined respectively.

7. The method according to claim 6, characterized in that The step of determining the gain coefficients corresponding to the color channels of the compensation area according to the unsaturated area in the compensation area of ​​the original image data and the unsaturated area in the compensation area corresponding to the color channels, respectively, comprises: For any of the color channels in the compensation area, the following process is performed: Determine a pixel mean value of an unsaturated area of ​​the compensation area corresponding to the color channel as a fifth mean value; Determine a mean value of pixels corresponding to the color channel in an unsaturated area of ​​the compensation area of ​​the original image data as a sixth mean value; The ratio of the fifth average value to the sixth average value is used as a gain coefficient corresponding to the color channel of the compensation area.

8. The method according to claim 1, characterized in that The step of determining a dynamic range gain coefficient corresponding to a compensation area according to the original image data and the full-size image data of the target channel includes: Obtaining a bit width value corresponding to the original image data; Obtaining a reference value corresponding to the bit width value; Determine a mean value of pixels in the compensation area in the full-size image data corresponding to the infrared channel as a seventh mean value; A difference between the reference value and the seventh average value is determined, and a ratio of the reference value to the difference is used as a dynamic range gain coefficient corresponding to the compensation area.

9. The method according to claim 1, characterized in that: The step of compensating the full-size image data of each color channel of the compensation area after infrared separation according to the dynamic range gain coefficient corresponding to the compensation area comprises: For any of the color channels in the compensation area, the following process is performed: Each pixel value belonging to the compensation area in the full-size image data after infrared separation of the color channel is multiplied by the dynamic range gain coefficient corresponding to the compensation area to obtain the image data of the compensation area after the color channel is compensated.

10. The method according to claim 1, characterized in that The step of determining the full-size image data corresponding to the infrared channel and the full-size image data corresponding to each color channel after infrared separation according to the original image data comprises: According to the original image data, full-size image data corresponding to the infrared channel and full-size image data corresponding to each color channel are obtained; The full-size image data corresponding to the infrared channel and the separation coefficient are used to perform infrared separation on the full-size image data corresponding to each color channel to obtain the full-size image data after separation of each color channel.

11. The method according to claim 1, characterized in that: The method further comprises: Down-sampling is performed on the compensated full-size image data after red separation of each of the color channels to obtain image data in a Bayer format.

12. An image compensation device, characterized in that: include: An acquisition module is used to acquire the original image data collected by the RGB-IR image sensor; An infrared separation module, used to determine, based on the original image data, full-size image data corresponding to the infrared channel and full-size image data corresponding to each color channel after infrared separation; A coefficient determination module, used to determine the dynamic range gain coefficient corresponding to the compensation area according to the original image data and the full-size image data of the target channel; wherein the full-size image data of the target channel includes at least one of the following: the full-size image data of the infrared channel, and the full-size image data of each color channel after infrared separation; The compensation module is used to compensate the full-size image data of each color channel of the compensation area after infrared separation according to the dynamic range gain coefficient corresponding to the compensation area.

13. An electronic device, characterized in that: include: at least one processor; A memory having at least one program stored thereon, wherein when the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 11; At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

14. A computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images