Image processing method and device based on color learning

Abstract

The embodiment of the present application provides an image processing method and device based on color learning, the method comprising: a red, green, blue, infrared (RGBIR) module and a corresponding module of the RGBIR module shoot the same scene under a light source of the same wavelength to obtain a first original image and a second original image, wherein the corresponding module has an infrared cutoff characteristic; according to the first original image and the second original image, determining the interference value of the infrared light component on the red, green, and blue channels in the first original image; performing color learning according to the first original image and the second original image to reduce the interference value of the infrared light component on the red, green, and blue channels in the first original image. Through the present application, the problem of inaccurate images caused by the presence of infrared light components in the red, green, and blue channels of the images shot by the RGBIR module is solved, thereby achieving the effect of reducing the influence of the infrared light component on the red, green, and blue channels on the basis of retaining the infrared light in the images shot by the RGBIR module.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and specifically, to an image processing method and device based on color learning. Background Art

[0002] In the related art, there is a deviation between the red, green, and blue (RGB) pixel values ​​of the color restored image and the actual values. In severe cases, this will lead to a deviation in brightness after color restoration (correction), which will make it difficult to control exposure and even cause image brightness fluctuations. In addition, the deviations of the three channels R, G, and B are different, which makes it difficult to estimate the white balance color temperature of the RGBIR image.

[0003] Related technologies will result in different RGBIR correction matrices at different color temperatures (wavelengths), so the color temperature needs to be estimated first before the correction parameters can be used correctly. If the color temperature estimation is inaccurate and the wrong correction parameters are used, the final color image will be distorted. To estimate the color temperature on an RGBIR image without RGBIR color restoration / correction, it is necessary to develop an automatic white balance algorithm that is different from Bayer Raw, which makes product development more difficult. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and device based on color learning, so as to at least solve the problem in the related art that images taken by the RGBIR module are inaccurate due to the presence of infrared light components in the red, green and blue channels.

[0005] According to one embodiment of the present application, a color learning-based image processing method is provided, including: a red, green, blue, infrared (RGBIR) module and a corresponding module of the RGBIR module shoot the same scene under a light source of the same wavelength to obtain a first original image and a second original image, wherein the corresponding module has an infrared cutoff characteristic; according to the first original image and the second original image, determining an interference value of the infrared light component on the red, green, and blue channels in the first original image; and performing color learning based on the first original image and the second original image to reduce the interference value of the infrared light component on the red, green, and blue channels in the first original image.

[0006] According to another embodiment of the present application, there is provided an image processing device based on color learning, including: a shooting module, used for a red, green, blue, infrared (RGBIR) module and a corresponding module of the RGBIR module to shoot the same scene under a light source of the same wavelength to obtain a first original image and a second original image, wherein the corresponding module has an infrared cutoff characteristic; a determination module, used to determine the interference value of the infrared light component on the red, green, and blue channels in the first original image by comparing the first original image and the second original image; and a learning module, used to reduce the interference value of the infrared light component on the red, green, and blue channels in the first original image through color learning.

[0007] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0008] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] Through the present application, two original images are obtained by shooting the RGBIR module and the corresponding module of the RGBIR module under the same scene and the same wavelength of light source, wherein the first original image shot by the RGBIR module not only has infrared pixels in the infrared light channel, but also has infrared light components in the red, green and blue channels. The corresponding module has an infrared cutoff characteristic, and the infrared light is filtered when shooting the image. According to the first original image and the second original image, the interference value of the infrared light component on the red, green and blue channels in the first original image can be determined. In order to reduce the interference of the infrared light component on the red, green and blue channels in the first original image, color learning is performed based on the first original image and the second original image. Therefore, the problem of inaccurate images caused by the presence of infrared light components in the red, green and blue channels of the images shot by the RGBIR module can be solved, thereby achieving the effect of reducing the influence of the infrared light component on the red, green and blue channels on the basis of retaining the infrared light in the image shot by the RGBIR module. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a hardware structure block diagram of a computer terminal according to an image processing method based on color learning according to an embodiment of the present application;

[0011] Figure 2 is a flowchart of an image processing method based on color learning according to an embodiment of the present application;

[0012] Figure 3is a basic unit diagram of an RGBIR image sensor and a Bayer image sensor according to an embodiment of the present application;

[0013] Figure 4 is a schematic diagram of the RGBIR module and the corresponding module working simultaneously according to an embodiment of the present application;

[0014] Figure 5 is a schematic diagram of an RGBIR module and a corresponding module not working at the same time according to an embodiment of the present application;

[0015] Figure 6 is a structural block diagram of an image processing device based on color learning according to an embodiment of the present application;

[0016] Figure 7 is a system schematic diagram when the RGBIR module and the corresponding module according to an embodiment of the present application work simultaneously;

[0017] Figure 8 is a system schematic diagram when the RGBIR module and the corresponding module according to an embodiment of the present application do not work at the same time;

[0018] Fig. 9 is a flowchart of an image processing method based on color learning according to yet another embodiment of the present application;

[0019] Fig.10 It is a statistical diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0022] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 : is a hardware structure block diagram of a computer terminal of an image processing method based on color learning in an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0023] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image processing method based on color learning in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0025] In this embodiment, a color learning-based image processing method running on the above-mentioned computer terminal is provided. Figure 2 is a flowchart of an image processing method based on color learning according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0026] Step S202, a red, green, blue, infrared (RGBIR) module and a corresponding module of the RGBIR module shoot the same scene under a light source of the same wavelength to obtain a first original image and a second original image, wherein the corresponding module has an infrared cutoff characteristic;

[0027] In step S202 of this embodiment, the corresponding module includes one of the following: an image sensor equipped with an infrared cut-off filter, wherein the image sensor includes red, green and blue pixels; an RGBIR module equipped with an infrared cut-off filter.

[0028] In step S202 of this embodiment, the RGBIR module captures a first original image, and the corresponding module captures a second original image.

[0029] In an exemplary embodiment of the present application, the image sensor including red, green and blue pixels equipped with an infrared cutoff filter can be a Bayer image sensor equipped with an infrared filter, or a QuadBayer image sensor equipped with an infrared filter, etc.

[0030] Figure 3 is a basic unit diagram of an RGBIR image sensor and a Bayer image sensor according to an embodiment of the present application. A Bayer image sensor is one of the most common image sensors, such as Figure 3 As shown in (B), it is characterized by using 2x2 pixels as the basic unit. Each 2x2 pixel unit contains 2 G pixels in the diagonal position, 1 R pixel and 1 B pixel. The G pixels in the same row as the R pixels are called Gr pixels; the G pixels in the same row as the B pixels are called Gb pixels.

[0031] like Figure 3 As shown in (A), the RGBIR image sensor contains pixels that only sense infrared light, namely IR pixels. RGBIR image sensors can be of the type with 2x2 as the basic unit or of the type with 4x4 as the unit. Usually, RGBIR image sensors are equipped with filters that pass visible light and infrared light, and their RGB channels also contain infrared light components. If these infrared components are not processed, the color image after the RGBIR image sensor has processed the image digital signal will have color problems, such as color cast. In order to solve this problem, color restoration of the RGBIR image sensor is required.

[0032] In an exemplary embodiment of the present application, the RGBIR module and the corresponding module may work simultaneously or may not work simultaneously. Figure 4 is a schematic diagram of the RGBIR module and the corresponding module working simultaneously according to an embodiment of the present application, such as Figure 4As shown, 401 is an RGBIR module that needs to perform RGBIR color restoration, 402 is a lens, 403 is a filter that passes visible light and infrared light, and 404 is an RGBIR Sensor. 405 is a corresponding module, which can be an RGBIR module equipped with an infrared cutoff filter, or a Bayer Sensor module. 406 is a lens in module 405, 407 is an infrared cutoff filter, 408 can be an RGBIR Sensor, or a Bayer Sensor. 409 is an image digital signal processor. The RGBIR module 401 to be color restored outputs an RGBIR Raw image. Depending on the type of the corresponding sensor of module 405, the output of module 405 can be RGBIR Raw or Bayer Raw.

[0033] Figure 5 is a schematic diagram of the RGBIR module and the corresponding module not working at the same time according to an embodiment of the present application, such as Figure 5 As shown, 501 is an RGBIR module that needs to perform RGBIR color restoration. 502 is a lens; 503 is a filter that passes visible light and infrared light; 504 is an RGBIR Sensor; 505 is an ISP processor that performs digital signal processing on the RGBIR Raw image output by the 501 RGBIR module. 506 is a corresponding module, which can be an RGBIR module equipped with an infrared cutoff filter or a Bayer Sensor module. 507 is the lens of the 506 module; 508 is an infrared cutoff filter; 509 can be an RGBIRSensor or a Bayer Sensor; 510 is an ISP processor that performs digital signal processing on the Raw image output by the 506 module.

[0034] The difference between the RGBIR module and the corresponding module working simultaneously and not working simultaneously is:

[0035] When two modules work at the same time, it can be considered that they are shooting the same scene at the same time, so the color temperature of the light source in the scene is consistent and the shooting object is the same. The two modules do not need to work at the same time, they can be shot at different times, and the color temperature of the light source in the scene is the same and the shooting object is the same;

[0036] When the two modules work simultaneously, it is necessary to use computer vision methods to determine the same number of image blocks from the Raw images of module 401 and module 405 respectively. The two modules are not required to work simultaneously, and computer vision methods or manual labeling methods can be used to determine the same number of image blocks from the Raw images of module 501 and module 506 respectively.

[0037] Step S204, determining interference values ​​of infrared light components on red, green and blue channels in the first original image according to the first original image and the second original image;

[0038] In step S204 of this embodiment, it includes: performing color restoration on the first original image captured by the RGBIR module to obtain first white balance statistical information, and determining a loss function based on a weighted distance sum between the first white balance statistical information and second white balance statistical information of the second original image captured by the corresponding module, wherein the white balance statistical information is a ratio of color statistical information of different channels in the image.

[0039] It should be noted that, in an exemplary embodiment, the loss value of the loss function is the interference value of the infrared light component on the red, green and blue channels in the first original image.

[0040] It should be noted that the color statistical information is data obtained by statistically analyzing the color channels of the first original image or the second original image. For example, a statistical method that can be used is to divide the original image into small areas, calculate the average value of the same color pixels in each small area, and thus obtain the statistical value of the color channels.

[0041] White balance statistics are usually used as statistics for white balance algorithms, which can be the ratio of color statistics of different channels.

[0042] In an exemplary embodiment, before performing color restoration on the first original image captured by the RGBIR module to obtain first white balance statistical information, the method further includes: determining statistical information of each sub-block of the first original image and statistical information of each sub-block of the second original image; determining an image block from the first original image and the second original image, wherein the image block in the first original image is I i , i=1,...,N, the image block in the second original image is J i ,i=1,...,N, where the I i and the J i The corresponding image blocks have the same color and have only one color, and the value of N is not less than the number of elements in the RGBIR color restoration parameter; and the first original image is calculated according to the statistical information of each sub-block of the first original image. i The original color statistical information of the first statistical partition covered by J is calculated according to the statistical information of each sub-block of the second original image. i The second color statistical information of the second statistical partition covered; wherein the number of the first statistical partition and the second statistical partition is greater than a preset threshold, the first statistical partition includes at least one sub-block of the first original image, and the second statistical partition includes at least one sub-block of the second original image.

[0043] It should be noted that for I i and J i For regions with the same color, the processing method may be to average their color statistics to obtain the color statistics corresponding to the color; or only one region may be selected and other regions with the same color may be discarded.

[0044] In an exemplary embodiment, determining statistical information of each sub-block of a first original image and statistical information of each sub-block of a second original image includes: evenly dividing the first original image into a plurality of sub-blocks, calculating respective average values ​​of red, green, blue, and infrared channels in each sub-block of the first original image, determining statistical information of each sub-block of the first original image, evenly dividing the second original image into a plurality of sub-blocks, calculating respective average values ​​of red, green, and blue channels in each sub-block of the second original image, and determining statistical information of each sub-block of the second original image.

[0045] In an exemplary embodiment, the number of sub-blocks in the first original image that are blocked by I is calculated according to the statistical information of each sub-block of the first original image. i The original color statistical information of the first statistical partition covered by J is calculated according to the statistical information of each sub-block of the second original image. i The second color statistical information of the second statistical partition covered by I i The statistical information of the sub-blocks of the first statistical partition covered is averaged to determine the I i The original color statistics of the corresponding red, green, blue and infrared channels; i The statistical information of the sub-blocks of the second statistical partition covered is averaged to determine the J i The second color statistics for the corresponding red, green, and blue channels.

[0046] In an exemplary embodiment, determining the image block from the first original image and the second original image includes: determining the image block from the first original image and the second original image by computer vision or manual annotation.

[0047] Step S206: performing color learning according to the first original image and the second original image to reduce the interference value of the infrared light component on the red, green and blue channels in the first original image.

[0048] In step S206 of this embodiment, it includes: calculating the loss value of the loss function according to the initial value of the RGBIR color restoration parameter, and updating the RGBIR color restoration parameter through periodic nonlinear optimization until the iteration stop condition is met or the loss value is lower than a preset threshold.

[0049] In an exemplary embodiment, a loss value of a loss function is calculated based on an initial value of an RGBIR color restoration parameter, and the RGBIR color restoration parameter is updated through periodic nonlinear optimization until an iteration stop condition is met or the loss value is lower than a preset threshold, including: calculating a loss value of a loss function based on an initial value of the RGBIR color restoration parameter, and determining whether the iteration stop condition is met or the loss value is lower than a preset threshold; when the iteration stop condition is not met and the loss value is lower than the preset threshold, determining the updated RGBIR color restoration parameter using a machine learning algorithm according to a preset period, wherein the machine learning algorithm includes a gradient descent algorithm.

[0050] In an exemplary embodiment, performing color restoration on a first original image captured by the RGBIR module to obtain first white balance statistical information includes: determining the first white balance statistical information according to a vector M and the original color statistical information, wherein the vector M is an initial value of an RGBIR color restoration parameter.

[0051] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0052] In this embodiment, an image processing device based on color learning according to an embodiment of the present application is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0053] Figure 6 is a structural block diagram of an image processing device based on color learning according to an embodiment of the present application, such as Figure 6 As shown, the device includes: a shooting module 10, a determination module 20, and a learning module 30.

[0054] A shooting module 10, used for a red, green, blue, infrared (RGBIR) module and a corresponding module of the RGBIR module to shoot the same scene under a light source of the same wavelength to obtain a first original image and a second original image, wherein the corresponding module has an infrared cutoff characteristic;

[0055] A determination module 20, configured to determine interference values ​​of infrared light components on red, green and blue channels in the first original image according to the first original image and the second original image;

[0056] The learning module 30 is used to perform color learning according to the first original image and the second original image to reduce the interference value of the infrared light component on the red, green and blue channels in the first original image.

[0057] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0058] To facilitate the understanding of the technical solution provided by the present application, a detailed explanation will be given below in combination with embodiments of specific scenarios.

[0059] In an exemplary embodiment, in order to solve the problem of RGBIR color correction, the related technology in the field generally adopts a standard light source environment in a laboratory, and the RGBIR image sensors are respectively equipped with infrared cut-off filters (such as a standard 650nm filter) to shoot a standard 24-color card through visible light and infrared light filters. The specific steps are:

[0060] The average value of the four-channel pixels of each color block in the color card image taken with an infrared cutoff filter is denoted as M o , where M represents a channel of RGBIR.

[0061] Under the same lighting conditions, a color card is photographed with a filter that can pass visible light and infrared light (such as a double-peak filter). Calculate the pixel average of the four channels of each color block of the color card, denoted as M c , where M represents a channel of RGBIR.

[0062] Use the least squares method to calculate the correction matrix coefficient a in formula (1) under the current lighting conditions xx .

[0063]

[0064] Repeat the above steps under different lighting conditions to obtain several sets of correction matrices.

[0065] In a similar embodiment, it can be the following formula (2).

[0066]

[0067] Formula (1) represents a mapping from RGBIR pixel values ​​under an infrared cutoff filter to RGBIR pixel values ​​under a visible light and infrared filter. Formula (2) represents a mapping from RGBIR pixel values ​​under a visible light and infrared filter to RGBIR pixel values ​​under an infrared cutoff filter. It is easy to find that the matrix [a xx ] is full rank and reversible. Therefore, formula (1) and formula (2) are two equivalent relation calculations of a mapping relation.

[0068] This scheme has strict requirements on experimental equipment and operation. When equipped with filters that pass visible light and infrared light, the RGB channel can be approximated as the superposition of the pure visible light component M and the infrared component IR, as shown in formula (3).

[0069] M c =M+IR (3)

[0070] When equipped with an infrared cut-off filter, the RGB channels can be modeled using equation (4).

[0071] M o =α·M+β·IR (4)

[0072] α is equal to 1 only when the infrared cutoff filter has ideal cutoff characteristics, the lighting conditions are exactly the same during the two experiments, and the relative position and angle between the RGBIR module and the color card have not changed. The light intensity reflected from the color card to the RGBIR module is different due to different relative positions and angles. During the laboratory calibration process, it is impossible to guarantee that the filter has ideal cutoff characteristics, nor is it possible to guarantee that the relative position and angle between the RGBIR module and the color card equipped with different filters are exactly the same. Therefore, in general, α≠1 (5)

[0073] Since there are almost no infrared cut-off filters with ideal cut-off characteristics, o There must be infrared residue in the equation, that is, β≠0, β<1 (6)

[0074] Under different light source color temperatures (wavelengths), the wavelength cutoff rate of the infrared cutoff filter is different. Therefore, under different color temperature (wavelength) light sources, the value of 3 is different, that is, the intensity of infrared residual is different.

[0075] Figure 7 is a schematic diagram of a system when the RGBIR module and the corresponding module according to an embodiment of the present application work simultaneously, such as Figure 7 As shown, 701 is an RGBIR image input unit corresponding to the RGBIR module, 702 is an original image input unit corresponding to the corresponding module, 703 is an RGBIR statistical unit, 705 is a statistical unit, 704 is an ISP Pipeline (image digital signal processing link), 708 is a color image output corresponding to the 701 input unit (the output of the RGBIR module), 709 is a color image output corresponding to the 702 input unit (the output of the corresponding module), 706 is an image segmentation and registration analysis unit, which has two inputs, namely 708 and 709, 706 outputs the segmentation of the 708 color image, the segmentation and registration information of the 709 image, 706 can use the traditional computer vision image segmentation and registration algorithm commonly used in the field of image processing, and can also use the commonly used deep learning image segmentation and registration algorithm, 707 is an RGBIR color restoration parameter optimization unit, and the input of 707 includes:

[0076] (1) Segmentation of the color image 708, and segmentation and registration information of the image 709. The segmentation and registration results of the image segmentation and registration unit 708 are the I i , i=1, ..., N, the segmentation and registration result of 709 is J in the above embodiment i , i=1,...,N;

[0077] (2) Results of 703RGBIR statistical units, The RGBIR Raw image of the 701 original RGBIR image input unit is evenly divided into M1 sub-blocks, is the average value of R, G, B and IR channels in the i-th sub-block;

[0078] (3) Results of 704 statistical units, The Raw image of the image input unit corresponding to 702 is evenly divided into M2 sub-blocks, is the average value of R, G, and B channels in the i-th sub-block;

[0079] The output of the RGBIR color restoration optimization unit is the optimized and smoothed RGBIR color restoration parameters.

[0080] Figure 8 is a system schematic diagram when the RGBIR module and the corresponding module according to the embodiment of the present application do not work at the same time, such as Figure 8As shown, 801 is the original Raw image input unit, 802 is the ISP Pipeline (digital image information processing link), 803 is the color image output after the ISP Pipeline processing, and 804 is the dump (output) path of the Raw image output by 801. Through the output path 804, the original Raw image can be saved to a memory card or even a PC. According to the type of the current camera module, the original Raw image input unit 801 can correspond to the RGBIR module with color restoration and the module corresponding to the color restoration processing.

[0081] Fig. 9 is a flowchart of an image processing method based on color learning according to another embodiment of the present application. Fig. 9 As shown, the following steps are included:

[0082] Step S901, calculating statistical information of monochrome areas of the RGBIR Raw image and the corresponding module Raw image;

[0083] Specifically, calculate I i , i = 1, ..., N and J i , i = 1, ..., N statistical information, optional, such as Fig.10 The statistical information diagram shown, 1001 may be, i or J i , 1002 can be Figure 7 The output of the statistical unit 703 or the statistical unit 705 in.

[0084] Step S902, determining whether the prerequisite for color learning is met;

[0085] Specifically, if the judgment is yes, the process proceeds to step S903, and if the judgment is no, the process ends;

[0086] Specifically, if there is a i or J i Complete coverage of statistical partitions, average calculation of statistical results of statistical partitions to obtain monochrome area I i (or J i ) corresponding statistics (or ).

[0087] If there is no completely covered statistical partition, I i (or J i ) has no corresponding statistical information, N will decrease. i (or J i ) have the same hue, only one I can be retained i (or J i ) corresponding statistics.

[0088] If the size of N exceeds a preset threshold, such as 12, the prerequisite for color learning is met. If the prerequisite for color learning is not met, the RGBIR color restoration parameters obtained by previous color learning are used.

[0089] Step S903, determining the initial value of the color restoration parameter;

[0090] Specifically, before the following RGBIR color restoration / correction parameters are completed for the first time, the given initial RGBIR color restoration / correction parameters are used.

[0091] Step S904, calculating the loss value (objective function value);

[0092] Specifically, the loss function is determined, and the weighted distance sum between the color statistical information ratio of the RGBIR module after RGBIR color restoration / correction and the statistical information ratio of the same type of the corresponding module in the color statistical information ratio coordinate system is used as the loss function. Given the initial value of RGBIR color restoration / correction, any nonlinear optimization method, such as gradient descent optimization, is used to obtain the parameter value of RGBIR color restoration / correction.

[0093] The loss function in this application can be a weighted sum of Euclidean distances, a weighted sum of squared Euclidean distances, or a weighted sum of Manhattan distances.

[0094] The color statistics ratio in this application is as follows Where a1, b1, b2, c2 are constants greater than 0, a2 and c1 are constants greater than or equal to 0, and They are the statistics of the RGB channels respectively.

[0095] Here is an embodiment, which uses formula (7) to calculate the loss:

[0096]

[0097] Among them, ω i is the weight, ω i ≥0;

[0098] and It is the statistical information after RGBIR color transformation with parameter M;

[0099] and It is the statistical information or average statistical information on the Raw graph of the corresponding module;

[0100] f is a color transformation with the vector M as a parameter. f can be the transformation shown in formula (8) or the transformation shown in formula (9).

[0101]

[0102]

[0103] In formula (8), M = [m0, ..., m i1 ] T In formula (9), M = [m0, ..., m5] T .

[0104] In this embodiment, formula (7) is further specified to obtain:

[0105]

[0106] Step S905, determining whether the iteration stop condition has been met or the loss value is lower than a preset threshold value;

[0107] Specifically, the iteration stopping condition may be the number of iterations, or other common gradient descent optimization calculation iteration stopping conditions.

[0108] If the judgment result is yes, proceed to step S906, if the judgment result is no, end the process;

[0109] Step S906, calculating the gradient value of the optimization target with respect to the parameter;

[0110] Step S907: Use any gradient descent algorithm to calculate and obtain updated parameters.

[0111] The above RGBIR color restoration / correction parameter calculation process is performed periodically, and the RGBIR color restoration / correction parameters are updated by using the parameter smoothing update method. Assume that the initial color restoration / correction parameter optimized this time is M init , the parameter update rate is γ, 0≤γ≤1, and the updated color parameter M is obtained according to formula (12) cur ,

[0112] M cur =(1-γ)·M init +γ·M opt (12)

[0113] Updated parameter M cur It will be the initial parameter M for the next periodic calculation init The parameter update rate can be determined according to the number of valid corresponding regions N in this calculation process. Generally, the larger N is, the larger γ is; the smaller N is, the smaller γ is.

[0114] For systems where the RGBIR module and the corresponding module do not work at the same time, the vector M can be calculated according to the above method on a PC or other computing platform. opt , then vector M opt As a parameter set to Figure 5 For systems where the RGBIR module and the corresponding module do not work at the same time, manual setting of area I can be adopted i , J i , then in region I i Calculate the average value of RGBIR channels In area J i Calculate the average value of the internal RGB channels

[0115] Through the embodiments of the present application, after the RGBIR color restoration parameters are obtained, the parameters are used to remove the IR components in the RGB channel under different ambient color temperatures, and good effects are achieved. This shows that the parameters are suitable for different color temperatures, and when the white point statistical information after RGBIR color restoration and the white point statistical information of the corresponding module overlap, the RGBIR module equipped with a filter that passes visible light and infrared light has the same white balance statistical information and white balance calibration data as the corresponding module equipped with an infrared cutoff filter after color restoration, so the automatic white balance algorithm of RGBIR can be simplified.

[0116] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0117] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0118] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0119] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0120] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0121] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0122] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method based on color learning, characterized in that: include: A red, green, blue, infrared (RGBIR) module and a corresponding module of the RGBIR module shoot the same scene under a light source of the same wavelength to obtain a first original image and a second original image, wherein the corresponding module has an infrared cutoff characteristic; Determine, according to the first original image and the second original image, interference values ​​of the infrared light component on the red, green and blue channels in the first original image; Color learning is performed according to the first original image and the second original image to reduce the interference value of the infrared light component on the red, green and blue channels in the first original image.

2. The method according to claim 1, characterized in that The corresponding module includes one of the following: An image sensor equipped with an infrared cut-off filter, wherein the image sensor comprises RGB pixels; RGBIR module equipped with infrared cut-off filter.

3. The method according to claim 1, characterized in that: The step of determining the interference value of the infrared light component on the red, green and blue channels in the first original image according to the first original image and the second original image includes: Perform color restoration on the first original image captured by the RGBIR module to obtain first white balance statistical information, and determine a loss function based on a weighted distance sum between the first white balance statistical information and second white balance statistical information of the second original image captured by the corresponding module, wherein the white balance statistical information is a ratio of color statistical information of different channels in the image.

4. The method according to claim 3, characterized in that The performing color learning according to the first original image and the second original image to reduce the interference value of the infrared light component on the red, green and blue channels in the first original image includes: The loss value of the loss function is calculated according to the initial value of the RGBIR color restoration parameter, and the RGBIR color restoration parameter is updated through periodic nonlinear optimization until an iteration stop condition is met or the loss value is lower than a preset threshold.

5. The method according to claim 3, characterized in that: Before performing color restoration on the first original image captured by the RGBIR module to obtain first white balance statistical information, the method further includes: Determine statistical information of each sub-block of the first original image and statistical information of each sub-block of the second original image; An image block is determined from the first original image and the second original image, wherein the image block in the first original image is I i , i=1,...,N, the image block in the second original image is J i ,i=1,...,N, where the I i and the J i The corresponding image blocks have the same color and only one color, and the value of N is not less than the number of elements in the RGBIR color restoration parameter; Calculate the number of sub-blocks in the first original image that are I according to the statistical information of each sub-block of the first original image. i The original color statistical information of the first statistical partition covered by J is calculated according to the statistical information of each sub-block of the second original image. i Second color statistical information of the covered second statistical partition; The number of the first statistical partition and the second statistical partition is greater than a preset threshold, the first statistical partition includes at least one sub-block of the first original image, and the second statistical partition includes at least one sub-block of the second original image.

6. The method according to claim 4, characterized in that Calculating a loss value of a loss function according to an initial value of an RGBIR color restoration parameter, and updating the RGBIR color restoration parameter through periodic nonlinear optimization until an iteration stop condition is met or the loss value is lower than a preset threshold, including: Calculate the loss value of the loss function according to the initial value of the RGBIR color restoration parameter, and determine whether the iteration stop condition is met or the loss value is lower than a preset threshold; When the iteration stop condition is not met and the loss value is lower than a preset threshold, a machine learning algorithm is used to determine the updated RGBIR color restoration parameters according to a preset period, wherein the machine learning algorithm includes a gradient descent algorithm.

7. The method according to claim 4, characterized in that Performing color restoration on the first original image captured by the RGBIR module to obtain first white balance statistical information includes: The first white balance statistical information is determined according to the vector M and the original color statistical information, wherein the vector M is an initial value of the RGBIR color restoration parameter.

8. The method according to claim 5, characterized in that Determining statistical information of each sub-block of the first original image and statistical information of each sub-block of the second original image includes: The first original image is evenly divided into a plurality of sub-blocks, the average values ​​of the red, green, blue and infrared channels in each sub-block of the first original image are calculated, and the statistical information of each sub-block of the first original image is determined; the second original image is evenly divided into a plurality of sub-blocks, the average values ​​of the red, green and blue channels in each sub-block of the second original image are calculated, and the statistical information of each sub-block of the second original image is determined.

9. The method according to claim 5, characterized in that Calculate the number of sub-blocks in the first original image that are I according to the statistical information of each sub-block of the first original image. i The original color statistical information of the first statistical partition covered by J is calculated according to the statistical information of each sub-block of the second original image. i The second color statistics of the second statistical partition covered include: I i The statistical information of the sub-blocks of the first statistical partition covered is averaged to determine the I i The raw color statistics of the corresponding red, green, blue, and infrared channels; For the said i The statistical information of the sub-blocks of the second statistical partition covered is averaged to determine the J i The second color statistics for the corresponding red, green, and blue channels.

10. The method according to claim 5, characterized in that Determining an image block from the first original image and the second original image includes: The image block is determined from the first original image and the second original image by computer vision or manual annotation.

11. An image processing device based on color learning, characterized in that: include: A shooting module, used for a red, green, blue, infrared (RGBIR) module and a corresponding module of the RGBIR module to shoot the same scene under a light source of the same wavelength to obtain a first original image and a second original image, wherein the corresponding module has an infrared cutoff characteristic; A determination module, configured to determine, based on the first original image and the second original image, an interference value of the infrared light component on the red, green and blue channels in the first original image; A learning module is used to perform color learning according to the first original image and the second original image to reduce the interference value of the infrared light component on the red, green and blue channels in the first original image.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 10 when executed by a processor.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 10 are implemented.

Images