A self-supervised color correction method for AWB enhancement in multi-device domains

Abstract

In view of the data shortage problem faced by existing color constancy technologies, the present invention provides a self-supervised color correction method for multi-device domain AWB enhancement, which generates illumination device-independent images through illumination invariant description theory; generates self-supervised labels for corresponding learning of illumination device-independent images using the Von Kries model and illumination invariant description theory; and uses a deep convolutional neural network to learn the mapping of illumination device-independent images to corresponding labels. This method can achieve the purpose of enhancing AWB through joint training of multi-device domain data, thereby effectively improving the generalization and accuracy of AWB technology and reducing the consumption of manpower and material resources.

Description

Technical Field

[0001] The present invention relates to the technical field of color constancy, and in particular to a color cast image correction method based on illumination invariant description. Background Art

[0002] Research in visual cognition and psychology shows that the human system has a "color constancy" function, that is, the human visual system can automatically adjust the color of scene objects in the human eye according to the scene seen, so as to eliminate the influence of the color of the light source in the scene on the color of the object, so that the color of the object can maintain a certain "constancy" under different light sources. However, it is difficult for computer systems to judge what is the appropriate color, and it is difficult to automatically correct the color cast of objects in a huge lighting space. Therefore, color constancy technology hopes to automatically restore a color-free image under white light from a color-cast picture to correct the image information. In practical applications, color constancy technology is often carried in the AWB module in the image signal processing flow for AWB enhancement.

[0003] The existing color constancy methods are mainly divided into four types: based on low-order statistical features of images, based on illumination mapping, based on fusion, and based on learning. The method based on low-order statistical features of images predicts illumination by proposing hypotheses based on statistical image color or texture information. It has low complexity and consumes little memory, but its applicable scene range is small and it only has a good correction effect on some scenes. The method based on illumination mapping counts the colors of each picture under different illuminations, and all colors form a color gamut. This method restores the image by estimating the possibility that the input image falls into each color gamut. This method has good results, but it has high complexity and is difficult to implement. The method based on fusion statistically analyzes the scene image features that are most suitable for each algorithm, thereby achieving the purpose of fusing multiple algorithms. The method based on learning directly regresses the illumination or performs Bayesian estimation, which has good results, but the annotation cost is very high, the number of data sets is too small, and it is difficult to generalize. In general, the learning-based method has the best effect and has received widespread attention.

[0004] However, learning-based methods cannot directly use data from different camera domains because different cameras have different sensors. Studies have shown that data from different sensors are difficult to improve the performance of a single model, which further limits the practical application of learning-based models. For a mounted camera, a large amount of data needs to be collected to train the model. When the same model is mounted on other cameras, the same data still needs to be collected to support model training. When data is already expensive, how to adapt a single model to multiple camera domains is crucial. Summary of the invention

[0005] In view of the problems existing in the above-mentioned prior art, the present invention provides a self-supervised color correction method for multi-device domain AWB enhancement, which generates an illumination device-independent image through the illumination invariant description theory; generates self-supervised labels for corresponding learning of illumination device-independent images using the Von Kries model and the illumination invariant description theory; and uses a deep convolutional neural network to learn the mapping of illumination device-independent images to white-balanced images. This method can achieve the purpose of joint training of multi-device domain data to enhance AWB, thereby effectively improving the generalization and accuracy of AWB technology and reducing the consumption of manpower and material resources.

[0006] The technical solution of the present invention is as follows:

[0007] A self-supervised color correction method for multi-device domain AWB enhancement, comprising the following steps:

[0008] S1 input image;

[0009] S2 calculates the illumination invariant image;

[0010] S3 obtains the illumination label GroundTruth corresponding to the input image;

[0011] S4 maps the illumination label GroundTruth to the correct form corresponding to the illumination invariant image;

[0012] S5 inputs the illumination invariant image into the neural network model and predicts the corresponding label;

[0013] S6 calculates the error between the label obtained in step S4 and the label obtained in step S5;

[0014] S7 repeats steps S1-S6, uses the error to calculate the loss, and completes the training of the neural network model;

[0015] S8 calculates an illumination invariant image from the image to be color corrected, inputs the illumination invariant image into a trained neural network model to obtain its corresponding label, and corrects the illumination invariant image through a VonKries model to obtain a white balanced image.

[0016] Preferably, the input image in step S1 is an original RAW image taken by a mobile phone or a camera, which is expressed as:

[0017] RAW rgb =LEC

[0018] Among them, L represents illumination, E represents the reflectivity of objects in the scene, and C represents the sensitivity coefficient of the camera lens.

[0019] Preferably, the illumination invariant image in step S2 is obtained by the following method:

[0020] Based on the Von Kries diagonal model, the illumination invariance description is introduced to obtain an image with illumination invariance:

[0021]

[0022] st H(LEC)=LC·H(E)

[0023] Where H(·) is a linear function.

[0024] Preferably, the step S3 specifically obtains the illumination tag GroundTruth in the following manner:

[0025] Place an 18% gray face or a color plate with gray faces in the input image to obtain the illumination label ill rgb =LC.

[0026] Preferably, in step S4, the illumination label GroundTruth is mapped to a label corresponding to the illumination invariant image:

[0027]

[0028] st H(LEC)=LC·H(E)

[0029] Where H(·) is a linear function.

[0030] Preferably, the neural network model in step S5 fits the mapping function between the illumination invariant image and the label:

[0031]

[0032] Preferably, the neural network in step S5 is a deep convolutional neural network.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The self-supervised color correction method for multi-device domain AWB enhancement of the present invention can achieve the effect of enhancing AWB through joint training of multi-device domain data by overcoming the difference in illumination labels between different camera devices caused by the camera sensitivity coefficient, thereby achieving the purpose of improving the generalization and accuracy of AWB technology; in addition, the method can be further expanded to the sRGB data domain (which can be regarded as an image taken by a certain 8-bit camera), so that the AWB technology can obtain a wide range of scene prior information from the massive sRGB images on the Internet, which is beneficial to further reduce the consumption of manpower and material resources when implementing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Some specific embodiments of the present invention are described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. The objects and features of the present invention will become more apparent in view of the following description in conjunction with the accompanying drawings, in which:

[0036] Figure 1 is a flow chart of obtaining images and labels for illumination-invariant description according to an embodiment of the present invention;

[0037] Figure 2 is an input picture of an embodiment of the present invention;

[0038] Figure 3 is an illumination invariant image according to an embodiment of the present invention;

[0039] Figure 4 is a white balance picture according to an embodiment of the present invention;

[0040] Figure 5 4 is a flow chart of establishing a self-supervisory model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings, but are not intended to limit the protection scope of the present invention.

[0042] In the fields of image restoration and photography, due to the influence of light, photos often have color cast. The general method is for professionals to use their naked eyes to adjust the image color to a state that makes people feel comfortable, but this method is unreliable and difficult to promote. The purpose of color constancy technology is to automatically remove the influence of light and correct the color cast image to the white light without color cast. However, in the existing technology, this method has always faced the problem of data scarcity.

[0043] In view of the above problems, the embodiment of the present invention firstly proposes an image generation method based on illumination invariant description. Figure 1 As shown, it mainly includes the following steps:

[0044] S1 reads the input image I in the form of RGB three channels = (R in ,G in ,B in ), where (R in =L R E R C R , G in =L G E G C G , B in =L B E B CB ),like Figure 2 shown.

[0045] S2 uses the following formula to process the input image I so that the image acquires illumination invariance, such as Figure 3 As shown:

[0046]

[0047] st H(kX)=k·H(X)

[0048] Among them, X is a pixel value on the image, and k is a constant.

[0049] S3 gets the light tag GroundTruth:

[0050] Usually, an 18% gray surface or a color plate with a gray surface is placed in the input image I, and the illumination label illus = (R ill ,G ill ,B ill ), where (R ill =L R C R , G ill =L G C G , B ill =L B C B ).

[0051] S4 maps the illumination label GroundTruth to the correct form corresponding to the illumination invariant image illus inv =(A R ,A G ,A B ): Specifically, the same operation as in step S2 is adopted to map the illumination label in step S3 to the illumination invariant label corresponding to the illumination invariant image.

[0052]

[0053] After obtaining the illumination-invariant images and corresponding labels, a self-supervised model is established such as Figure 5 shown, including:

[0054] S5 inputs the illumination-invariant image in S2 into the neural network model and predicts the label in S4;

[0055]

[0056] S6 calculates the angle error between the label obtained in step S4 and the label predicted in step S5;

[0057] S7 repeats steps S1-S6, uses the angle error to calculate the loss, and completes the training of the neural network model.

[0058] S8 calculates the illumination invariant image from the image to be color corrected, inputs the illumination invariant image into the trained neural network model, obtains its corresponding label, and corrects the illumination invariant image through the VonKries model, thereby obtaining a white balance image, such as Figure 4 shown.

[0059]

[0060] The technical solutions provided by the embodiments of the present invention are described in detail above. The principles and implementation methods of the embodiments of the present invention are described by using specific examples in the present invention. The description of the above embodiments is only applicable to help understand the principles of the embodiments of the present invention. At the same time, those skilled in the art may make changes in the specific implementation methods and application scopes according to the embodiments of the present invention. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A self-supervised color correction method for multi-device domain AWB enhancement, characterized in that The following steps are involved: S1 input image; S2 calculates the illumination invariant image; S3 obtains the illumination label corresponding to the input image; S4 maps the illumination label to the correct form corresponding to the illumination invariant image; S5 inputs the illumination invariant image into the neural network model and predicts the corresponding label; S6: Calculate the error between the label obtained in step S4 and the label obtained in step S5; S7 repeats steps S1-S6, uses the error to calculate the loss, and completes the training of the neural network model; S8 calculates an illumination invariant image from the image to be color corrected, inputs the illumination invariant image into a trained neural network model to obtain a corresponding label, and corrects the illumination invariant image through a Von Kries model, thereby obtaining a white balance image; The input image in step S1 is an original RAW image taken by a mobile phone or camera, which is expressed as: Among them, L represents illumination, E represents the reflectivity of objects in the scene, and C represents the sensitivity coefficient of the camera lens; The illumination invariant image in step S2 is obtained by the following method: Based on the Von Kries diagonal model, the illumination invariance description is introduced to obtain an image with illumination invariance: Where H(·) is a linear function; The step S3 specifically obtains the light label in the following manner: Place an 18% gray face or a color plate with gray faces in the input image, and obtain the lighting label through the color plate ; In step S4, the illumination label is mapped to a label corresponding to the illumination invariant image: Where H(·) is a linear function.

2. The self-supervised color correction method for multi-device domain AWB enhancement according to claim 1, characterized in that The neural network model in step S5 fits the mapping function between the illumination invariant image and the label: 。 3. The self-supervised color correction method for multi-device domain AWB enhancement according to claim 1, characterized in that The neural network in step S5 is a deep convolutional neural network.

Images