Multi-light-source color constancy method and system based on color decoupling

Abstract

The invention discloses a multi-light-source color constancy method and system based on color decoupling, multi-light-source color constancy is realized by using a pixel-level multi-light-source color constancy model, and the system comprises a content color perception module CCAM and a contrast coordination module CHAM. CCAM learns scene content color feature priori, scene content colors and illumination colors are separated by providing color features of the content for the model, and light source prediction outside the color gamut is reduced. CHAM uses spatial correlation to make the model more sensitive to the relationship between adjacent features, and uses illumination parallax to guide feature classification; the local edge contrast is improved by enhancing the uniqueness of homogeneous illumination features and the distinctiveness of heterogeneous illumination features; the soft discrimination of similar illumination features is emphasized by distributing fine-grained edge coefficients, and the local contrast is further enhanced. The high-precision multi-light-source color constancy is realized by introducing the advanced semantic information to assist the model in understanding the scene and optimizing the feature representation through the fine-grained feature constraint.

Description

Technical Field

[0001] The present invention belongs to the field of computer and information service technology, and in particular relates to a multi-light source color constancy method for performing white balance correction on a color-biased image, and specifically relates to a multi-light source color constancy method and system based on color decoupling. Background Art

[0002] The human visual system has the ability to perceive changes in ambient lighting and eliminate the influence of light sources. This ability is called color constancy (CC). Computer color constancy simulates this ability to correct image color deviation caused by the environment and equipment, which is a basic task in the field of computer vision.

[0003] Most color constancy algorithms are currently designed to solve the single light source problem, that is, assuming that the illumination is uniform everywhere in the scene. The main goal of the single light source color constancy task is to generate a set of light source colors for the scene, which can be divided into three main methods: statistical-based methods, gamut mapping-based methods, and learning-based methods. Statistical-based methods propose many classic assumptions about reflectance properties, such as Gray-World, White-Patch, and Gray-Edge. Such methods are fast to calculate, but are only applicable to specific scenes. Gamut mapping-based methods assume that the specification of a camera can be constructed through a large amount of real data. They obtain the light source through a mapping function from the color gamut of the input image to the specification color gamut. Learning-based methods learn light source reasoning through a large number of training samples. Some studies use regression algorithms such as Bayesian and SVR to infer light sources. Some studies use instantiation matching to calculate corresponding representative colors for various semantic classes or design different algorithms for different scenes. In recent years, convolutional neural networks (CNNs) have been used for color constancy tasks. Some of these methods use CNN to directly regress the light source, such as DS-Net, FC4, C4, IGTN, CLCC, etc. Some methods, such as CCC and FFCC, propose that in the same scene, changes in the light source will cause changes in the position of the image frequency domain space, and CNN can be used to infer the light source through spatial positioning. There are also some methods that generate candidate light sources by clustering, converting light source prediction into a classification task, such as MHCC. CNN-based methods have brought significant performance improvements and better stability.

[0004] However, the illumination of real scenes is often uneven, so the multi-light source color constancy task was proposed. Most of the early multi-light source color constancy methods were based on local prediction, which assumes that the illumination in the local area is uniform. Riess et al. predicted the local light source based on the physical model two-color reflection model. Gijsenij et al. partitioned the image and used the traditional single-light source algorithm to predict the local light source. Uchimi et al. divided the image into superpixels and drew a color line for each superpixel that reflects the relationship between the position and the color of the object. Joze et al. divided the image into different surfaces and calculated the representative color for color mapping. Beigpour defined the local light source prediction as an energy minimization problem in a conditional random field. Mutimbu et al. constructed a pyramid factor graph based on patch and weighted the prediction results of sub-images of different scales. The earliest CNN-based multi-light source method PatchCNN is also based on patch. However, the assumption of uniform illumination in local areas is obviously not in line with reality. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a multi-light source color constancy method and system based on color decoupling, which proposes a pixel-level multi-light source color constancy network capable of realizing color decoupling based on an encoder-decoder structure, and uses an object color perception module to decouple the light source and content, and uses a contrast coordination module to decouple the light source color, thereby solving the multi-light source color constancy problem.

[0006] According to one aspect of the present invention, a multi-light source color constancy method based on color decoupling is provided, comprising:

[0007] Acquire a color-biased image to be corrected;

[0008] Input the color-biased image to be corrected into the trained pixel-level multi-light source color constancy model CDNet, output the predicted value of the pixel-level light source map, and perform pixel-level correction through Von Kries transformation to obtain an image restored to white balance;

[0009] The training of the pixel-level multi-light source color constancy model CDNet includes:

[0010] Construct a multi-light source color constancy dataset;

[0011] A pixel-level multi-light source color constancy model CDNet is constructed, including a pixel-level light source prediction network, a scene content color perception module CCAM and a contrast coordination module CHAM, and is jointly constrained by light source prediction loss, semantic similarity loss and contrast coordination loss; the pixel-level light source prediction network includes an encoder and a decoder, and the input image generates an image feature map through the encoder, and then the decoder predicts the pixel-level light source map; the scene content color perception module CCAM learns the feature representation of the scene under white light, separates the color features from it, and integrates the scene content-color features into the light source reasoning process in a targeted manner by evaluating the influence of the content on the light source prediction; the contrast coordination module CHAM zooms in or out the feature representations of different pixels to different degrees according to the light source difference and spatial position prior, and guides the model to learn features that can distinguish the degree of light source color difference;

[0012] The constructed multi-light source color constancy dataset is used for model training, and the trained pixel-level multi-light source color constancy model CDNet is saved.

[0013] As a further technical solution, the scene content color perception module CCAM further includes:

[0014] Use a large-scale pre-trained model to extract scene content features of color-biased images;

[0015] Use the semantic feature learning module to calibrate scene content features by using white-balanced images and semantic similarity loss in the training phase to correct the content feature representation;

[0016] Use standard colors to match and recalibrate scene content colors;

[0017] The features are integrated to predict the influence of scene content in each area on light source prediction.

[0018] As a further technical solution, use standard colors to match and recalibrate scene content colors, including:

[0019] Standard color sampling and standard color encoding are performed, and the correlation between scene content features and standard color features is calculated.

[0020] As a further technical solution, the contrast coordination module CHAM further includes:

[0021] Pixels are divided into homologous light source, heterologous light source or similar light source groups according to the light source difference between pixels;

[0022] Enhance the uniqueness of homologous light source feature representation and the distinguishability of heterologous light source feature representation;

[0023] Fine-grained boundary coefficients are assigned to features of similar light sources, making the feature representation of pixels with different light source differences softly distinguishable.

[0024] As a further technical solution, the model training is performed using the constructed multi-light source color constancy dataset, which also includes:

[0025] Perform data augmentation on the image samples in the dataset and divide the dataset into training set, validation set and test set;

[0026] The enhanced image samples are input into the constructed pixel-level multi-light source color constancy model to generate the light source prediction feature map, semantic feature map and pixel-level light source prediction map. The three parts of loss are calculated through the loss function and back-propagation is performed.

[0027] After each round of training, the model effect is tested on the validation set, and the model with the best performance on the validation set is saved.

[0028] According to one aspect of the present invention, a multi-light source color constancy system based on color decoupling is provided, comprising:

[0029] A first main module is used to obtain a color-biased image to be corrected;

[0030] The second main module is used to input the color-biased image to be corrected into the trained pixel-level multi-light source color constancy model CDNet, output the predicted value of the pixel-level light source map, and perform pixel-level correction through Von Kries transformation to obtain an image restored to white balance;

[0031] The training of the pixel-level multi-light source color constancy model CDNet includes:

[0032] Construct a multi-light source color constancy dataset;

[0033] A pixel-level multi-light source color constancy model CDNet is constructed, including a pixel-level light source prediction network, a scene content color perception module CCAM and a contrast coordination module CHAM, and is jointly constrained by light source prediction loss, semantic similarity loss and contrast coordination loss; the pixel-level light source prediction network includes an encoder and a decoder, and the input image generates an image feature map through the encoder, and then the decoder predicts the pixel-level light source map; the scene content color perception module CCAM learns the feature representation of the scene under white light, separates the color features from it, and integrates the scene content-color features into the light source reasoning process in a targeted manner by evaluating the influence of the content on the light source prediction; the contrast coordination module CHAM zooms in or out the feature representations of different pixels to different degrees according to the light source difference and spatial position prior, and guides the model to learn features that can distinguish the degree of light source color difference;

[0034] The constructed multi-light source color constancy dataset is used for model training, and the trained pixel-level multi-light source color constancy model CDNet is saved.

[0035] According to one aspect of the present invention, there is provided a multi-light source color constancy system based on color decoupling, comprising a memory and a processor; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the multi-light source color constancy method based on color decoupling.

[0036] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the steps of the multi-light source color constancy method based on color decoupling.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] a) The present invention proposes a semantic-based scene content color perception module, which refers to the memory color phenomenon of the human visual system and alleviates the problem of coupling between scene content color and light source color by learning the color feature priors of scene content, thereby reducing the unrealistic light source color prediction.

[0039] b) The present invention proposes a contrast coordination module based on feature optimization, which optimizes feature representation through mutual calibration between adjacent features, thereby separating illumination and enhancing local contrast, thereby improving the quality of prediction results.

[0040] c) The model structure of the present invention has strong data adaptability and can process multi-light source data as well as single-light source data.

[0041] d) The present invention enables the model to adapt to small-scale data through sufficient data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1Schematic diagram of color coupling problem analysis provided by an embodiment of the present invention. The basic model refers to an encoder-decoder based model that directly predicts the light source without considering the relationship between the imaging components. In the scatter plot in the first row, the red dots represent the real light source distribution. In the second row, the red boxes represent local areas where the light source changes smoothly and suddenly. The local light source histogram statistics correspond to the areas marked by the red boxes.

[0044] Figure 2 A schematic diagram of the CDNet network structure provided by an embodiment of the present invention.

[0045] Figure 3 Schematic diagram of the visualization results of the ablation experiment provided for the embodiment of the present invention. The first column is the input sample. The second column is the visualization result of the model that only uses the basic pixel-level light source prediction network and the light source prediction loss. The third and fourth columns respectively indicate that the CCAM and CHAM components are added to the basic model. The fifth column uses all the components proposed by the present invention. The sixth column is the true value. The three scenes (a) to (c) correspond to a single light source scene, a two-light source scene, and a three-light source scene, respectively. For each (a) to (c), the first row is a visualization of the light source map, the second row is a scatter plot of the light source color distribution, and the third row is a histogram of the local light source distribution (corresponding to the red box area).

[0046] Figure 4 The correction results of the experiment provided in the embodiment of the present invention are compared with the correction results of other methods. The method of the present invention obtains a higher quality corrected image, the overall color is more harmonious and smooth, and is closer to the true value. DETAILED DESCRIPTION

[0047] In recent years, some pixel-level multi-light source color constancy methods based on neural networks have been proposed. Yan et al. directly predicted the light source map based on the encoding-decoding structure. Sidorov et al. proposed AngularGAN based on the generative model GAN, which directly generated the corrected image and realized end-to-end light source correction. Partha et al. converted the light source regression task into a classification task, which obtained multi-light source seed points from the dataset and generated probability maps based on GAN. Afifi et al. predefined multiple groups of white balance settings to correct the image, and obtained the probability map of each predefined light source through CNN. The final light source map was obtained by weighted summation of these predefined light sources. The pixel-level method based on deep learning has achieved significant performance optimization in multi-light source scenes. However, the color displayed by a pixel is determined by multiple factors, including the inherent color of the scene content, the colors of multiple light sources, and the weights of multiple light sources, expressed as:

[0048]

[0049] Among them, I(x,y) is the RGB image, S(x,y) is the surface reflectivity of the scene content, ⊙ is the pixel multiplication, ω i (x,y) is the i-th light source E i The scaling term at (x, y). If the relationship between multiple factors is not considered and the light source value is directly predicted, color coupling often occurs. On the one hand, there is color coupling between the light source and the scene content. Due to the interference of the scene content color, many non-existent light source colors will be predicted, resulting in the predicted light source color gamut being much larger than the actual light source color gamut and lacking consistency. Figure 1 As shown in the figure, the content colors such as yellow wallpaper, green wallpaper, and red books mislead the light source prediction, making the predicted light source color closer to the content color rather than the actual light source color. The scatter plot also proves that the network outputs more non-existent light source values. The real light sources are arranged linearly, while the distribution of the predicted light sources is more dispersed. On the other hand, there is color coupling between light sources in local areas. On the same light-receiving surface, the light source prediction is often dominated by high-frequency colors, while at the junction of the light-receiving surfaces, multiple light source colors interfere with each other, resulting in reduced local contrast. Figure 1 The red box area and its corresponding histogram are shown in the figure. In a small local area, the color change of the real light source on the same light-receiving surface usually presents a slow and smooth change, and the light source value follows an approximate Gaussian distribution. At the junction of the light-receiving surfaces, the light source color suddenly changes, and the boundaries between different light sources are obvious. The predicted light source often shows a cluster of predicted values ​​on the same light-receiving surface, and a few high-frequency colors dominate. The boundary is blurred at the junction of the light-receiving surfaces.

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0051] Based on the encoder-decoder structure, the present invention proposes a pixel-level multi-illuminant color constancy network (CDNet) that can achieve color decoupling. For the problem of color coupling between illuminants and scene content, the present invention proposes a object color perception module (CCAM) to decouple illuminants and content by predicting content color. CCAM uses semantic similarity constraints to learn feature representations of images under standard illuminants and uses it to extract color features of scene content. In addition, it evaluates confidence based on semantic features, so that content that has a greater impact on illuminant prediction has a greater response. By integrating scene content color features into illuminant prediction, CCAM mitigates the misleading of scene content color and improves consistency.

[0052] In order to solve the problem of mutual coupling of light source colors in local areas, the present invention proposes a contrast coordination module (CHAM) to decouple light source colors by mutual calibration between adjacent features. CHAM uses spatial correlation priors to assign greater weights to nearby features, guiding the model to focus on the relationship between feature representations of adjacent pixels. Specifically, CHAM divides pixels into homologous light sources, heterologous light sources, or similar light source groups according to the degree of light source difference between pixels. By enhancing the uniqueness of homologous light source feature representations and the distinguishability of heterologous light source feature representations, CHAM reduces mutual interference between different light sources, thereby improving local edge contrast. In addition, by assigning fine-grained boundary coefficients to features of similar light sources, CHAM makes the feature representations of pixels with different light source differences softly distinguishable, thereby alleviating local clustering and further enhancing local contrast.

[0053] The present invention first trains a pixel-level multi-light source color constancy model CDNet, then performs color cast correction through the trained model, inputs the color cast image into the trained model, predicts the pixel-level light source map, and performs pixel-level correction through Von Kries transform to restore the image to white balance.

[0054] The method for training the pixel-level multi-light source color constancy model CDNet in the technical solution of the present invention comprises the following steps:

[0055] Step 1, select a multi-light source color constancy dataset and divide the dataset into a training set, a validation set and a test set;

[0056] Step 2: Construct a pixel-level multi-light source color constancy model CDNet;

[0057] Step 3: Model training, save the model with the best performance on the validation set.

[0058] Furthermore, the model CDNet in step 2 contains the following:

[0059] The model includes a basic pixel-level light source prediction network, a scene content color perception module CCAM and a CHAM module. The light source prediction loss L Angular , semantic similarity loss L SM , contrast coordination loss L CH Common constraints.

[0060] The pixel-level light source prediction network consists of two parts: the encoder and the decoder. The input image is generated by the encoder to generate an image feature map, and then the decoder predicts the pixel-level light source map. It can be expressed as:

[0061] E(x, y) = f(I(x, y))

[0062] Where f(·) represents the pixel-level light source prediction network, is the input image, is the light source value of each pixel in the predicted image, that is, the light source map. This part is constrained by the true value light source map to predict the light source value. In multi-light source color constancy, the average angle error is usually used as the light source prediction loss L Angular , expressed as:

[0063]

[0064] Among them, E * (x, y) is the true value light source map.

[0065] For the CCAM module, in order to extract the color features of the scene content, CCAM learns the feature representation of the scene under white light, separates the color features from it, and integrates the scene content-color features into the light source reasoning process in a targeted manner by evaluating the impact of the content on the light source prediction. Figure 2 As shown in the CCAM module in Figure 1, first, we use the large-scale pre-trained model CLIP to extract scene content features of color-biased images. Second, the features extracted from color-biased images are usually misleading or even wrong. Therefore, we use the semantic feature learning (SF) module to calibrate the scene content features by using white-balanced images and semantic similarity loss in the training phase to correct the content feature representation. Third, we propose to use standard colors to further match and recalibrate the scene content colors. This step includes standard color sampling and standard color encoding through the color feature encoding (CF) module, and calculating the correlation between scene content features and standard color features using the scene content color feature extraction (CC) module. Finally, we propose a confidence prediction (CE) module to predict the degree of influence of scene content in each region on light source prediction by integrating CLIP features. Specifically, the input sample I(x, y) and its corresponding white-balanced image I B(x, y) A large-scale pre-trained model CLIP with fixed input parameters, which obtains image feature representations of scenes under colored light. Image feature representation of scenes under white light Where h and w are consistent with the features obtained by the encoder. Adjust F I Feature shape, and obtain the feature through a layer of 1×1conv F′ I It consists of two parts, which are input into the CE module and the SF module respectively. One part integrates the information of each channel of the image feature through 1*1conv, infers the influence of the light source color on the feature according to the scene features, and predicts the confidence of each patch. The other part uses 1*1conv to compress information and enhance features to obtain scene features. F′ Ic The input is fed into the CE module, normalized by the BN layer, and the scores are normalized to the range of [0, 1] by sigmoid to obtain the final confidence score map, which is expressed as:

[0066] Coff=sigmoid(bn(F Ic )).

[0067] SF module learning is based on F′ Is Reasoning about the feature representation of scenes under white light. SF first obtains the features through a 5-layer attention block The Attention block uses multi-head attention to capture the dependencies between patches and the content features of the scene from a global and multi-angle perspective, ensuring that the network can fully identify and enhance the features that are important under white light conditions, while suppressing the features caused by light source interference. SM Make F IB Gradually approaching F B Among them, L SM F B and F IB The L2 distance is expressed as:

[0068] L SM =‖F B -F IB ‖2

[0069] Then, F IB After a layer of 1*1conv, the features are filtered and normalized by the BN layer. SF finally obtains the scene content feature representation Among them, c′ is consistent with the number of feature channels obtained by the encoder.

[0070] The CF module encodes the features of each color in the color table. In the HSL color space, the saturation S and brightness L are both set to 50% to balance the vividness and brightness of the color, making the encoded color features more robust. The hue is then uniformly sampled to generate h×w-3 color values, and these colors are converted to the RGB color space. Uniform sampling ensures that the selected colors are evenly distributed on the hue circle, covering a variety of colors and avoiding bias towards specific colors. In addition, three grayscale colors [255,255,255], [128,128,128] and [0, 0, 0] are selected to ensure that the corresponding color features can be extracted for scenes without hue. Finally, a total of h×w color values ​​are obtained and reshaped into a color table. In order to avoid feature interactions between color values, the CF module uses a 4-layer 1×1 conv-relu block for encoding to obtain color features. After a layer of conv-bn block, feature screening is performed to finally obtain the color features of each color value.

[0071] Scene content feature F′ IB and color feature F′ Color The CC module extracts the color features of the scene content. The CC module converts F′ IB As query (Q), F′ Color As key (K) and value (V), the correlation between scene content features and color features is calculated through the cross-attention mechanism, and the matching relationship is dynamically adjusted so that the scene content features can match the most relevant color features, thereby obtaining the color features F of the initial scene content sco Then, the confidence weight Coff is assigned to F through element-wise operation. sco , get the adjusted final scene content-color feature F sc , expressed as:

[0072] F sc =Coff⊙F sco

[0073] F sc It is concatenated with the encoder features to incorporate scene content-color prior knowledge as a hint into the light source reasoning process.

[0074] For the CHAM module, CHAM zooms in or out the feature representations of different pixels to varying degrees based on the light source difference and spatial position priors, guiding the model to learn features that can distinguish the degree of light source color difference, thereby alleviating the coupling phenomenon between light source colors and enhancing local contrast. The input image I(x, y) is obtained through the encoder and decoder to obtain the feature map F(x, y). A pixel "a" is randomly selected, and its feature vector is recorded as F a The difference Ang(x, y) between the light source value of pixel "a" and other pixels is calculated through the true light source map, that is, the angular error:

[0075]

[0076] Among them, E a and E (x，y) are the light source values ​​at pixel point "a" and (x, y) respectively. According to the Ang(x, y) value, the homologous, similar and heterologous pixel masks L(x, y) are established:

[0077]

[0078] Where δ is the preset threshold value, which is set to 15. Homologous pixels are pixels whose light source values ​​are exactly the same as the light source value of the extracted pixel "a". For homologous light source pixels, we hope that the distance between their central features and the feature representation of the extracted "a" pixel is as small as possible. The central feature F of the homologous pixel O It is expressed as:

[0079] F o =∑F(x,y)*(1-w dist (x, y)), if L(x, y) = 1.

[0080] Among them, w dist (x, y) is the spatial correlation weight, that is, the normalized Euclidean distance on the two-dimensional coordinates, expressed as:

[0081]

[0082] Where min(dist) and max(dist) are the minimum and maximum distances between pixel “a” and all other pixels, respectively. Contrast coordination loss of homologous pixels It is expressed as:

[0083]

[0084] Among them, τ0 is the temperature hyperparameter, which is set to 0.6. The difference between the light source color of similar light source pixels and the "a" pixel is less than the set threshold δ, and their light source colors are similar to "a" but not the same. CHAM assigns different feature distinction boundaries to each similar light source pixel. We hope that the distance between the features of each similar light source pixel and the "a" pixel is close to the boundary corresponding to each pixel, neither too far nor too close. The boundary of each similar pixel is:

[0085]

[0086] Among them, m is the boundary coefficient, which is set to 2. is the light source color difference weight of similar light source pixels, that is, the normalized light source difference, expressed as:

[0087]

[0088] Where min(ang) is the minimum angular error between the light source of pixel "a" and all other pixels. Contrast coordination loss of similar pixels It is expressed as:

[0089]

[0090] Among them, τ1 is set to 5. The difference in light source color between the heterogeneous light source pixels and the "a" pixel is greater than the set threshold, and their light source color is quite different from "a". The heterogeneous light source pixels classify the light source color of the pixels according to the light source weight map provided by the dataset, and calculate the central features of multiple light sources respectively. For heterogeneous light source pixels, we hope that the distance between the central features of multiple light sources and the feature representation of the "a" pixel is greater than the preset boundary, which has strong distinguishability. Assuming that there are k light sources in the image, the light source weight map coe(x,y) provided by the dataset gives the proportion of each light source at each pixel. Based on this, we establish the light source category mask C(x,y) of each pixel, expressed as:

[0091] C(x,y)=index(max(coe(x,y)))

[0092] Central features of various light sources for heterogeneous pixels It is expressed as:

[0093]

[0094] in, is the light source color difference weight of the heterogeneous light source pixel, expressed as:

[0095]

[0096] Where max(ang) is the maximum angular error between the light source of pixel "a" and all other pixels. Contrast loss of heterogeneous pixels It is expressed as:

[0097]

[0098] Contrast coordination loss L CH The overall expression is:

[0099]

[0100] The overall loss of CDNet is expressed as:

[0101] Loss = L Angular +L SM +L CH

[0102] Furthermore, the model training in step 3 includes the following steps:

[0103] During the training process, in order to increase data diversity, the training samples are first enhanced. Data enhancement consists of two parts. The first part is light source enhancement. For each light source label in each training image, three values ​​between 0.6 and 1.4 are randomly extracted, and the R, G, and B channels are independently numerically scaled. Correspondingly, the pixel color of the input sample is also scaled in accordance with the light source. The second part includes steps such as random cropping and random rotation. A value between 0.1-1 is randomly extracted, and its multiplication by the pixel value of the short side of the sample is the side length of the cropped area, and a small square area is cropped from the input sample. A value between -30° and 30° is randomly extracted to rotate the cropped image. Then the input samples are uniformly scaled to 224×224 pixels.

[0104] The enhanced image is input into the constructed CDNet to generate the light source prediction feature map, semantic feature map and pixel-level light source prediction map. The three parts of loss are calculated through the loss function and back propagated. After each round of training, the model effect is tested on the validation set, and the model with the best performance on the validation set is saved, that is, the model with the smallest mean of the average angle error of all samples.

[0105] Furthermore, the present invention uses the trained model to perform color cast correction, including the following steps:

[0106] The image is predicted by CDNet at the pixel level. Then, according to the VonKries coefficient law, the gain of the RGB channels of each pixel is independently corrected by diagonal transformation using E(x,y). The value of each channel of the corrected image I'(x,y) can be expressed as:

[0107]

[0108] As a preferred embodiment, the present invention takes the LSMI dataset as an example. LSMI consists of 7,486 images captured by 3 cameras, each image is illuminated by 1-3 light sources. When the image is captured, three Macbeth Color Checkers (MCC) are placed in the scene to extract the light source value in the scene.

[0109] The present invention provides a pixel-level multi-light source color constancy method based on color decoupling, and proposes a CDNet based on CCAM and CHAM, so that the model can generate a light source map that is more in line with the distribution law of real light sources and closer to real light sources, and realizes two-stage pixel-level image color correction. The network structure is as follows Figure 2 The model construction specifically includes the following steps:

[0110] Step 1: Based on the UNet structure, build a basic pixel-level light source prediction network, including an encoder and a decoder, to infer the light source value of each pixel in the image:

[0111] a. The encoder uses the feature extraction part of the small model RseNet18, while ensuring the model accuracy and inference speed, and uses the pre-trained model based on ImageNet to initialize the model parameters of the encoder to obtain a higher training starting point;

[0112] b. The decoder is a combination of multiple blocks consisting of upsampling, convolution, and relu. The encoder and decoder are skipped for feature transmission.

[0113] c. The decoder finally generates a pixel-level light source map and constrains the light source prediction value through the average angle error loss to ensure that the model prediction results gradually approach the true value during the training process.

[0114] Step 2: Construct a content color awareness module CCAM (Content Color Awareness Module). The light source color of each pixel in the scene exists in a fixed range, that is, within the area formed by the chromatic connection of multiple light sources. However, when only the light source value is used as a model constraint, it is often difficult for the model to learn this intrinsic relationship, and instead predicts a more diverse light source color. Predictions beyond the light source color gamut lead to unreasonable color restoration. This module alleviates the coupling of scene content color and light source color by adding the color feature prior of the object in the light source reasoning process, and reduces unreasonable light source color reasoning:

[0115] a. Obtain semantic features. The input sample and its corresponding white-balanced image are segmented into small patches and input into a large-scale pre-trained model clip with fixed parameters to extract semantic features.

[0116] b. Construct semantic similarity loss. Under the constraint of semantic similarity loss, a small network learns to infer the feature representation of objects under white light through the features of the input samples, that is, the feature representation of the white-balanced image, which is used to subsequently extract the color features of the scene content.

[0117] c, uniformly sample color values. Construct a convolutional network consisting of 5 conv-relu layers to learn standard color feature encoding.

[0118] d. Obtain the color features of the scene content. The semantic features of each patch are used as the query (Q), and the color encoding results are used as the key (K) and value (V) to obtain the color features of the object.

[0119] e, predicted object-color prior confidence. The content that is more likely to mislead the network prediction should be paid more attention to. The CCAM module predicts the confidence of each patch through semantics, making the area that is more likely to mislead the light source prediction have a greater impact.

[0120] f, Feature fusion. The confidence-adjusted scene content color features are spliced ​​into the features obtained by the encoder of the pixel-level light source prediction network to provide clues for subsequent decoding as prior knowledge.

[0121] Step 3: Construct a contrast coordination module (CHAM). When the light source color changes smoothly, it presents a quasi-Gaussian distribution, and when it changes suddenly, there is a clear boundary. However, the predicted light source value is concentrated and the boundary is blurred, which does not conform to the actual light source color change rules. By constructing a regularized loss constraint model feature representation distribution, the model learns features that can distinguish the degree of light source color difference:

[0122] a, randomly select a pixel as the anchor, calculate the angle error between the anchor and the light source value of the remaining pixels, and divide the pixels into three categories: positive sample homology, heterology and similar pixels according to the angle error, which are used to execute different feature constraints.

[0123] b. Calculate the light source color difference weight and spatial correlation weight between the anchor and other pixels, which will be used to calculate the feature boundary or feature center weight when executing feature constraints later.

[0124] c. According to different sample classifications, different feature constraints are applied to each pixel, that is, the feature representation is brought closer or further away from the feature representation of the anchor in the feature space, so that the features finally generated by the decoder can reflect the degree of difference in the light source color of different pixels, and the local changes of the light source map generated by the model are more consistent with the real light source changes.

[0125] Model training is implemented in the Ubuntu operating system. The model is built using the Pytorch deep learning framework and the GPU is used for calculation. The specific hardware and software parameters used in this example are shown in Table 1. This example uses the Adam optimizer for optimization, with the batch size set to 16 and the learning rate set to 0.0001, and performs optimization for 600 epochs. During training, 10 anchors are randomly selected on each sample.

[0126] Table 1 Hardware and software parameters

[0127]

[0128] When training the model, the data set is first divided, 70% of the data is randomly selected as the training set, 20% of the data is used as the validation set, and 10% of the data is used as the test set. Before training, all the Mac white cards in the image need to be blocked to prevent the model from obtaining light source cues from the color cards. Then the input image is data enhanced. The processed color-biased image is input into the built model.

[0129] (1) After the image of this example is input into the model, a 224*224*64 feature map is obtained. After a layer of convolution, a pixel-level light source map is obtained, that is, a 224*224 group of light source prediction results. The angular error between each pixel of the predicted light source map and each pixel of the true light source map is calculated and averaged, that is, the average angular error loss:

[0130]

[0131] Where E(x,y) represents the predicted value of the light source at (x,y), E * (x,y) represents the true value of the light source at (x,y). During training, the loss is minimized so that the model prediction results gradually approach the real light source. During verification and testing, the loss is used to evaluate the model prediction effect.

[0132] When only the light source value is used as a training constraint, it is difficult for the model to learn the inherent laws of the real light source distribution and changes. Figure 3As shown in the second column. From the predicted light source diagram, it can be intuitively found that its light source prediction is easily affected by the color of objects in the scene, resulting in unrealistic light source prediction results. It can also be observed from the scatter plot that the predicted light source colors are diverse and lack overall consistency. Observing the local light source changes, in the single light source scene, the light source color of the basic model in a small local area is also more diverse. In the scene with two light sources, two light-receiving surfaces are selected for statistical analysis. The histogram distribution of the basic model at this location shows that the predicted values ​​of the same light-receiving surface are concentrated, but the light source color distribution of the two light-receiving surfaces lacks a clear boundary. In the scene with three light sources, three light-receiving surfaces are selected for statistical analysis. Compared with the true value, it is difficult to observe obvious light source color partitioning in the histogram distribution of the basic model in this area, indicating that the basic model does not perceive clear changes in the light-receiving surface in this area.

[0133] (2) The CCAM module in this example obtains semantic feature representation from the large model and extracts the color features of the scene content from it. The color features of the scene content are passed to the model decoder as prior knowledge, which alleviates the light source prediction that does not conform to reality and greatly reduces the light source range. The specific approach is:

[0134] The input sample image is cropped into 32*32 patches, 49 patches are obtained, and input into the large model CLIP to extract semantic features. During training, the same operation is performed on the white-balanced image corresponding to the input sample. Then the original semantic features obtained from the input sample are input into a small network consisting of 5 layers of self-attention, and the semantic feature representation of the image in the state without color deviation is learned by minimizing the semantic similarity loss. The loss function is expressed as:

[0135] L SM =||F B -F IB ||2

[0136] Among them, F B and F IB They represent the feature maps of the color-biased image obtained by a small network composed of 5 layers of self-attention and the feature map of the white-balanced image obtained by CLIP, respectively. ||·||2 represents the L2 distance. During verification and testing, the semantic feature representation of the image without color bias is directly predicted by a small network composed of 5 layers of self-attention. The color values ​​are uniformly sampled and encoded. Through the cross-attention mechanism, the semantic features are used as Q, and the color codes are used as K and V to extract the color features of the scene content from the semantic features. Then, the confidence of each patch is predicted by the semantic features of the original color-biased image, and the response degree of the object color features is adjusted. Finally, the color features of the scene content are spliced ​​with the features generated by the encoder.

[0137] like Figure 3As shown in the fourth column, after the introduction of CCAM, the influence of the intrinsic color of the scene content on the predicted light source color is alleviated, making the prediction result more consistent with the real light source color. As can be seen from the scatter plot, the range of the predicted light source color is significantly reduced. However, the problem of local light source color coupling is still not solved. The predicted light source map shows low local contrast, the light source color is too smooth, and the boundary at the junction of the illuminated surface is blurred. The local histogram further supports this observation.

[0138] (3) The CHAM module of this example uses the feature representation obtained by the constrained decoder to enable the model to perceive the illumination changes in the local area and obtain features that can distinguish the degree of color difference of the light source, thereby avoiding excessive concentration of prediction values ​​in the local area and making the boundary of the illuminated surface clearer. The specific approach is:

[0139] A feature vector is randomly extracted from the feature map obtained by the decoder as an anchor, and the light source angle error, normalized light source color difference weight and spatial correlation weight between the anchor and the remaining pixels are calculated. After classifying each pixel, the loss of pixels with the same source, different sources and similar light sources is calculated respectively. This module optimizes the feature representation by minimizing these three types of losses as much as possible. The loss function is expressed as:

[0140]

[0141] in and They are the losses of the same light source pixel group, the pixel light source pixel group and the heterogeneous light source pixel group, and the specific calculation method is as described above.

[0142] like Figure 3 As shown in the third column, the introduction of CHAM alone alleviates the problem of local light source color coupling to a certain extent. The predicted light source map shows that the interface between the illuminated surface is clearer, and the local light source color histogram distribution is closer to the true value. However, CHAM causes the model to pay more attention to the difference in feature expression of nearby pixels. Therefore, in some areas, the light source prediction is still affected by the color of the scene content and cannot be calibrated by distant pixels.

[0143] The CCAM module and CHAM module introduced in this example improve the predicted light source in terms of overall color distribution and local color variation. The distribution of the global scatter plot and local histogram is more consistent with the distribution of the true value.

[0144] During the training process, after each round of training, the model is verified on the validation set, the mean of the average angular error of all samples in the validation set is calculated, and the current test result is compared with the previous test result. If it is better than the previous performance, the model is saved.

[0145] After the model is trained, the model performance is tested on the test set. During the test, only the samples are masked with a Mac white card and scaled to 224*224, but no data augmentation is performed. The test includes both objective and subjective evaluations.

[0146] Objective evaluation is the calculation of indicators. The average angular error between the light source map predicted by the model and the true value of the light source is calculated. The evaluation indicators are the mean, median, and three means of the average angular error of all samples, as well as the average of the best 25% and the average of the worst 25%.

[0147] The subjective evaluation is to observe the model correction effect through the correction image. The image is corrected by pixel-level Von Kries transformation, and the model is subjectively evaluated. Figure 4 The following are some test results of this example and other methods. It can be observed that the color changes of the image restored by the method proposed in the present invention are smooth and natural, and the color of the scene content is closer to the true value.

[0148] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a multi-light source color constancy system based on color decoupling, and the device is used to execute a multi-light source color constancy method based on color decoupling in the above method embodiment.

[0149] The system includes: a first main module for acquiring a color-biased image to be corrected; a second main module for inputting the color-biased image to be corrected into a trained pixel-level multi-light source color constancy model CDNet, outputting a predicted value of a pixel-level light source map, and Kries transform is used to perform pixel-level correction to obtain an image restored to white balance; wherein the training of the pixel-level multi-light source color constancy model CDNet includes: constructing a multi-light source color constancy dataset; constructing a pixel-level multi-light source color constancy model CDNet, including a pixel-level light source prediction network, a scene content color perception module CCAM and a contrast coordination module CHAM, and jointly constrained by a light source prediction loss, a semantic similarity loss, and a contrast coordination loss; the pixel-level light source prediction network includes an encoder and a decoder, and the input image generates an image feature map through the encoder, and then the decoder predicts the pixel-level light source map; the scene content color perception module CCAM learns the feature representation of the scene under white light, separates the color feature therefrom, and integrates the scene content-color feature into the light source reasoning process in a targeted manner by evaluating the influence of the content on the light source prediction; the contrast coordination module CHAM performs different degrees of zooming in or out on the feature representation of different pixels according to the light source difference and the spatial position prior, and guides the model to learn features that can distinguish the degree of light source color difference; the constructed multi-light source color constancy dataset is used to train the model, and the trained pixel-level multi-light source color constancy model CDNet is saved.

[0150] An embodiment of the present invention provides a multi-light source color constancy system based on color decoupling. Based on an encoder-decoder structure, a pixel-level multi-light source color constancy network capable of achieving color decoupling is proposed, and an object color perception module is used to decouple the light source and content, and a contrast coordination module is used to decouple the light source color, thereby solving the multi-light source color constancy problem.

[0151] It should be noted that the system embodiment provided by the present invention is used to implement the methods in the above method embodiment as well as the methods in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set. The principle is basically the same as the principle of the above system embodiment provided by the present invention. As long as the technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above system embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining the technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above system embodiment to obtain the corresponding system class embodiments, which are used to implement the methods in other method class embodiments. For example:

[0152] Based on the contents of the above system embodiment, as a preferred embodiment, a multi-light source color constancy system based on color decoupling is provided in an embodiment of the present invention, wherein the scene content color perception module CCAM further includes:

[0153] Use a large-scale pre-trained model to extract scene content features of color-biased images;

[0154] Use the semantic feature learning module to calibrate scene content features by using white-balanced images and semantic similarity loss in the training phase to correct the content feature representation;

[0155] Use standard colors to match and recalibrate scene content colors;

[0156] The features are integrated to predict the influence of scene content in each area on light source prediction.

[0157] Based on the contents of the above system embodiment, as a preferred embodiment, a multi-light source color constancy system based on color decoupling provided in an embodiment of the present invention uses standard color matching and recalibration of scene content color, including:

[0158] Standard color sampling and standard color encoding are performed, and the correlation between scene content features and standard color features is calculated.

[0159] Based on the contents of the above system embodiment, as a preferred embodiment, a multi-light source color constancy system based on color decoupling is provided in the embodiment of the present invention, wherein the contrast coordination module CHAM further includes:

[0160] Pixels are divided into homologous light source, heterologous light source or similar light source groups according to the light source difference between pixels;

[0161] Enhance the uniqueness of homologous light source feature representation and the distinguishability of heterologous light source feature representation;

[0162] Fine-grained boundary coefficients are assigned to features of similar light sources, making the feature representation of pixels with different light source differences softly distinguishable.

[0163] Based on the content of the above system embodiment, as a preferred embodiment, a multi-light source color constancy system based on color decoupling provided in the embodiment of the present invention uses the constructed multi-light source color constancy dataset for model training, and also includes:

[0164] Perform data augmentation on the image samples in the dataset and divide the dataset into training set, validation set and test set;

[0165] The enhanced image samples are input into the constructed pixel-level multi-light source color constancy model to generate the light source prediction feature map, semantic feature map and pixel-level light source prediction map. The three parts of loss are calculated through the loss function and back-propagation is performed.

[0166] After each round of training, the model effect is tested on the validation set, and the model with the best performance on the validation set is saved.

[0167] Based on the same inventive concept as the above embodiment, an embodiment of the present invention further provides a multi-light source color constancy system based on color decoupling, comprising a memory and a processor; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the multi-light source color constancy method based on color decoupling.

[0168] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the steps of the multi-light source color constancy method based on color decoupling.

[0169] In summary, the present invention proposes a color decoupling network (CDNet) to achieve high-quality multi-light source color constancy. The network includes a content color perception module (CCAM) and a contrast coordination module (CHAM). CCAM draws on the memory color effect in the human visual system, learns the scene content color feature prior, and separates the scene content color and illumination color by providing the model with the color features of the content, thereby reducing the prediction of the light source outside the color gamut and enhancing the consistency. CHAM optimizes the feature representation and separates the light source by mutual calibration between adjacent features. CHAM uses spatial correlation to make the model more sensitive to the relationship between adjacent features, and uses illumination parallax to guide feature classification. CHAM improves local edge contrast by enhancing the uniqueness of homogeneous illumination features and the distinctiveness of heterogeneous illumination features. In addition, the soft distinctiveness of similar illumination features is emphasized by assigning fine-grained edge coefficients, further enhancing the local contrast. The present invention achieves high-precision multi-light source color constancy by introducing high-level semantic information to assist the model in understanding the scene, while optimizing the feature representation through fine-grained feature constraints.

[0170] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A multi-light source color constancy method based on color decoupling, characterized in that: include: Acquire a color-biased image to be corrected; Input the color-biased image to be corrected into the trained pixel-level multi-light source color constancy model CDNet, output the predicted value of the pixel-level light source map, and perform pixel-level correction through Von Kries transformation to obtain an image restored to white balance; The training of the pixel-level multi-light source color constancy model CDNet includes: Construct a multi-light source color constancy dataset; A pixel-level multi-light source color constancy model CDNet is constructed, including a pixel-level light source prediction network, a scene content color perception module CCAM and a contrast coordination module CHAM, and is jointly constrained by light source prediction loss, semantic similarity loss and contrast coordination loss; the pixel-level light source prediction network includes an encoder and a decoder, and the input image generates an image feature map through the encoder, and then the decoder predicts the pixel-level light source map; the scene content color perception module CCAM learns the feature representation of the scene under white light, separates the color features from it, and integrates the scene content-color features into the light source reasoning process in a targeted manner by evaluating the influence of the content on the light source prediction; the contrast coordination module CHAM zooms in or out the feature representations of different pixels to different degrees according to the light source difference and spatial position prior, and guides the model to learn features that can distinguish the degree of light source color difference; The constructed multi-light source color constancy dataset is used for model training, and the trained pixel-level multi-light source color constancy model CDNet is saved.

2. The method for color constancy of multiple light sources based on color decoupling according to claim 1, characterized in that: The scene content color perception module CCAM further includes: Use a large-scale pre-trained model to extract scene content features of color-biased images; Use the semantic feature learning module to calibrate scene content features by using white-balanced images and semantic similarity loss in the training phase to correct the content feature representation; Use standard colors to match and recalibrate scene content colors; The features are integrated to predict the influence of scene content in each area on light source prediction.

3. The method for color constancy of multiple light sources based on color decoupling according to claim 2, characterized in that: Use standard colors to match and recalibrate scene content colors, including: Standard color sampling and standard color encoding are performed, and the correlation between scene content features and standard color features is calculated.

4. The method for color constancy of multiple light sources based on color decoupling according to claim 1, characterized in that: The contrast coordination module CHAM further comprises: Pixels are divided into homologous light source, heterologous light source or similar light source groups according to the light source difference between pixels; Enhance the uniqueness of homologous light source feature representation and the distinguishability of heterologous light source feature representation; Fine-grained boundary coefficients are assigned to features of similar light sources, making the feature representation of pixels with different light source differences softly distinguishable.

5. The method for color constancy of multiple light sources based on color decoupling according to claim 1, characterized in that: Model training using the constructed multi-light source color constancy dataset also includes: Perform data augmentation on the image samples in the dataset and divide the dataset into training set, validation set and test set; The enhanced image samples are input into the constructed pixel-level multi-light source color constancy model to generate the light source prediction feature map, semantic feature map and pixel-level light source prediction map. The three parts of loss are calculated through the loss function and back-propagation is performed. After each round of training, the model effect is tested on the validation set, and the model with the best performance on the validation set is saved.

6. A multi-light source color constancy system based on color decoupling, characterized in that: include: A first main module is used to obtain a color-biased image to be corrected; The second main module is used to input the color-biased image to be corrected into the trained pixel-level multi-light source color constancy model CDNet, output the predicted value of the pixel-level light source map, and perform pixel-level correction through Von Kries transformation to obtain an image restored to white balance; The training of the pixel-level multi-light source color constancy model CDNet includes: Construct a multi-light source color constancy dataset; A pixel-level multi-light source color constancy model CDNet is constructed, including a pixel-level light source prediction network, a scene content color perception module CCAM and a contrast coordination module CHAM, and is jointly constrained by light source prediction loss, semantic similarity loss and contrast coordination loss; the pixel-level light source prediction network includes an encoder and a decoder, and the input image generates an image feature map through the encoder, and then the decoder predicts the pixel-level light source map; the scene content color perception module CCAM learns the feature representation of the scene under white light, separates the color features from it, and integrates the scene content-color features into the light source reasoning process in a targeted manner by evaluating the influence of the content on the light source prediction; the contrast coordination module CHAM zooms in or out the feature representations of different pixels to different degrees according to the light source difference and spatial position prior, and guides the model to learn features that can distinguish the degree of light source color difference; The constructed multi-light source color constancy dataset is used for model training, and the trained pixel-level multi-light source color constancy model CDNet is saved.

7. A multi-light source color constancy system based on color decoupling, characterized in that: It comprises a memory and a processor; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of a multi-light source color constancy method based on color decoupling as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the steps of the multi-light source color constancy method based on color decoupling as claimed in any one of claims 1 to 5.

Images