A Multi-Light Source Color Constancy Method and System Based on Color Decoupling

Abstract

This invention discloses a multi-source color constancy method and system based on color decoupling. It utilizes a pixel-level multi-source color constancy model to achieve multi-source color constancy, including a Content Color Awareness (CCAM) module and a Contrast Coordination (CHAM) module. CCAM learns prior color features of scene content, separating scene content color from illumination color by providing the model with these color features, thus reducing the prediction of out-of-gamut light sources. CHAM leverages spatial correlation to make the model more sensitive to the relationships between adjacent features and uses illumination disparity to guide feature classification; it improves local edge contrast by enhancing the uniqueness of homogeneous illumination features and the distinguishability of heterogeneous illumination features; and it further enhances local contrast by assigning fine-grained edge coefficients to emphasize the soft distinguishability of similar illumination features. This invention achieves high-precision multi-source color constancy by introducing advanced semantic information to assist the model in understanding the scene and optimizing feature representation through fine-grained feature constraints.

Description

Technical Field

[0001] This invention belongs to the field of computer and information service technology, and specifically relates to a multi-source color constancy method for white balance correction of color-biased images, and more specifically to a multi-source color constancy method and system based on color decoupling. Background Technology

[0002] The human visual system has the ability to perceive changes in ambient lighting and eliminate the influence of light sources; this ability is called color constancy (CC). Computer color constancy corrects image color shifts caused by environment and device by simulating this ability, and is a fundamental task in the field of computer vision.

[0003] Most current color constancy algorithms aim to solve the single-light source problem, assuming uniform illumination throughout the scene. The main goal of the single-light source color constancy task is to generate a set of light source colors for the scene, and it mainly falls into three categories: statistical methods, gamut mapping-based methods, and learning-based methods. Statistical methods propose many classic assumptions about reflectance properties, such as Gray-World, White-Patch, and Gray-Edge. These methods are computationally fast but only applicable to specific scenes. Gamut mapping-based methods assume that a camera's specification can be constructed from a large amount of real data; they obtain the light source through a mapping function from the input image's color gamut to the canonical color gamut. Learning-based methods learn light source inference through a large number of training samples. Some studies use regression algorithms such as Bayesian and SVR to infer light sources. Some studies utilize instantiation matching to compute corresponding representative colors for various semantic classes or design different algorithms for different scenes. In recent years, convolutional neural networks (CNNs) have been used for color constancy tasks. Some of these methods directly regress light sources using CNNs, such as DS-Net, FC4, C4, IGTN, and CLCC. Some methods, such as CCC and FFCC, propose that changes in light sources within the same scene lead to changes in the image's frequency domain spatial location, and CNNs can be used to infer the light source through spatial localization. Other methods, such as MHCC, generate candidate light sources through clustering, transforming light source prediction into a classification task. CNN-based methods offer significant performance improvements and better stability.

[0004] However, lighting in real-world scenes is often uneven, hence the development of multi-source color constancy tasks. Early multi-source color constancy methods were mostly based on local prediction, assuming uniform illumination within local regions. Riess et al. predicted local light sources based on a physical model of dichroic reflection. Gijsenij et al. partitioned the image and used traditional single-source algorithms to predict local light sources. Uchimi et al. divided the image into superpixels and drew color lines for each superpixel that represented the relationship between location and object color. Joze et al. divided the image into different surfaces and calculated representative colors for color mapping. Beigpour defined local light source prediction as an energy minimization problem within a conditional random field. Mutimbu et al. constructed a pyramid factor graph based on patches and weighted the prediction results of sub-images at different scales. The earliest CNN-based multi-source method, PatchCNN, was also based on patches. However, the assumption of uniform illumination within local regions clearly does not conform to reality. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a multi-light source color constancy method and system based on color decoupling. It proposes a pixel-level multi-light source color constancy network based on an encoder-decoder structure that can achieve color decoupling, and uses an object color perception module to decouple the light source and content, and a contrast coordination module to decouple the light source color, thus solving the multi-light source color constancy problem.

[0006] According to one aspect of the present invention, a multi-light source color constancy method based on color decoupling is provided, comprising:

[0007] Obtain the color cast image to be corrected;

[0008] The color-shifted image to be corrected is input into the trained pixel-level multi-light source color constancy model CDNet, which outputs the predicted value of the pixel-level light source map. Then, pixel-level correction is performed through Von Kries transform to obtain an image restored to white balance.

[0009] The training of the pixel-level multi-light source color constancy model CDNet includes:

[0010] Construct a multi-source color constancy dataset;

[0011] A pixel-level multi-light source color constancy model, CDNet, is constructed, comprising a pixel-level light source prediction network, a scene content color perception module (CCAM), and a contrast coordination module (CHAM), constrained by light source prediction loss, semantic similarity loss, and contrast coordination loss. The pixel-level light source prediction network includes an encoder and a decoder. The input image is encoded into an image feature map by the encoder, and then the decoder predicts the pixel-level light source map. The scene content color perception module (CCAM) learns the feature representation of the scene under white light, extracts color features from it, and selectively integrates scene content-color features into the light source inference process by evaluating the influence of content on light source prediction. The contrast coordination module (CHAM), based on the light source difference and spatial location prior, brings the feature representations of different pixels closer or further away to different degrees, guiding the model to learn features that can distinguish the degree of color difference between light sources.

[0012] The model was trained using the constructed multi-source color constancy dataset, and the trained pixel-level multi-source color constancy model CDNet was saved.

[0013] As a further technical solution, the scene content color perception module (CCAM) also includes:

[0014] Use large-scale pre-trained models to extract scene content features from color-skewed images;

[0015] The semantic feature learning module is used to calibrate scene content features by using white balance images and semantic similarity loss during the training phase to correct the content feature representation.

[0016] Use standard colors to match and recalibrate scene content colors;

[0017] The features are integrated to predict the degree of influence of scene content in each region on light source prediction.

[0018] As a further technical solution, standard color matching and recalibration of scene content colors are used, including:

[0019] Perform standard color sampling and standard color encoding, and calculate the correlation between scene content features and standard color features.

[0020] As a further technical solution, the contrast coordination module CHAM also includes:

[0021] Pixels are categorized into groups with the same light source, different light sources, or similar light sources based on the degree of difference in light sources between them.

[0022] Enhance the uniqueness of characteristic representations of sources with the same source and the distinguishability of characteristic representations of sources with different sources;

[0023] Fine-grained boundary coefficients are assigned to the features of similar light sources, enabling the feature representation of pixels with different light source differences to have soft discriminative properties.

[0024] As a further technical solution, model training using a constructed multi-light source color constancy dataset also includes:

[0025] Data augmentation is performed on the image samples in the dataset, and the dataset is divided into training set, validation set and test set;

[0026] The enhanced image samples are input into the constructed pixel-level multi-light source color constancy model to generate light source prediction feature maps, semantic feature maps and pixel-level light source prediction maps. The three parts of the loss are calculated through the loss function and backpropagation is performed.

[0027] After each round of training, the model is tested on the validation set, and the model that performs best on the validation set is saved.

[0028] According to one aspect of the present invention, a multi-source color constancy system based on color decoupling is provided, comprising:

[0029] The first main module is used to acquire the color cast image to be corrected;

[0030] The second main module is used to input the color-shifted image to be corrected into the trained pixel-level multi-light source color constancy model CDNet, output the predicted value of the pixel-level light source map, and perform pixel-level correction through Von Kries transform to obtain an image restored to white balance.

[0031] The training of the pixel-level multi-light source color constancy model CDNet includes:

[0032] Construct a multi-source color constancy dataset;

[0033] A pixel-level multi-light source color constancy model, CDNet, is constructed, comprising a pixel-level light source prediction network, a scene content color perception module (CCAM), and a contrast coordination module (CHAM), constrained by light source prediction loss, semantic similarity loss, and contrast coordination loss. The pixel-level light source prediction network includes an encoder and a decoder. The input image is encoded into an image feature map by the encoder, and then the decoder predicts the pixel-level light source map. The scene content color perception module (CCAM) learns the feature representation of the scene under white light, extracts color features from it, and selectively integrates scene content-color features into the light source inference process by evaluating the influence of content on light source prediction. The contrast coordination module (CHAM), based on the light source difference and spatial location prior, brings the feature representations of different pixels closer or further away to different degrees, guiding the model to learn features that can distinguish the degree of color difference between light sources.

[0034] The model was trained using the constructed multi-source color constancy dataset, and the trained pixel-level multi-source color constancy model CDNet was saved.

[0035] According to one aspect of the present invention, a multi-source color constancy system based on color decoupling is provided, comprising a memory and a processor; the memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to perform the steps of the multi-source color constancy method based on color decoupling.

[0036] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the steps of the described method for multi-source color constancy based on color decoupling.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] a) This invention proposes a semantic-based scene content color perception module. By referencing the color memory phenomenon of the human visual system, it learns the color feature prior of scene content, which alleviates the problem of coupling between scene content color and light source color and reduces unrealistic light source color prediction.

[0039] b) This invention proposes a feature-optimized contrast coordination module, which optimizes feature representation through mutual calibration between adjacent features, thereby separating illumination and enhancing local contrast, and improving the quality of prediction results.

[0040] c) The model structure of this invention has strong data adaptability and can handle multi-source data as well as single-source data.

[0041] d) This invention enables the model to adapt to small-scale data through sufficient data augmentation. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1This is a schematic diagram illustrating the color coupling problem analysis provided in an embodiment of the present invention. The base model refers to an encoder-decoder-based model that directly predicts the light source without considering the relationship between imaging components. In the scatter plot in the first row, red dots represent the actual distribution of light sources. In the second row, red boxes represent local areas where light source changes are stable or abrupt. The local light source histogram statistics correspond to the areas marked by the red boxes.

[0044] Figure 2 This is a schematic diagram of the CDNet network structure provided in an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram illustrating the visualization results of the ablation experiment provided in this embodiment of the invention. The first column shows the input samples. The second column shows the visualization results of the model using only the basic pixel-level light source prediction network and the light source prediction loss. The third and fourth columns show the addition of CCAM and CHAM components to the basic model, respectively. The fifth column shows the use of all components proposed in this invention. The sixth column shows the ground truth. Scenes (a) to (c) correspond to a single-light source scene, a two-light source scene, and a three-light source scene, respectively. For each (a) to (c), the first row is a visualization of the light source map, the second row is a scatter plot of the light source color distribution, and the third row is a histogram of the local light source distribution (corresponding to the red box area).

[0046] Figure 4 This diagram illustrates a comparison between the correction results of the experiment provided in this embodiment of the invention and the correction results of other methods. The method of this invention yields higher quality corrected images with more harmonious and smoother overall colors, and is closer to the true value. Detailed Implementation

[0047] In recent years, several pixel-level multi-light source color constancy methods based on neural networks have been proposed. Yan et al. directly predicted the light source map based on an encoder-decoder structure. Sidorov et al. proposed AngularGAN based on the generative model GAN, which directly generates corrected images and achieves end-to-end light source correction. Partha et al. transformed the light source regression task into a classification task, obtaining multi-light source seed points from the dataset and generating probability maps based on GAN. Afifi et al. predefined multiple white balance settings to correct images and obtained the probability maps of each predefined light source through CNN. The final light source map is obtained by weighted summation of these predefined light sources. Pixel-level methods based on deep learning have achieved significant performance optimization in multi-light source scenes. However, the color displayed by a pixel is determined by multiple factors, including the inherent color of the scene content, the colors of multiple light sources, and the weights of multiple light sources, expressed as:

[0048]

[0049] Where I(x,y) is the RGB image, S(x,y) is the surface reflectance of the scene content, ⊙ is pixel multiplication, and ω i (x,y) represents the i-th light source E i The scaling term at (x,y). If the relationships between multiple factors are not considered, directly predicting light source values ​​often leads to color coupling. On one hand, there is color coupling between the light source and the scene content; due to interference from the scene content colors, many non-existent light source colors will be predicted, resulting in a predicted light source color gamut that is much larger than the actual light source color gamut, lacking consistency. For example... Figure 1 As shown, the colors of content such as yellow wallpaper, green wallpaper, and red books mislead the light source prediction, making the predicted light source color closer to the content color than the actual light source color. The scatter plot also proves that the network outputs more non-existent light source values; the real light sources are linearly arranged, while the predicted light sources are more dispersed. On the other hand, there is color coupling between light sources in local areas. On the same illuminated surface, light source prediction often prioritizes high-frequency colors, while at the boundaries of illuminated surfaces, multiple light source colors interfere with each other, leading to reduced local contrast. For example... Figure 1 The area within the red box and its corresponding histogram are shown. Within small local areas, the color change of the actual light source on the same illuminated surface typically exhibits a slow and smooth change, with the light source values ​​following an approximately Gaussian distribution. At the boundary between illuminated surfaces, the light source color changes abruptly, and the boundaries between different light sources are clear. In contrast, predicted light sources often show a clustering of predicted values ​​on the same illuminated surface, with a few high-frequency colors dominating. The boundaries are blurred at the boundary between illuminated surfaces.

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0051] This invention proposes a pixel-level multi-light source color constancy network (CDNet) based on an encoder-decoder structure, enabling color decoupling. To address the color coupling problem between light sources and scene content, this invention proposes an object color perception module (CCAM) that decouples light sources and content by predicting content color. CCAM uses semantic similarity constraints to learn feature representations of images under standard light sources and uses them to extract color features of scene content. Furthermore, it evaluates confidence based on semantic features, enabling content with a greater impact on light source prediction to have a larger response. By integrating scene content color features into light source prediction, CCAM mitigates the misleading nature of scene content color and improves consistency.

[0052] To address the issue of color coupling between light sources within local regions, this invention proposes a Contrast Coordination Module (CHAM) that decouples light source colors through mutual calibration between adjacent features. CHAM leverages spatial correlation priors to assign greater weights to nearby features, guiding the model to focus on the relationships between the feature representations of adjacent pixels. Specifically, CHAM categorizes pixels into groups with the same light source, different light sources, or similar light sources based on the degree of difference between them. By enhancing the uniqueness of feature representations with the same light source and the discriminativeness of feature representations with different light sources, CHAM reduces mutual interference between different light sources, thereby improving local edge contrast. Furthermore, by assigning fine-grained boundary coefficients to features of similar light sources, CHAM enables soft discriminativeness in the feature representations of pixels with different light source differences, thereby mitigating local clustering and further enhancing local contrast.

[0053] This invention first trains a pixel-level multi-light source color constancy model CDNet, and then performs color cast correction through the trained model. The color cast image is input into the trained model to predict the pixel-level light source map, and pixel-level correction is performed through Von Kries transform to restore the image to white balance.

[0054] The method for training a pixel-level multi-light source color constancy model CDNet in the technical solution of this invention includes the following steps:

[0055] Step 1: Select a multi-light source color constancy dataset and divide the dataset into a training set, a validation set, and a test set.

[0056] Step 2: Construct the pixel-level multi-light source color constancy model CDNet;

[0057] Step 3: Model training, save the model that performs best on the validation set.

[0058] Furthermore, the CDNet model in step 2 includes the following:

[0059] The model includes a basic pixel-level light source prediction network, a scene content color perception module (CCAM), and a CHAM module. The light source prediction loss is L... Angular Semantic similarity loss L SM Contrast compensation loss L CH Common constraints.

[0060] The pixel-level light source prediction network consists of two parts: an encoder and a decoder. The input image is encoded into an image feature map by the encoder, and then the decoder predicts pixel-level light source maps. This is represented as:

[0061] E(x,y)=f(I(x,y))

[0062] Where f(·) represents a pixel-level light source prediction network, For the input image, This section describes the predicted light source values ​​for each pixel in the image, i.e., the light source map. The predicted light source values ​​are constrained by the ground truth light source map. In multi-light source color constancy, the average angular error is typically used as the light source prediction loss L. Angular , is represented as:

[0063]

[0064] Among them, E * (x, y) is the true light source image.

[0065] For the CCAM module, in order to extract the color features of scene content, CCAM learns the feature representation of the scene under white light, separates the color features from it, and then, by evaluating the degree of influence of the content on the light source prediction, selectively integrates the scene content-color features into the light source inference process. For example... Figure 2 As shown in the CCAM module, firstly, we use the large-scale pre-trained model CLIP to extract scene content features from color-biased images. Secondly, the features extracted from color-biased images are often misleading or even erroneous. Therefore, we use the Semantic Feature Learning (SF) module to calibrate the scene content features by using white balance images and semantic similarity loss during the training phase to correct the content feature representation. Thirdly, we propose using standard colors to further match and recalibrate scene content colors. This step includes standard color sampling and encoding via the Color Feature Encoding (CF) module, and calculating the correlation between scene content features and standard color features using the Scene Content Color Feature Extraction (CC) module. Finally, we propose a Confidence Prediction (CE) module to predict the degree of influence of scene content on light source prediction in each region by integrating CLIP features. Specifically, the input sample I(x, y) and its corresponding white balance image I BUsing a large-scale pre-trained CLIP model with fixed (x, y) input parameters, image feature representations of scenes under colored light are obtained. Image feature representation of a scene under white light Where h and w are consistent with the features obtained by the encoder. Adjust F I The feature shape is obtained by passing it through a 1×1 convolution layer. F′ I It consists of two parts, which are input to the CE module and the SF module, respectively. One part integrates the image feature information from each channel through a 1*1 convolution, infers the influence of light source color on the features based on scene features, and predicts the confidence of each patch. The other part uses 1*1 convolutions for information compression and feature enhancement to obtain scene features. F′ Ic The input CE module is normalized by the BN layer, and the scores are normalized to the range [0, 1] using a sigmoid function to obtain the final confidence score map, represented as:

[0066] Coff = sigmoid(bn(F) Ic )).

[0067] SF module learning is based on F′ Is Inferring feature representations of scenes under white light. SF first obtains features through 5 layers of attention blocks. The attention block uses multi-head attention to capture dependencies between patches and scene content features from multiple global perspectives, ensuring the network can fully identify and enhance features important under white light conditions while suppressing features affected by light source interference. This is achieved through feature similarity loss L... SM Make F IB Gradually approaching F B Among them, L SM For F B and F IB The L2 distance is expressed as:

[0068] L SM =||F B -F IB ||2

[0069] Then, F IB Feature filtering is performed using a 1x1 convolutional layer, followed by normalization using a batch normalization layer. SF ultimately yields the scene content feature representation. Here, c′ is consistent with the number of feature channels obtained by the encoder.

[0070] The CF module encodes the characteristics of each color in the color table. In the HSL color space, both saturation (S) and brightness (L) are set to 50% to balance the vividness and brightness of the colors, making the encoded color features more robust. Then, hue is uniformly sampled to generate h×w-3 color values, which are then converted to the RGB color space. Uniform sampling ensures that the selected colors are evenly distributed on the color wheel, covering a wide range of colors and avoiding bias towards specific colors. Furthermore, three grayscale colors, [255,255,255], [128,128,128], and [0,0,0], are selected to ensure that the corresponding color features can be extracted in achromatic scenes. Finally, a total of h×w color values ​​are obtained and reshaped into a color table. To avoid feature interactions between color values, the CF module uses four layers of 1×1 conv-relu blocks for encoding to obtain color features. After passing through a conv-bn block for feature filtering, the color features of each color value are finally obtained.

[0071] Scene content features F′ IB With color feature F′ Color The CC module extracts color features from the scene content. The CC module will then use F′... IB As query(Q), F′ Color Using key(K) and value(V) as key and value, the correlation between scene content features and color features is calculated through a cross-attention mechanism. The matching relationship is dynamically adjusted so that scene content features can be matched with the most relevant color features, thereby obtaining the initial color features F of the scene content. sco Subsequently, the confidence weight Coff is assigned to F through an element-wise operation. sco Obtain the adjusted final scene content - color feature F sc , is represented as:

[0072] F sc =Coff⊙F sco

[0073] F sc By concatenating with encoder features, scene content and color prior knowledge are incorporated as clues into the light source reasoning process.

[0074] For the CHAM module, CHAM, based on the prior knowledge of light source differences and spatial locations, brings the feature representations of different pixels closer or further apart to varying degrees. This guides the model to learn features that can distinguish the degree of color difference between light sources, thereby alleviating the coupling phenomenon between light source colors and enhancing local contrast. The input image I(x, y) obtains the feature map F(x, y) through the encoder and decoder. A randomly selected pixel "a" has its feature vector denoted as F. a The difference Ang(x, y) between pixel "a" and other pixel light source values ​​is calculated using the true light source map; this is the angular error.

[0075]

[0076] Among them, E a and E (x，y) Let Ang(x, y) be the light source values ​​at pixel "a" and (x, y), respectively. Based on the Ang(x, y) values, construct pixel masks L(x, y) for homologous, similar, and heterologous origins:

[0077]

[0078] Where δ is a preset threshold, set to 15. A source pixel is a pixel whose light source value is exactly the same as the light source value of the extracted pixel "a". For source pixels, we want the distance between its center feature and the feature representation of the extracted pixel "a" to be as small as possible. The center feature F of the source pixel... O Represented as:

[0079] F o =∑F(x, y)*(1-w dist (x, y)), if L(x, y) = 1.

[0080] Among them, w dist (x, y) represents the spatial correlation weight, i.e., the normalized Euclidean distance on two-dimensional coordinates, expressed as:

[0081]

[0082] Where min(dist) and max(dist) are the minimum and maximum distances between pixel "a" and all other pixels, respectively. Contrast reconciliation loss for source pixels. Represented as:

[0083]

[0084] Where τ0 is the temperature hyperparameter, set to 0.6. Similar light source pixels have a color difference of less than a set threshold δ with pixel "a"; their light source colors are similar to but not identical to "a". CHAM assigns different feature discrimination boundaries to each similar light source pixel. We want the distance between the features of each similar light source pixel and pixel "a" to approximate the boundary of each pixel, neither too far nor too close. The boundaries of each similar pixel are:

[0085]

[0086] Where m is the boundary coefficient, set to 2. The weight of the light source color difference between pixels with similar light sources, i.e., the normalized light source difference, is expressed as:

[0087]

[0088] In the formula, min(ang) represents the minimum angular error between pixel "a" and all other pixels in terms of light source. Contrast coordination loss for similar pixels. Represented as:

[0089]

[0090] τ1 is set to 5. Pixels with heterogeneous light sources have a color difference greater than a set threshold from pixel "a", indicating a significant difference in color between them. Pixels with heterogeneous light sources are classified according to the light source weight map provided in the dataset, and the center features of multiple light sources are calculated separately. For pixels with heterogeneous light sources, we aim that the distance between the center features of multiple light sources and the feature representation of pixel "a" is greater than a preset boundary, demonstrating strong discriminative power. Assuming there are k light sources in the image, the light source weight map coe(x,y) provided in the dataset gives the proportion of each light source at each pixel. Based on this, we establish a light source category mask C(x,y) for each pixel, expressed as:

[0091] C(x,y)=index(max(coe(x,y)))

[0092] Central features of various light sources in heterogeneous pixels Represented as:

[0093]

[0094] in, The color difference weight of pixels from different light sources is expressed as:

[0095]

[0096] In the formula, max(ang) represents the maximum angular error between pixel "a" and the light sources of all other pixels. Contrast loss of heterogeneous pixels. Represented as:

[0097]

[0098] Contrast compensation loss L CH The whole is represented as:

[0099]

[0100] The overall loss of CDNet is expressed as:

[0101] Loss = L Angular +L SM +L CH

[0102] Furthermore, model training in step 3 includes the following steps:

[0103] During training, to increase data diversity, data augmentation was first performed on the training samples. Data augmentation consisted of two parts. The first part was light source augmentation. For each light source label in each training image, three values ​​between 0.6 and 1.4 were randomly selected, and their R, G, and B channels were independently scaled. Correspondingly, the pixel colors of the input samples were also scaled to match the light source. The second part included random cropping and rotation. Values ​​between 0.1 and 1 were randomly selected, and their multiplication by the shorter side pixel value of the sample yielded the side length of the cropping region, cropping a small square area from the input samples. Values ​​between -30° and 30° were randomly selected to rotate the cropped image. Finally, the input samples were uniformly scaled to 224×224 pixels.

[0104] The enhanced image is input into the constructed CDNet to generate a light source prediction feature map, a semantic feature map, and a pixel-level light source prediction map. The loss for these three parts is calculated using a loss function, and backpropagation is performed. After each round of training, the model performance is tested on a validation set, and the model that performs best on the validation set, i.e., the model with the smallest mean angular error across all samples, is saved.

[0105] Furthermore, the present invention utilizes a trained model for color cast correction, including the following steps:

[0106] The image is generated by predicting pixel-level light sources using CDNet. Then, according to the Von Kries coefficient theorem, the gain of each pixel's RGB channels is independently corrected using E(x,y) through a diagonal transformation. The values ​​of each channel of the corrected image I'(x,y) can be expressed as:

[0107]

[0108] As a preferred embodiment, this invention uses the LSMI dataset as an example. LSMI consists of 7,486 images captured by three cameras, each illuminated by one to three light sources. During image capture, three Macbeth Color Checkers (MCCs) were placed in the scene to extract the light source values.

[0109] This invention provides a pixel-level multi-light source color constancy method based on color decoupling. It proposes a CDNet based on CCAM and CHAM, enabling the model to generate light source maps that better conform to the distribution patterns of real light sources and more closely resemble actual light sources. This achieves two-stage pixel-level image color correction. The network structure is as follows: Figure 2 As shown. The model construction specifically includes the following steps:

[0110] Step 1: Based on the UNet architecture, construct a basic pixel-level light source prediction network, including an encoder and a decoder, to infer the light source values ​​of each pixel in the image:

[0111] a. The encoder uses the feature extraction part of the small model RseNet18 to ensure both model accuracy and inference speed, and uses a pre-trained model based on ImageNet to initialize the encoder's model parameters to obtain a high training starting point.

[0112] b. The decoder is a combination of multiple blocks consisting of upsampling, convolution, and ReLU. There are skip connections between the encoder and decoder for feature transfer.

[0113] c. The decoder finally generates a pixel-level light source map. The light source prediction value is constrained by the average angular error loss to ensure that the model prediction result gradually approaches the true value during the training process.

[0114] Step 2: Construct the Content Color Awareness Module (CCAM). The light source color of each pixel within a scene exists within a fixed range, that is, the area formed by the chromaticity connections of multiple light sources. However, when only light source values ​​are used as model constraints, the model often struggles to learn this inherent relationship, instead predicting a wider variety of light source colors. Predictions exceeding the light source color gamut lead to unreasonable color inpainting. This module alleviates the coupling between scene content color and light source color by incorporating prior color features of objects during the light source inference process, reducing unreasonable light source color inference.

[0115] a. Obtain semantic features. Segment the input sample and its corresponding white balance image into small patches, and input them into a large-scale pre-trained model clip with fixed parameters to extract semantic features.

[0116] b. Construct a semantic similarity loss. Under the constraint of the semantic similarity loss, a small network learns to infer the feature representation of objects under white light, i.e., the feature representation of the white balance image, based on the features of the input samples, which is then used to extract the color features of the scene content.

[0117] c. Uniformly sample color values. Construct a convolutional network consisting of 5 conv-relu layers to learn standard color feature encoding.

[0118] d. Obtain the color features of the scene content. The semantic features of each patch are used as the query (Q), and the color encoding results are used as the key (K) and value (V) to obtain the color features of the objects.

[0119] e. Predict object-color prior confidence. Content that is more likely to mislead network predictions should receive greater attention. The CCAM module uses semantic prediction to assess the confidence of each patch, thus significantly impacting areas more prone to misleading light source predictions.

[0120] f. Feature fusion. The scene content color features adjusted by confidence are concatenated into the features obtained by the encoder of the pixel-level light source prediction network, serving as prior knowledge to provide clues for subsequent decoding.

[0121] Step 3: Construct the Contrast Harmonization Module (CHAM). When the light source color changes smoothly, it exhibits a Gaussian-like distribution; when there are abrupt changes, clear boundaries exist. However, the predicted light source values ​​are concentrated and have blurred boundaries, which does not conform to the true rules of light source color change. By constructing a regularized loss constraint model feature representation distribution, the model learns features capable of discerning the degree of color difference in the light source:

[0122] a. Randomly select a pixel as the anchor, calculate the angle error between the anchor and the light source values ​​of the other pixels, and classify the pixels into three categories based on the angle error: positive sample homogeneous pixels, heterogeneous pixels, and similar pixels, to perform different feature constraints.

[0123] b. Calculate the weights of the light source color difference between the anchor and the other pixels, as well as the weights of the spatial correlation, which are used to calculate the weights of the feature boundary or feature center when performing feature constraints later.

[0124] c. Based on different sample classifications, different feature constraints are applied to each pixel, that is, the feature representation and the anchor feature representation are brought closer or pushed further apart in the feature space, so that the features finally generated by the decoder can reflect the degree of difference in the light source color of different pixels, and make the local changes of the light source map generated by the model more consistent with the changes of the real light source.

[0125] Model training was implemented on the Ubuntu operating system, using the PyTorch deep learning framework to build the model and employing a GPU for computation. The specific hardware and software parameters used in this example are shown in Table 1. The Adam optimizer was used for optimization, with a batch size of 16 and a learning rate of 0.0001, for 600 epochs. During training, 10 anchors were randomly selected for each sample.

[0126] Table 1 Hardware and software parameters

[0127]

[0128] During model training, the dataset is first divided into three parts: 70% is randomly selected as the training set, 20% as the validation set, and 10% as the test set. Before training, all white cards in the images need to be occluded to prevent the model from receiving light source cues from the color cards. Then, data augmentation is performed on the input images. The processed color-shifted images are then input into the built model.

[0129] (1) In this example, after the image is input into the model, a feature map of 224*224*64 is obtained. After one convolution layer, a pixel-level light source map is obtained, which is 224*224 sets of light source prediction results. The angular error between each pixel in the predicted light source map and each pixel in the true light source map is calculated and averaged, which is the average angular error loss:

[0130]

[0131] Where E(x,y) represents the predicted value of the light source at (x,y), E * (x,y) represents the true value of the light source at (x,y). During training, this loss is minimized to gradually make the model's predictions closer to the true light source. During validation and testing, this loss is used to evaluate the model's prediction performance.

[0132] When only light source values ​​are used as training constraints, the model struggles to learn the intrinsic patterns of real light source distribution and changes. For example... Figure 3The second column shows the predicted light source plot. It's clear from the plot that the predicted light source is easily influenced by the colors of objects in the scene, resulting in unrealistic predictions. The scatter plot also shows a variety of predicted light source colors, lacking overall consistency. Observing local light source variations, in single-light source scenes, the base model exhibits considerable color diversity within small local areas. In two-light source scenes, statistical analysis was performed at the boundary between two illuminated surfaces. The base model's histogram distribution at this location shows a concentration of predicted values ​​for the same illuminated surface, but the color distribution between the two illuminated surfaces lacks clear boundaries. In three-light source scenes, statistical analysis was performed at the boundary between the three illuminated surfaces. Compared to the ground truth, the base model's histogram distribution in this region shows little clear color zoning, indicating that the base model does not perceive clear changes in illuminated surfaces in this area.

[0133] (2) In this example, the CCAM module obtains semantic feature representations from the large model and extracts color features of the scene content from them. These color features are then passed as prior knowledge to the model decoder, mitigating unrealistic light source predictions and significantly narrowing the range of light sources. The specific implementation is as follows:

[0134] The input sample image is cropped into 32*32 pixels, resulting in 49 patches. These patches are then fed into a large model (CLIP) to extract semantic features. During training, the same operation is performed on the white balance image corresponding to the input sample. The raw semantic features obtained from the input sample are then fed into a small network consisting of 5 layers of self-attention. The semantic feature representation of the image in a color-bias-free state is learned by minimizing the semantic similarity loss. The loss function is expressed as:

[0135] L SM =||F B -F IB ||2

[0136] Among them, F B and F IB The first two images represent the feature maps obtained from a color-biased image using a small network consisting of 5 layers of self-attention, and the second image, a white balance image using CLIP, respectively. ||·||² represents the L2 distance. During validation and testing, the small network of 5 layers of self-attention directly predicts the semantic feature representation of the image in the color-bias-free state. Color values ​​are uniformly sampled and encoded. Through a cross-attention mechanism, semantic features are used as Q, and color encoding as K and V, to extract the color features of the scene content from the semantic features. The confidence of each patch is then predicted using the semantic features of the original color-biased image, adjusting the response level of the object color features. Finally, the color features of the scene content are concatenated with the features generated by the encoder.

[0137] like Figure 3As shown in the fourth column, the introduction of CCAM mitigates the influence of the inherent colors of the scene content on the predicted light source colors, thus making the prediction results more consistent with the real light source colors. The scatter plot shows that the range of predicted light source colors is significantly reduced. However, the problem of local light source color coupling remains unresolved. The predicted light source map exhibits low local contrast, overly smooth light source colors, and blurred boundaries at the junctions of illuminated surfaces. Local histograms further support this observation.

[0138] (3) In this example, the CHAM module uses the feature representation obtained by the constraint decoder to enable the model to perceive changes in illumination within a local area, acquire features that can distinguish the degree of color difference of the light source, avoid overly concentrated prediction values ​​in local areas, and make the boundaries at the junction of the illuminated surfaces clearer. The specific implementation is as follows:

[0139] A feature vector is randomly selected from the feature map obtained from the decoder as an anchor. The light source angle error between the anchor and the remaining pixels, the normalized light source color difference weight, and the spatial correlation weight are calculated. After classifying each pixel, the losses for pixels with the same, different, and similar light sources are calculated separately. This module optimizes the feature representation by minimizing these three types of losses as much as possible. The loss function is expressed as:

[0140]

[0141] in and The losses are respectively for pixel groups with the same source light source, pixel groups with different source light sources, and pixel groups with different source light sources. The specific calculation method is as described above.

[0142] like Figure 3 As shown in the third column, introducing CHAM alone alleviates the problem of local light source color coupling to some extent. The predicted light source map shows clearer boundaries at the illuminated surfaces, and the local light source color histogram distribution is closer to the true value. However, CHAM causes the model to focus more on the feature representation differences of nearby pixels. Therefore, in some areas, light source prediction is still affected by the color of the scene content and cannot be calibrated using distant pixels.

[0143] The CCAM and CHAM modules introduced in this example improve the prediction of both the overall color distribution and local color variations of the light source. The distributions of the global scatter plot and local histograms are now more consistent with the true value distribution.

[0144] During training, after each round of training, the model is validated on the validation set. The mean of the average angular error of the model on all samples on the validation set is calculated, and the current test result is compared with the previous test result. If the current result is better than the previous result, the model is saved.

[0145] After the model is trained, its performance is tested on the test set. During testing, only the samples are occluded with a Macbeth card and scaled to 224*224 pixels, but no data augmentation is performed. The test includes both objective and subjective evaluations.

[0146] Objective evaluation is achieved through index calculation. The average angular error between the image predicted by the model and the true light source is calculated. The evaluation indices are the mean, median, and three-means of the average angular error for all samples, as well as the average of the best 25% and the average of the worst 25%.

[0147] Subjective evaluation involves observing the model's correction effect through the corrected image. The model is subjectively evaluated by correcting the image using pixel-level Von Kries transform. Figure 4 The following are partial test results for this example and other methods. It can be observed that the method proposed in this invention restores images with smooth and natural color changes, and the colors of the scene content are closer to the ground truth.

[0148] The implementation of the various embodiments of the present invention is based on programmed processing by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a multi-source color constancy system based on color decoupling. This device is used to execute a multi-source color constancy method based on color decoupling from the above method embodiments.

[0149] The system includes: a first main module for acquiring the color-shifted image to be corrected; and a second main module for inputting the color-shifted image to be corrected into a trained pixel-level multi-light source color constancy model CDNet, outputting the predicted values ​​of the pixel-level light source map, and transmitting the predicted values ​​through Von. Kries transform is used for pixel-level correction to obtain an image restored to white balance. The training of the pixel-level multi-light source color constancy model CDNet includes: constructing a multi-light source color constancy dataset; constructing the pixel-level multi-light source color constancy model CDNet, including a pixel-level light source prediction network, a scene content color perception module (CCAM), and a contrast coordination module (CHAM), constrained by light source prediction loss, semantic similarity loss, and contrast coordination loss; the pixel-level light source prediction network includes an encoder and a decoder, where the input image is encoded into an image feature map by the encoder, and then the decoder predicts a pixel-level light source map; the scene content color perception module (CCAM) learns the feature representation of the scene under white light, extracts color features from it, and selectively integrates scene content-color features into the light source inference process by evaluating the influence of content on light source prediction; the contrast coordination module (CHAM) adjusts the feature representation of different pixels to varying degrees based on light source difference and spatial location priors, guiding the model to learn features that can distinguish the degree of color difference between light sources; the model is trained using the constructed multi-light source color constancy dataset, and the trained pixel-level multi-light source color constancy model CDNet is saved.

[0150] This invention provides a multi-light source color constancy system based on color decoupling. Based on an encoder-decoder structure, a pixel-level multi-light source color constancy network capable of color decoupling is proposed. An object color perception module is used to decouple the light source and content, and a contrast coordination module is used to decouple the light source color, thus solving the multi-light source color constancy problem.

[0151] It should be noted that the system embodiments provided by this invention, in addition to implementing the methods in the above method embodiments, are also used to implement the methods in other method embodiments provided by this invention. The difference lies only in setting corresponding functional modules, and their principles are basically the same as those of the above system embodiments provided by this invention. As long as those skilled in the art, based on the above system embodiments and referring to the specific technical solutions in other method embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and improve the modules in the above system embodiments while ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments. For example:

[0152] Based on the above system embodiments, as a preferred embodiment, the present invention provides a multi-light source color constancy system based on color decoupling, wherein the scene content color perception module (CCAM) further includes:

[0153] Use large-scale pre-trained models to extract scene content features from color-skewed images;

[0154] The semantic feature learning module is used to calibrate scene content features by using white balance images and semantic similarity loss during the training phase to correct the content feature representation.

[0155] Use standard colors to match and recalibrate scene content colors;

[0156] The features are integrated to predict the degree of influence of scene content in each region on light source prediction.

[0157] Based on the above system embodiments, as a preferred embodiment, this invention provides a multi-light source color constancy system based on color decoupling, which uses standard color matching and recalibration of scene content colors, including:

[0158] Perform standard color sampling and standard color encoding, and calculate the correlation between scene content features and standard color features.

[0159] Based on the above system embodiments, as a preferred embodiment, the multi-light source color constancy system based on color decoupling provided in this embodiment of the invention further includes a contrast coordination module (CHAM) that includes:

[0160] Pixels are categorized into groups with the same light source, different light sources, or similar light sources based on the degree of difference in light sources between them.

[0161] Enhance the uniqueness of characteristic representations of sources with the same source and the distinguishability of characteristic representations of sources with different sources;

[0162] Fine-grained boundary coefficients are assigned to the features of similar light sources, enabling the feature representation of pixels with different light source differences to have soft discriminative properties.

[0163] Based on the above system embodiments, as a preferred embodiment, the present invention provides a multi-source color constancy system based on color decoupling, which uses a constructed multi-source color constancy dataset for model training, and further includes:

[0164] Data augmentation is performed on the image samples in the dataset, and the dataset is divided into training set, validation set and test set;

[0165] The enhanced image samples are input into the constructed pixel-level multi-light source color constancy model to generate light source prediction feature maps, semantic feature maps and pixel-level light source prediction maps. The three parts of the loss are calculated through the loss function and backpropagation is performed.

[0166] After each round of training, the model is tested on the validation set, and the model that performs best on the validation set is saved.

[0167] Based on the same inventive concept as the above embodiments, this embodiment of the invention also provides a multi-source color constancy system based on color decoupling, including a memory and a processor; the memory stores program instructions that are executed by the processor, and the processor calls the program instructions to execute the steps of the multi-source color constancy method based on color decoupling.

[0168] Based on the same inventive concept as the above embodiments, this embodiment of the invention also provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the steps of the multi-light source color constancy method based on color decoupling.

[0169] In summary, this invention proposes a Color Decoupling Network (CDNet) to achieve high-quality multi-source color constancy. This network includes a Content Color Awareness Module (CCAM) and a Contrast Coordination Module (CHAM). CCAM draws on the memory color effect in the human visual system, learning prior color features of scene content. By providing the model with color features of content, it separates scene content color from illumination color, thereby reducing out-of-gamut light source predictions and enhancing consistency. CHAM optimizes feature representations and separates light sources through mutual calibration between adjacent features. CHAM utilizes spatial correlation to make the model more sensitive to the relationships between adjacent features and uses illumination disparity to guide feature classification. By enhancing the uniqueness of homogeneous illumination features and the distinguishability of heterogeneous illumination features, CHAM improves local edge contrast. Furthermore, by assigning fine-grained edge coefficients to emphasize the soft distinguishability of similar illumination features, it further enhances local contrast. This invention achieves high-precision multi-source color constancy by introducing high-level semantic information to assist the model in understanding the scene and by optimizing feature representations through fine-grained feature constraints.

[0170] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for multi-source color constancy based on color decoupling, characterized in that, include: Obtain the color cast image to be corrected; The color-shifted image to be corrected is input into the trained pixel-level multi-light source color constancy model CDNet, which outputs the predicted value of the pixel-level light source map. Then, pixel-level correction is performed through Von Kries transform to obtain an image restored to white balance. The training of the pixel-level multi-light source color constancy model CDNet includes: Construct a multi-source color constancy dataset; A pixel-level multi-light source color constancy model, CDNet, is constructed, comprising a pixel-level light source prediction network, a scene content color perception module (CCAM), and a contrast coordination module (CHAM), constrained by light source prediction loss, semantic similarity loss, and contrast coordination loss. The pixel-level light source prediction network includes an encoder and a decoder. The input image is encoded by the encoder to generate an image feature map, and then the decoder predicts the pixel-level light source map. The scene content color perception module (CCAM) learns the feature representation of the scene under white light, extracts color features from it, and selectively integrates scene content-color features into the light source inference process by evaluating the influence of content on light source prediction. The scene content color perception module (CCAM) further includes: extracting scene content features of the image using a large-scale pre-trained model; calibrating the scene content features using a semantic feature learning module, correcting the content feature representation by using white balance images and semantic similarity loss during the training phase; and obtaining the color features of each color value through standard color sampling and standard color encoding using a color feature encoding module. The scene content color feature extraction module calculates the correlation between scene content features and color features; the scene content color features adjusted by confidence are concatenated into the features obtained by the encoder of the pixel-level light source prediction network as prior knowledge to provide clues for subsequent decoding; the contrast coordination module CHAM, based on the light source difference degree and spatial location prior, brings the feature representations of different pixels closer or further apart to different degrees, guiding the model to learn features that can distinguish the degree of light source color difference; the contrast coordination module CHAM also includes: randomly extracting a feature vector from the feature map obtained by the decoder as an anchor, calculating the light source angle error between the anchor and the other pixels, the normalized light source color difference degree weight, and the spatial correlation weight; classifying pixels into groups of same source light sources, different source light sources, or similar light sources based on the light source angle error between pixels; enhancing the uniqueness of the feature representation of the same source light source and the distinguishability of the feature representation of the different source light source; assigning fine-grained boundary coefficients to the features of similar light sources, so that the feature representations of pixels with different light source differences have soft distinguishability. The model was trained using the constructed multi-source color constancy dataset, and the trained pixel-level multi-source color constancy model CDNet was saved.

2. The multi-source color constancy method based on color decoupling according to claim 1, characterized in that, Model training using a constructed multi-light source color constancy dataset also includes: Data augmentation is performed on the image samples in the dataset, and the dataset is divided into training set, validation set and test set; The enhanced image samples are input into the constructed pixel-level multi-light source color constancy model to generate light source prediction feature maps, semantic feature maps and pixel-level light source prediction maps. The three parts of the loss are calculated through the loss function and backpropagation is performed. After each round of training, the model is tested on the validation set, and the model that performs best on the validation set is saved.

3. A multi-source color constancy system based on color decoupling, characterized in that, include: The first main module is used to acquire the color cast image to be corrected; The second main module is used to input the color-shifted image to be corrected into the trained pixel-level multi-light source color constancy model CDNet, output the predicted value of the pixel-level light source map, and perform pixel-level correction through Von Kries transform to obtain an image restored to white balance. The training of the pixel-level multi-light source color constancy model CDNet includes: Construct a multi-source color constancy dataset; A pixel-level multi-light source color constancy model, CDNet, is constructed, comprising a pixel-level light source prediction network, a scene content color perception module (CCAM), and a contrast coordination module (CHAM), constrained by light source prediction loss, semantic similarity loss, and contrast coordination loss. The pixel-level light source prediction network includes an encoder and a decoder. The input image is encoded by the encoder to generate an image feature map, and then the decoder predicts the pixel-level light source map. The scene content color perception module (CCAM) learns the feature representation of the scene under white light, extracts color features from it, and selectively integrates scene content-color features into the light source inference process by evaluating the influence of content on light source prediction. The scene content color perception module (CCAM) further includes: extracting scene content features of the image using a large-scale pre-trained model; calibrating the scene content features using a semantic feature learning module, correcting the content feature representation by using white balance images and semantic similarity loss during the training phase; and obtaining the color features of each color value through standard color sampling and standard color encoding using a color feature encoding module. The scene content color feature extraction module calculates the correlation between scene content features and color features; the scene content color features adjusted by confidence are concatenated into the features obtained by the encoder of the pixel-level light source prediction network as prior knowledge to provide clues for subsequent decoding; the contrast coordination module CHAM, based on the light source difference degree and spatial location prior, brings the feature representations of different pixels closer or further apart to different degrees, guiding the model to learn features that can distinguish the degree of light source color difference; the contrast coordination module CHAM also includes: randomly extracting a feature vector from the feature map obtained by the decoder as an anchor, calculating the light source angle error between the anchor and the other pixels, the normalized light source color difference degree weight, and the spatial correlation weight; classifying pixels into groups of same source light sources, different source light sources, or similar light sources based on the light source angle error between pixels; enhancing the uniqueness of the feature representation of the same source light source and the distinguishability of the feature representation of the different source light source; assigning fine-grained boundary coefficients to the features of similar light sources, so that the feature representations of pixels with different light source differences have soft distinguishability. The model was trained using the constructed multi-source color constancy dataset, and the trained pixel-level multi-source color constancy model CDNet was saved.

4. A multi-source color constancy system based on color decoupling, characterized in that, It includes a memory and a processor; the memory stores program instructions that are executed by the processor, which invokes the program instructions to perform the steps of the multi-light source color constancy method based on color decoupling as described in any one of claims 1 to 2.

5. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the steps of the multi-source color constancy method based on color decoupling as described in any one of claims 1 to 2.

Images