A semantically preserved color constancy method and system

Abstract

This invention proposes a semantically preserved color constancy method and system. Leveraging the crucial role of memory color in human color constancy, this method incorporates semantics into color constancy by using a pre-trained classification model on the ImageNet dataset as network initialization parameters. A Semantic Constraint Module (SCM) is designed to measure the feature differences between the color constancy model and the pre-trained model using feature similarity loss, ensuring the network continuously learns semantic knowledge during training and preventing semantic forgetting. An Auxiliary Calibration Module (ACM) is also designed, using small local regions as auxiliary input to enhance features in non-semantic regions with limited intrinsic color ranges, and avoiding potential biases through local-global consistency constraints. This invention achieves high-precision color constancy without increasing inference computation by guiding the correction of low-level color features with high-level semantic information.

Description

Technical Field

[0001] This invention belongs to the field of computer and information service technology, and specifically relates to a method and system for color constancy in white balance correction of color-biased images. Background Technology

[0002] Color constancy (CC) refers to the human visual system's ability to perceive relatively stable object colors despite changes in ambient light. Computer color constancy simulation, which mimics the color constancy of the human visual system and corrects color-biased images, is not only a fundamental task for image processing in devices such as cameras, but also beneficial for downstream advanced vision tasks in the field of computer vision, such as object detection and semantic segmentation. This has led to widespread attention being paid to computer color constancy.

[0003] Computer color constancy typically corrects color cast by predicting and removing light sources in a scene. Current methods are mostly single-light-source color constancy, assuming that all pixels in an image have the same light source. Single-light-source color constancy is generally divided into statistical and learning-based methods. Statistical methods rely on assumptions about the reflectivity properties of a scene. Gray-World assumes that the average reflectivity of all surfaces in a scene is gray, and the average pixel value of the image corresponds to the scene's light source. White-Patch assumes that the brightest point in the image perfectly reflects the scene's light source, thus considering the maximum value of a pixel in the image as the color of the light source. Gray-Edge extends light source calculation to the derivative of the image and proposes that the average reflectivity of object boundaries in a scene is gray. Subsequent methods have extended these assumptions and made some progress. These methods do not require training and are computationally fast, but because they have few free parameters, they are difficult to handle complex real-world scenes.

[0004] Learning-based methods utilize large amounts of data to learn light source prediction, such as gamut mapping-based methods, machine learning-based methods, instantiation-based methods, and CNN-based methods. Gamut mapping methods assume that the camera's canonical color gamut can be obtained from a large amount of data; these methods compute light sources by mapping the image's color gamut to the canonical color gamut. Some methods leverage machine learning to design more robust light source regression algorithms, such as SVR-based algorithms, ridge regression-based algorithms, and Bayesian-based algorithms. Furthermore, instantiation-based methods infer illumination based on scene content. They classify images or each region of an image based on handcrafted features and then select the optimal algorithm for each class for combined prediction. Van et al. proposed that images are a mixture of semantic classes such as sky and grass, with a constant average reflectance for each semantic class. Based on this concept, Joze et al. further divided images into different surfaces based on texture and color features and used the average reflectance of each surface as the representative color of the corresponding class for color transformation. Bianco et al. observed that skin colors form a cluster in color space, and faces can provide reliable clues for predicting scene light sources.

[0005] With the development of CNNs, many CNN-based color constancy methods have emerged in recent years. Bianco et al. proposed the first CNN-based local region light source regression algorithm, assuming that the light source is uniform in small local regions, and thus cropped the image into multiple patches as input. DS-Net is also based on patches, designing a dual-branch structure to capture different light source patterns, thereby adapting to different local scenes and improving the robustness of light source prediction. FFCC revealed that changes in image color lead to spatial variations in the frequency domain, thus defining color constancy as a spatial localization task. IGTN proposed a triplet network that constructs a light source-sensitive but content-insensitive feature space by combining semantic and color features. C4 proposed a cascaded network that optimizes the results from coarse to fine by cascading three light source prediction networks. CLCC proposed that the limitation of training data volume will cause the network to tend to predict light sources based on scene content such as indoor and outdoor settings. This method designs a color constancy network based on contrastive learning, learning light source-dependent features rather than stray correlations by constructing contrast pairs suitable for the color constancy task.

[0006] CNN-based methods have demonstrated superior performance in the field of color constancy. However, color constancy is ill-conditioned, meaning that different combinations of object surfaces and light sources can produce the same color; for example, a white wall with a red light source will appear red, and a red wall with a white light source will also appear red. When information is insufficient, CNN networks face the challenge of estimating ambiguity in their illumination predictions. To overcome this problem, some studies have introduced semantic information to prompt the network to learn the relationship between color and objects. Inspired by semantic segmentation, Wang and Afifi et al. have manually or algorithmically labeled semantic masks and provided them to the network along with the input image, but their accuracy is susceptible to the quality of the masks. Other methods use pre-trained models to incorporate semantics. Lou et al. first proposed obtaining semantic and hierarchical features through pre-training on the ImageNet dataset, which are beneficial for light source prediction. Hu et al.'s FC4 extracts semantic features through a feature extraction network pre-trained on ImageNet and calculates the confidence weight of each patch based on the semantic values ​​to avoid the negative impact of semantically ambiguous patches. However, methods based on pre-trained models face the problem of semantic knowledge loss during training. Summary of the Invention

[0007] To overcome the shortcomings of existing methods, this invention proposes a semantically preserving color constancy method, SPNet (Semantic Preserving Network). SPNet consists of two modules: a semantic constraint module (SCM) and an auxiliary calibration module (ACM). The SCM utilizes the semantic features of the pre-trained model to construct a semantic preservation mechanism to alleviate the problem of semantic knowledge loss. This module proposes a semantic feature similarity loss to ensure that semantic knowledge is continuously learned during training.

[0008] However, addressing the semantic loss problem presents a new challenge: guiding the model to focus on semantic regions leads to the neglect of features in non-semantic background regions. Some non-semantic background regions also contain important information about color constancy. Some scenes, such as the sky, grass, and faces, exhibit limited color diversity under standard white light, meaning they have a limited intrinsic color range. This implies that they are easy to learn corresponding color features. However, for ImageNet-based pre-trained models, these scenes are all non-semantic background regions. To address this issue, we further designed the ACM module. We randomly sample image patches to calibrate the local-global light source relationship. By providing small image patches to the network, we avoid interference from semantic foregrounds, thus enhancing the features of object regions with limited intrinsic color ranges. Simultaneously, the selected patches also provide consistency between local and global perspectives, mitigating potentially large prediction biases in certain regions.

[0009] The technical solution of this invention is a semantically preserved color constancy method, comprising the following steps:

[0010] Step 1: Select the color constancy dataset and divide the dataset into training, validation, and test sets;

[0011] Step 2: Construct the SPNet color constancy model, including a global illumination estimation network, a semantic constraint module (SCM), and an auxiliary calibration module (ACM).

[0012] The global illumination estimation network consists of two parts: a feature extractor and a light source estimation function. The image is first processed by the feature extractor to obtain an image feature map, and the feature map is then processed by the light source estimation function to predict the light source value.

[0013] The Semantic Constraint Module (SCM) for learning semantic knowledge consists of two branches: the upper branch is a pre-trained model based on ImageNet, and the lower branch is a feature extractor of the global illumination estimation network.

[0014] The Auxiliary Calibration Module (ACM), used to improve prediction accuracy, has the same network structure as the global illumination estimation network, and the two share parameters.

[0015] Step 3: Model training, save the model that performs best on the validation set;

[0016] Step 4, color cast correction: Input the color cast image into the trained model to predict the light source value, and then correct the color cast image through Von Kries transform.

[0017] Furthermore, the SPNet model in step 2 includes the following:

[0018] The model includes a basic global illumination estimation network, a semantic constraint module (SCM), and an auxiliary calibration module (ACM).

[0019] The global illumination estimation network consists of two parts: a feature extractor and a light source estimation function. The image is first processed by the feature extractor to obtain an image feature map, and then the feature map is used by the light source estimation function to predict the light source value. The specific formula is as follows:

[0020] E0 = f θ (I0)

[0021] Among them, f θ (·) represents the global illumination estimation network. This represents the RGB input image, and θ represents the model parameters. Indicates the light source prediction result [E] r E g E b ], where E r E represents the red light component. gE represents the green light component. b Indicates the blue light component. θ (·) is specifically represented as:

[0022] f θ (·)=τ μ (h σ (·))

[0023] In the above formula, σ and μ represent the feature extractor h, respectively. σ (·) and the light source estimation function τ μ (·) model parameters. h σ (·) The lightweight model SqueezeNet is used, and the initialization parameters for this part are based on the parameters of a pre-trained model using ImageNet. τ μ The (·) convolutional layer consists of two convolutional layers. The first layer has a 6x6 kernel, a stride of 1, and padding of 3. The second layer has a 1x1 kernel and a stride of 1. The loss function for this part is expressed as the angular error between the predicted and true light source values, and the specific formula is as follows:

[0024]

[0025] In the formula, E * This represents the true value of the light source.

[0026] The SCM module consists of two branches; the upper branch is a pre-trained model based on ImageNet. The next branch is the feature extractor h of the global illumination estimation network. σ (·). Input image I0 input Obtain semantic features The parameters of this branch are frozen. Meanwhile, the input image I0 input h... σ (·) Obtain feature map F0 = h σ (I0) uses similarity loss to narrow the distance between F0 and F1, making the feature encodings of the two branches more similar. This process can be written as:

[0027]

[0028] Among them, L s Let L1 distance represent the similarity loss between F0 and F1. The loss function of SCM is defined as:

[0029] L s (F0,F1)=||F0-F1||1

[0030] The ACM module has the same structure as the global illumination estimation network model, and the two share parameters. The models jointly optimize the parameters through consistency loss. ACM uniformly crops the training samples I0 into an m*m grid. n patches are randomly selected from this grid, denoted as... p s It includes two types of patches: patches with limited intrinsic color range. and patches with uncertain intrinsic color range The extracted n patches are fed into the network along with the input image. s Through the global illumination estimation network f θ (·) Obtain the predicted values ​​for each patch. Based on the single-source assumption, theoretically p s Predicted values ​​for each patch It should have strong consistency with the overall prediction result E0. Therefore, by narrowing the gap between E0 and E... s Consistency supervision is achieved by using the angular error between the parameters. This process can be represented as:

[0031]

[0032] In the formula, L c E represents s The angular error between E0 and E0 is defined as:

[0033]

[0034] As mentioned earlier, SPNet's loss function includes three types: the light source prediction loss L of the illumination estimation network. main The feature similarity loss of the semantic constraint module (SCM) L s Consistency loss L with auxiliary calibration module (ACM) c The final loss is expressed as:

[0035] L SPCC =L main +αL s +βL c

[0036] In the formula, α = 1 and β = 2.2 are weight parameters.

[0037] Furthermore, model training in step 3 includes the following steps:

[0038] During training, the training images undergo data augmentation, primarily including random cropping, random rotation, and light source scaling. First, a small square region is cropped from the original input image, with a crop size between 0.1 and 1 times the shorter side of the image. The cropped image is then randomly rotated between -30° and 30°. These images are then scaled to 512×512 pixels and finally randomly flipped horizontally with a probability of 0.5. To increase data diversity, the light source labels in each image are also randomly scaled, with three values ​​between 0.6 and 1.4 randomly selected to scale the R, G, and B channels respectively. Simultaneously, the pixel colors of the original image are scaled to match the light source. Furthermore, gamma correction is applied to convert linear images into non-linear images, and the image values ​​are normalized to [0,1].

[0039] The enhanced image is input into the constructed SPNet to generate feature maps, global light source predictions, and light source predictions for each patch. The loss is calculated using a loss function, and backpropagation is performed. The model's performance is validated on a validation set every 20 training epochs, and the model with the smallest mean angular error is saved.

[0040] Furthermore, in step 4, when using the trained model, the image only needs to pass through the global illumination estimation network to obtain the predicted light source values; the SCM and ACM modules are no longer needed, as these modules are only used for feature selection during training. After obtaining the predicted light source values ​​through the global illumination estimation network, the image is corrected using the Von Kries transform, which is expressed as:

[0041] I′0=I o / E0

[0042] In the formula, I′0 represents the corrected white balance image.

[0043] The present invention also provides a semantically preserved color constancy system, comprising the following units:

[0044] The dataset acquisition unit is used to select the color constancy dataset and divide the dataset into training set, validation set and test set;

[0045] The model building unit is used to build the SPNet color constancy model, including the global illumination estimation network, the semantic constraint module SCM, and the auxiliary calibration module ACM.

[0046] The global illumination estimation network consists of two parts: a feature extractor and a light source estimation function. The image is first processed by the feature extractor to obtain an image feature map, and the feature map is then processed by the light source estimation function to predict the light source value.

[0047] The Semantic Constraint Module (SCM) for learning semantic knowledge consists of two branches: the upper branch is a pre-trained model based on ImageNet, and the lower branch is a feature extractor of the global illumination estimation network.

[0048] The Auxiliary Calibration Module (ACM), used to improve prediction accuracy, has the same network structure as the global illumination estimation network, and the two share parameters.

[0049] The model training unit is used for model training and stores the model that performs best on the validation set.

[0050] The correction unit is used for color cast correction. It inputs the color cast image into the trained model. The color cast image only needs to pass through the global illumination estimation network to obtain the light source prediction value, and then the color cast image is corrected by Von Kries transform.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] a) The model makes full use of the semantic knowledge in the pre-trained model, enabling the model to learn the color features required for the light source prediction task and the semantic features that have the function of assisting light source inference during the training process.

[0053] b) The model focuses on the semantic foreground while also paying attention to the background area with a limited range of inherent colors, making full use of the color information contained in the surfaces of different objects.

[0054] c) Consistency calibration makes the model's estimation results for each region more consistent, avoiding the problem of insufficient information and large bias in some regions, which leads to low accuracy of the final prediction results.

[0055] d) Enable the model to adapt to small-scale data through sufficient data augmentation. Attached Figure Description

[0056] Figure 1 This is the SPNet network structure used in this invention.

[0057] Figure 2 These are the attention heatmap results from the experimental examples of this invention. From left to right, they represent the model attention at different training epochs: the first column is 1000 epochs, the second is 2000 epochs, the third is 3000 epochs, the fourth is 4500 epochs, and the fifth is 6000 epochs. From top to bottom, they are the model attention heatmaps using only the global illumination estimation network, the model attention heatmap with the SCM module added, and the model attention heatmap with SPNet. It can be observed that after adding the SCM module, the model pays attention to the foreground object (car), while the complete SPNet not only pays attention to the foreground object but also to the background region (trees, sky) with a limited intrinsic color range.

[0058] Figure 3 This is a visualization result of an example experiment of the present invention. From left to right, it consists of the input image, ground truth, SPNet corrected image, ground truth image of the light source, and predicted ground truth image. The value in the upper right corner of the third column is the angular error.

[0059] Figure 4 This is a poorly performed correction result from an example experiment of this invention. From left to right, it shows the input image, the ground truth image, and the SPNet corrected image. The value in the upper right corner of the SPNet corrected image represents the angular error between the predicted result and the ground truth image. The color bands on the right side of the ground truth image and the SPNet corrected image represent the color of the light source. Detailed Implementation

[0060] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0061] This example uses the color checker dataset. This dataset consists of 568 linear RAW images captured by two cameras, including indoor and outdoor scenes. During image capture, a Macbeth Color Checker (MCC) was placed in each scene. The MCC consists of 24 small blocks, each representing the actual color and grayscale value of a natural object. The true lighting colors were obtained from the Macbeth Color Checker placed in the scene.

[0062] This invention provides a semantically preserved color constancy method, proposing an SPNet based on semantic constraints and auxiliary calibration. This enables the model to generate light source values ​​close to real illumination, achieving two-stage image color correction. The network structure is as follows: Figure 1 As shown. The model construction specifically includes the following steps:

[0063] Step 1: Set up a basic global illumination estimation network, including a feature extractor and a light source estimation function, to infer the light source values ​​of the image:

[0064] a. The feature extractor uses the feature extraction part of the lightweight network SqueezeNet, which ensures both model accuracy and inference speed. The light source estimation function consists of two conv-relu blocks with a dropout layer in between to ensure model robustness.

[0065] b. By constraining the light source prediction value through angular error loss, the model focuses on light source information, ensuring that the model prediction result is close to the true value.

[0066] c. Initialize the model parameters of the feature extraction part using a pre-trained model based on ImageNet to introduce semantic information, enabling the model to recognize scene content and assist the model in reasoning about light sources through semantic information.

[0067] Step 2: Set up a Semantic Constraint Module (SCM). The structure of this module is similar to knowledge distillation. By continuously learning semantic knowledge, it makes the model pay more attention to the foreground region and guides the model to learn the correspondence between objects and colors.

[0068] a) The upper branch uses the feature extraction part of SqueezeNet as a semantic feature extractor to semantically encode the input image and generate a semantic feature map, acting as the teacher network. This part freezes its parameters during training, and the model parameters are pre-trained model parameters based on the ImageNet dataset.

[0069] b, the lower branch is the feature extraction part of the global illumination estimation network, and this branch also generates feature maps.

[0070] c. Introduce a feature similarity loss, which constrains the similarity of the feature maps generated by the upper and lower branches. The lower branch achieves semantic knowledge transfer by fitting the feature encoding results of the pre-trained model. Since the feature extraction part of the lower branch is shared with the global illumination estimation network, the feature learning of the global illumination estimation network is guided by the pre-trained model, expanding its feature representation from a simple color feature space to a mixed feature space of color and semantics.

[0071] Step 3: Set up an Auxiliary Calibration Module (ACM). Since the semantic constraint module makes the model focus on the foreground region, some background regions containing important light source cues are ignored, such as regions with limited intrinsic color range. This module re-enhances the features of such regions and avoids potential large deviations in certain regions through consistency constraints.

[0072] a) The image is cropped into multiple patches and fed into the network along with the original image, enabling the model to observe both the global and local regions of the image simultaneously. The input of the local regions allows the model to learn the light source inference of the background region independently, avoiding interference from foreground objects and enhancing the features of the background region.

[0073] b. The network simultaneously obtains the light source prediction values ​​of both the patch and the original image, and uses consistency loss to narrow down these two types of prediction values, making the model more consistent with the single light source assumption.

[0074] Model training was implemented on the Ubuntu operating system, using the PyTorch deep learning framework to build the model and utilizing the GPU for computation. The specific hardware and software parameters used in this example are shown in the table below. The Adam optimizer was used to train the network, with β1 = 0.9 and β2 = 0.999, and other settings kept at PyTorch's default settings. The maximum number of training epochs was set to 6000, the batch size to 16, and the learning rate to 0.0003.

[0075]

[0076] During model training, the dataset is first divided, with 80% randomly selected as the training set and 20% as the validation set. Before training, the data is preprocessed. Color swatches for all images are occluded to prevent the model from learning light source inference through color swatches. Then, the input images are augmented using the aforementioned method. Finally, the preprocessed and augmented color-shifted images are input into the constructed model.

[0077] (1) In this example, the image is input into the global illumination estimation network to obtain a 16*16 feature map, representing 16*16 groups of light source prediction results. Each group of light source prediction results corresponds to 32*32 pixels in the original image. The final prediction result is obtained by averaging these results. The loss is calculated by comparing the final prediction result with the ground truth light source values:

[0078]

[0079] Where E represents the predicted value of the light source, E * This represents the true value of the light source.

[0080] When only the ground truth of the light source is used as a constraint, even if a pre-trained model is used as the initialization parameter, the model's focus on features will gradually shift from semantic features to light source features during training due to the significant differences in feature requirements between image classification and color constancy tasks. This leads to the model forgetting semantic knowledge and losing semantic guidance. For example... Figure 2 As shown in the first row, as training progresses, the model's attention shifts from the foreground object (bus) to the background region (road), which means that the model forgets semantic knowledge.

[0081] (2) In the SCM module of this example, the features extracted by the feature extraction part of the global illumination estimation network are constrained by feature similarity loss, so that it can retain semantic features as much as possible while learning the light source feature encoding, thus helping the model understand the image content. The specific approach is as follows:

[0082] The image is simultaneously input into a pre-trained model branch, where semantic feature maps are obtained through encoding. The L1 loss between the feature map obtained by the feature extraction part of the global illumination estimation network and the feature map obtained by the pre-trained model branch is calculated. The loss is optimized to narrow the gap between the two. The loss function is expressed as:

[0083] L s (F0,F1)=||F0-F1||1

[0084] In the formula, F0 and F1 represent the feature encoding results of the global illumination estimation network and the feature encoding results of the pre-trained model, respectively, and ||·||1 represents the L1 norm, i.e., the L1 distance.

[0085] like Figure 2 As shown in the second row, through the semantic constraints of the SCM module, the model continuously focuses on the foreground target (bus) during training, and the phenomenon of attention shift no longer occurs.

[0086] (3) The SCM module focuses the model on foreground objects, but some background areas also contain important light source cues. Directly ignoring background areas will limit the model's accuracy. In this example, the ACM module crops the image into small regions and inputs them into the network, allowing the model to independently observe each region and learn the color features of regions with limited intrinsic color without interference from foreground objects. At the same time, consistency constraints ensure a high degree of consistency between local and global prediction results, avoiding large prediction deviations in certain regions that affect the overall prediction accuracy. The specific approach is as follows:

[0087] The image is cropped using a 3x3 grid, resulting in nine 171x171 patches. Five patches are randomly selected from these nine and input into the global illumination estimation network along with the original image. Each of the five patches generates five sets of light source predictions. The angular error between these five sets of light source predictions and the illumination estimation results of the original image is calculated. The training objective of this module is to minimize the angular error between the two, i.e., to implement local-global consistency constraints. The loss function is expressed as:

[0088]

[0089] E s E and E represent the light source prediction values ​​for the 5 patches and the light source prediction value for the original image, respectively.

[0090] like Figure 2 As shown in the third row, after adding the ACM module, the model focuses not only on the foreground object (bus) but also on regions with finite intrinsic color (sky, trees). It also combines the foreground object and background objects containing light source cues for light source estimation.

[0091] This example uses three loss functions. To balance these three loss functions, weight hyperparameters are set, and the overall loss function is expressed as follows:

[0092] L SPCC =L main +αL s +βL c

[0093] When α is set to 1 and β is set to 2.2, the model loss is in equilibrium and performs optimally.

[0094] During training, the model is validated every 20 rounds. The average angular error of the model on the validation set is calculated, and the current test result is compared with the previous test result. If the current result is better than the previous result, the model is saved.

[0095] After the model is trained, its performance is tested. During training, the network learns color and semantic knowledge simultaneously through three parts: a basic global illumination estimation network for predicting light source values, and two modules for feature selection. During testing, only the global illumination estimation network is used to predict light source values.

[0096] During testing, no data augmentation was performed; only preprocessing was conducted. During preprocessing, the white card was occluded, and to optimize testing speed, the image's width and height were reduced to half the original image size. Testing included both objective and subjective evaluations.

[0097] Objective evaluation is achieved through index calculation. The image is processed by a global illumination estimation network to predict light sources, and the angular error is calculated by comparing these predictions with the true light source values. Evaluation indices include the average angular error of all predictions, the median angular error, the three-mean angular error, the average of the best 25% of angular errors, and the average of the worst 25% of angular errors.

[0098] Subjective evaluation involves observing the model's correction effect through the corrected image. The model is subjectively evaluated using the Von Kries transform to correct the image. Figure 3 The following are some test results for this example, showing results for different performance levels. The model's overall angular error is small, making it difficult for the human eye to distinguish between the model-corrected image and the ground truth image. Observation of the dataset reveals that although it is a single-light source color constancy dataset, it still includes some multi-light source images, and the model performs poorly in multi-light source images, such as... Figure 4 As shown, this is because the input image violates the single-source assumption.

[0099] This invention also provides a semantically preserved color constancy system, comprising the following units:

[0100] The dataset acquisition unit is used to select the color constancy dataset and divide the dataset into training set, validation set and test set;

[0101] The model building unit is used to build the SPNet color constancy model, including the global illumination estimation network, the semantic constraint module SCM, and the auxiliary calibration module ACM.

[0102] The global illumination estimation network consists of two parts: a feature extractor and a light source estimation function. The image is first processed by the feature extractor to obtain an image feature map, and the feature map is then processed by the light source estimation function to predict the light source value.

[0103] The Semantic Constraint Module (SCM) for learning semantic knowledge consists of two branches: the upper branch is a pre-trained model based on ImageNet, and the lower branch is a feature extractor of the global illumination estimation network.

[0104] The Auxiliary Calibration Module (ACM), used to improve prediction accuracy, has the same network structure as the global illumination estimation network, and the two share parameters.

[0105] The model training unit is used for model training and stores the model that performs best on the validation set.

[0106] The correction unit is used for color cast correction. It inputs the color cast image into the trained model. The color cast image only needs to pass through the global illumination estimation network to obtain the light source prediction value, and then the color cast image is corrected by Von Kries transform.

[0107] The specific implementation methods of each unit are the same as those of each step, and will not be described in this invention.

[0108] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A semantically preserved color constancy method, characterized in that, Includes the following steps: Step 1: Select the color constancy dataset and divide the dataset into training, validation, and test sets; Step 2: Construct the SPNet color constancy model, including a global illumination estimation network, a semantic constraint module (SCM), and an auxiliary calibration module (ACM). The global illumination estimation network consists of two parts: a feature extractor and a light source estimation function. The image is first processed by the feature extractor to obtain an image feature map, and the feature map is then processed by the light source estimation function to predict the light source value. The specific processing procedure of the global illumination estimation network is represented by the following formula: in, This represents a global illumination estimation network. This represents an RGB input image, where H and W represent the image's height and width. Indicates model parameters, Indicates the light source prediction result ,in, Indicates the red light component. Indicates the green light component. Indicates the blue light component; Specifically, it is expressed as follows: In the above formula, and These represent feature extractors. and light source estimation function Model parameters, The lightweight model SqueezeNet is used, and the initialization parameters for this part are based on the parameters of the pre-trained model based on ImageNet. It consists of two layers of conv-relu blocks; The loss function of the global illumination estimation network is expressed as the angular error between the predicted and true values ​​of the light source, and the specific formula is as follows: In the formula, Represents the true value of the light source. Indicates the light source prediction result; The Semantic Constraint Module (SCM) for learning semantic knowledge consists of two branches: the upper branch is a pre-trained model based on ImageNet, and the lower branch is a feature extractor of the global illumination estimation network. The Auxiliary Calibration Module (ACM), used to improve prediction accuracy, has the same network structure as the global illumination estimation network, and the two share parameters. Step 3: Model training, save the model that performs best on the validation set; Step 4, color cast correction: Input the color cast image into the trained model. The color cast image only needs to pass through the global illumination estimation network to obtain the light source prediction value, and then the color cast image is corrected by Von Kries transform.

2. The semantically preserved color constancy method according to claim 1, characterized in that: The specific processing procedure of the Semantic Constraint Module (SCM) is as follows: Input image Input pre-trained model Obtain semantic features The parameters of this branch are frozen; simultaneously, the input image... Input Feature Extractor Obtain feature map By using similarity loss to bring them closer and The distance between them makes the feature codes of the two branches closer. This process can be written as: in, express and The similarity loss, i.e. Distance, the loss function of SCM is defined as: 。 3. The semantically preserved color constancy method according to claim 1, characterized in that: The Auxiliary Calibration Module (ACM) will use the training samples The grid is uniformly cut into an m*m grid, and n patches are randomly selected from it, denoted as... , It includes two types of patches: patches with limited intrinsic color range. and patches with uncertain intrinsic color range The extracted n patches are input into the network along with the input image. Global illumination estimation network Obtain the predicted values ​​for each patch. , According to the single-source assumption, theoretically Predicted values ​​for each patch Should be consistent with overall forecast results There is a strong consistency, therefore, by bringing them closer... and Consistency supervision is achieved by using the angular error between the parameters. The process is represented as: In the formula, express and The angular error between them is defined as: 。 4. The semantically preserved color constancy method according to claim 1, characterized in that: During training, the training images are first augmented with data, including random cropping, random rotation, and light source scaling. First, a small square region is cropped from the original input image, and the cropped image is randomly rotated. Then, these images are scaled and finally randomly flipped horizontally with a certain probability. In order to increase the diversity of data, the light source labels in each image are also randomly scaled. At the same time, the pixel colors of the original image are also scaled in accordance with the light source. Furthermore, gamma correction is applied to convert the linear image into a non-linear image and the image values ​​are normalized to [0,1]. The enhanced image is input into the constructed SPNet to generate feature maps, global light source predictions, and light source predictions for each patch. The loss is calculated using a loss function, and backpropagation is performed. The model performance is validated on the validation set after several training rounds, and the model with the smallest average angular error is saved.

5. The semantically preserved color constancy method according to claim 4, characterized in that: The loss function of the SPNet color constancy model includes three types: the light source prediction loss of the illumination estimation network. Feature similarity loss of the semantic constraint module (SCM) Consistency loss with the auxiliary calibration module (ACM) The final loss is expressed as: In the formula, and These are weight parameters.

6. The semantically preserved color constancy method according to claim 1, characterized in that: The Von Kries transform is expressed as: In the formula, This represents the corrected white balance image. This indicates the predicted result for the light source.

7. A semantically preserved color constancy system for implementing the semantically preserved color constancy method as described in any one of claims 1-6, characterized in that, Includes the following units: The dataset acquisition unit is used to select the color constancy dataset and divide the dataset into training set, validation set and test set; The model building unit is used to build the SPNet color constancy model, including the global illumination estimation network, the semantic constraint module SCM, and the auxiliary calibration module ACM. The global illumination estimation network consists of two parts: a feature extractor and a light source estimation function. The image is first processed by the feature extractor to obtain an image feature map, and the feature map is then processed by the light source estimation function to predict the light source value. The Semantic Constraint Module (SCM) for learning semantic knowledge consists of two branches: the upper branch is a pre-trained model based on ImageNet, and the lower branch is a feature extractor of the global illumination estimation network. The Auxiliary Calibration Module (ACM), used to improve prediction accuracy, has the same network structure as the global illumination estimation network, and the two share parameters. The model training unit is used for model training and stores the model that performs best on the validation set. The correction unit is used for color cast correction. It inputs the color cast image into the trained model. The color cast image only needs to pass through the global illumination estimation network to obtain the light source prediction value, and then the color cast image is corrected by Von Kries transform.

Images