Multi-optical AWB system and method based on local affine transformation

Abstract

The invention belongs to the technical field of color constancy in computer vision, and discloses a multi-light AWB system and method based on local affine transformation. By designing a multi-spectral image assisted RGB illumination estimation clue acquisition module, more illumination information is extracted from an RGB image and a multi-spectral image to realize more accurate illumination estimation, color correction for the RGB image is realized by using color context information in the RGB image, and the process of converting from a multi-spectral domain to an RGB domain is avoided. A multi-scale model is adopted to learn joint distribution of chromaticity and color context features through a multi-scale fusion module, an affine bilateral grid is predicted based on the joint distribution, and white balance without alignment semantic representation is carried out on an RGB image by using illumination color information predicted by a multi-spectral image. The invention further provides a multi-light AWB method, the color constancy of the color cast image under the multi-light-source condition can be effectively achieved, and the more accurate white balance effect is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of color constancy in computer vision, and in particular relates to a multi-light AWB system and method based on local affine transformation. Background Art

[0002] Computational color constancy is a crucial component of the image processing pipeline. It simulates the human visual system's ability to maintain color (perceive an object's color as constant even when scene lighting changes). This process, also known as white balancing, is the first step in the camera image processing pipeline. In other words, the purpose of calculating color constancy is to endow photographic equipment with color constancy, eliminating color casts caused by ambient lighting.

[0003] In order to achieve color constancy and eliminate color casts in images, researchers have proposed many color constancy methods. In the current field of multi-light color constancy technology, only RGB images are usually used for illumination estimation. Since RGB images only have three channels and can only capture the visible light range, there is a problem of insufficient illumination information. To solve this problem, a few methods have emerged that use multispectral images with rich illumination information. However, since it is still the RGB images that ultimately need color correction, existing methods that only use multispectral images for illumination estimation will inevitably involve conversion from the multispectral domain to the RGB domain. This conversion suffers from the problem of loss of illumination information. In order to use multispectral and RGB images simultaneously, there is still the problem that the background content of the two cannot be completely aligned. This research is also a current research hotspot and difficulty.

[0004] In summary, the multi-light color constancy algorithms using multispectral images in the existing technology still need to be improved and perfected. Summary of the Invention

[0005] In response to the deficiencies in the prior art, the present invention proposes a multi-light AWB system and method based on local affine transformation. Through the multi-spectral image-assisted RGB acquisition illumination estimation clue module in the multi-light AWB system, more illumination information is extracted from the RGB image and the multi-spectral image to achieve more accurate illumination estimation. At the same time, the color context information in the RGB image is used to achieve color correction for the RGB image, avoiding the process of converting from the multi-spectral domain to the RGB domain. The multi-scale fusion module uses a multi-scale model to learn the joint distribution of chromaticity and color context features, and predicts an affine bilateral grid based on the joint distribution, so that the model can use the illumination color information predicted by the multi-spectral image to perform white balance on the RGB image without aligning the semantic representation. The present invention also proposes a multi-light AWB method, which can effectively achieve color constancy for color-biased images under multi-light source conditions and achieve a more accurate white balance effect.

[0006] The technical solutions of the present invention are as follows:

[0007] A multi-light AWB system based on local affine transformation includes a multi-spectral image-assisted RGB illumination estimation clue module and a multi-scale fusion module, wherein:

[0008] The multispectral image-assisted RGB illumination estimation clue module includes a depthwise convolution block, a pointwise convolution block, a color space conversion module, an AWBU-Net convolutional neural network, three sputtering blocks, three 3×3 convolutional layers, and two 1×1 convolutional layers.

[0009] The multi-scale fusion module includes a local color feature extraction submodule and an illumination information feature extraction submodule.

[0010] Preferably, the depthwise convolution block is used to extract color context features from the RGB image.

[0011] Preferably, the pointwise convolution block is used to extract chromaticity information from a multispectral image.

[0012] Preferably, the color space conversion module is used to convert the RGB image from the RGB color space to the Loguv color space to extract color information.

[0013] Preferably, the AWBU-Net convolutional neural network is a UNet network, and a ReLU activation function and batch normalization are added after each convolution layer of the encoder.

[0014] Preferably, the AWBU-Net convolutional neural network is used to obtain low-level features based on color context features and chromaticity features, wherein the chromaticity features are composed of color information and chromaticity information.

[0015] Preferably, the illumination information feature extraction submodule includes 1 convolutional layer and a fully connected layer.

[0016] Preferably, the multi-scale fusion module extracts local color features and illumination information features from low-level features, thereby obtaining an affine bilateral grid.

[0017] A multi-light AWB method based on local affine transformation is implemented using the multi-light AWB system, comprising the following steps:

[0018] Step 1: Collect multiple multi-light RGB images and multispectral images from different scenes and lighting conditions. From all single-light source images corresponding to the multi-light RGB images, extract the color value of the ColorChecker gray block in each single-light source image as the illumination value of the single-light source image. Use the G channel value of each pixel in each single-light source image as the calculation weight. Multiply the illumination value of each single-light source image by the calculation weight of the corresponding pixel to obtain the true illumination value. Divide the multi-light RGB image by the true illumination value to obtain a true white balance image. The multi-light RGB images, multispectral images, and true white balance images constitute a dataset.

[0019] Step 2: training a multispectral image-assisted RGB illumination estimation clue module and a multi-scale fusion module in the multi-light AWB system based on the data set;

[0020] Step 3: Input the single RGB image to be processed into the multi-light AWB system, and obtain the affine bilateral grid through the multi-spectral image-assisted RGB illumination estimation clue module and the multi-scale fusion module;

[0021] Step 4: The single RGB image to be processed is processed by the affine bilateral grid to obtain a white-balanced RGB image.

[0022] Preferably, the step 2 specifically includes:

[0023] Step 2-1: Input the original-size multispectral image and the corresponding RGB image into the multispectral image auxiliary RGB acquisition illumination estimation clue module to extract low-level features at four scales;

[0024] Step 2-2: Input the extracted low-level features of the four scales into the multi-scale fusion module to generate an affine bilateral grid;

[0025] Step 2-3: Input the original size RGB image into the affine bilateral grid to obtain the predicted white balance image;

[0026] Step 2-4: Divide the predicted white balance image and the true white balance image by the original RGB image to obtain the predicted illumination map and the true illumination map respectively, calculate the angular error between the predicted illumination map and the true illumination map, and backpropagate to update the parameters;

[0027] Step 2-5: Return to step S2-1 until the loss function stops decreasing, and the trained multispectral image-assisted RGB illumination estimation clue module and multi-scale fusion module are obtained.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The multi-light AWB system based on local affine transformation proposed in this paper adopts multispectral images to assist RGB in obtaining illumination estimation clues. While utilizing the richer illumination and color information in multispectral images for illumination estimation, it also uses the color context features in RGB images to achieve color correction for RGB images, thus avoiding the problem of information loss in the conversion from multispectral domain to RGB domain.

[0030] 2. The multi-light AWB system based on local affine transformation proposed in this paper uses a multi-scale fusion module to learn the joint distribution of low-level features and predict an affine bilateral grid. This allows the multi-scale fusion module to use the illumination color information predicted by multispectral images to achieve color constancy in RGB images without the need for alignment of semantic representations. This solves the problem that the illumination map predicted using multispectral images cannot correspond to the correct position of the RGB image pixel by pixel due to the misalignment of the background content of the multispectral and RGB images.

[0031] 3. The multi-light AWB method based on local affine transformation proposed in the present invention can effectively achieve color constancy for color-shifted images under multi-light source conditions based on the multi-light AWB system, achieving a more accurate white balance effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention can be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 4 is an architecture diagram of the multi-light AWB system based on local affine transformation of the present invention.

[0034] Figure 2 4 is a flow chart of the multi-light AWB method based on local affine transformation of the present invention.

[0035] Figure 3 This is a workflow diagram of the multi-spectral assisted RGB illumination estimation clue module of the present invention.

[0036] Figure 4 This is a workflow diagram of the multi-scale fusion module of the present invention.

[0037] Figure 5 This is a flowchart of the multi-light AWB method based on local affine transformation in the application scenario of Example 1. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0040] To address the problem of insufficient illumination information extracted using only RGB images, the present invention introduces multispectral images based on RGB images. To further address the issue of information loss during conversion from the multispectral domain to the RGB domain, the present invention designs a multispectral image-assisted RGB illumination estimation clue module, enabling the network to extract more illumination information for more accurate illumination estimation. At the same time, the color context features in the RGB image are used to perform color correction on the RGB image, avoiding the process of converting from the multispectral domain to the RGB domain. To address the problem that the background content of the multispectral and RGB images cannot be fully aligned, the present invention designs a multiscale fusion module that uses a multiscale model to learn the joint distribution of chromaticity features and color context features. Based on this joint distribution, an affine bilateral grid is predicted, allowing the model to use the illumination color information predicted by the multispectral image to perform white balance on the RGB image without the need for alignment of semantic representations.

[0041] like Figure 1 As shown, the multi-light AWB system based on local affine transformation proposed in the present invention includes a multi-spectral image-assisted RGB illumination estimation clue acquisition module, a multi-scale fusion module and an affine bilateral grid generation module.

[0042] like Figure 2 As shown, the overall processing flow of the multi-light AWB system is as follows:

[0043] S1, the preprocessed RGB image I RGB and multispectral image I spec is the input image;

[0044] S2, RGB image I RGB and multispectral image I spec The illumination estimation clue module is obtained by multispectral assisted RGB, aiming to obtain low-level features of the four scales of the two images. It can simultaneously utilize the color context features of the RGB image and the chromaticity features with richer illumination information in the multispectral image;

[0045] S3. Use multi-spectral assisted RGB to obtain low-level features of four scales obtained by the illumination estimation clue module, and learn the joint distribution of these low-level features of four scales through the multi-scale fusion module to predict an affine bilateral grid.

[0046] S4, RGB image I RGB Input the affine bilateral grid to obtain the predicted white balance RGB image I p red .

[0047] The multispectral image-assisted RGB illumination estimation clue module specifically includes 1 depthwise convolution block, 1 pointwise convolution block, a color space conversion module, 1 AWBU-Net convolutional neural network, 3 sputtering blocks, 3 3×3 convolution layers and 2 1×1 convolution layers. Figure 3 The following figure shows the workflow of each part in the multispectral image assisted RGB acquisition illumination estimation clue module:

[0048] S2-1. Input RGB image I RGB Extract the RGB image I through the depthwise convolution block RGB Color context features;

[0049] S2-2, input RGB image I RGB The color space conversion module converts the color information into the Loguv color space.

[0050] S2-3. Input multispectral image I spec The chromaticity information is extracted through the pointwise convolution block;

[0051] S2-4, combining the color information with the chromaticity feature to obtain the chromaticity feature;

[0052] S2-5, input the color context features and chromaticity features into the AWBU-Net convolutional neural network to obtain low-level features;

[0053] S2-6. The low-level features are processed through three sputtering blocks, three 3×3 convolutional layers, and two convolutions to obtain low-level features of four scales.

[0054] The multi-scale fusion module includes a local color feature extraction submodule and a lighting information feature extraction submodule. Its workflow is as follows: Figure 4 shown.

[0055] S3-1. Input the four-scale low-level features obtained by the multi-spectral assisted RGB illumination estimation clue module into the multi-scale fusion module. The four-scale low-level features are passed through the fully convolutional local branch (i.e., the local color feature extraction submodule) for learning local color transformations, which are usually content-related. The four-scale low-level features are passed through the global branch (i.e., the illumination feature extraction submodule) for learning illumination information features, which includes a convolutional layer and a fully connected layer.

[0056] S3-2. The multi-scale fusion module predicts an affine bilateral grid.

[0057] Example 1

[0058] In combination with application scenarios, the multi-light AWB method based on the multi-light AWB system of the present invention is described in detail.

[0059] The proposed multi-light AWB method first processes the original-size color-cast multi-light RAW-RGB image and the color-cast multispectral image through a multispectral image-assisted RGB illumination estimation clue module and a multi-scale fusion module. This method obtains more accurate illumination estimation results on the original-size color-cast multi-light RAW-RGB image and obtains a predicted white balance image that is closer to the real image for further application in downstream tasks.

[0060] Step 1: Input color-cast multi-light RAW-RGB image I RGB , color cast multispectral image I spec , true white balance image GT.

[0061] Color cast multi-light RAW-RGB image I RGB , color cast multispectral image I spec The size of the true white balance image GT can be expressed as C×H×W, where C represents the number of channels of the image, H is the height of the image, which can be expressed by the number of pixels of the image in the vertical dimension, and W is the width of the image, which can be expressed by the number of pixels of the image in the horizontal dimension.

[0062] In this embodiment, the color-cast multi-light RAW-RGB image I RGB The size of the true white balance image GT is 3×512×512, which means that the image has three channels: red (Red, R), green (Green, G), and blue (Blue, B), with a height of 512 pixels (pixel, px) and a width of 512px. spec The size of is 9×512×512, which means it has 9 different spectral response bands.

[0063] Step 2: Color-cast multi-light RAW-RGB image I RGB Multispectral image with color cast I spec, assuming that the number of a batch image is 2, the color cast multi-light RAW-RGB image I RGB The scale is 2×3×512×512, the color cast multispectral image I spec The scale is 2×9×512×512. Figure 5 As shown in the figure, the original size 9-channel color-cast multispectral image and the original size 3-channel color-cast multi-light RAW-RGB image are first taken as two modal inputs. Since the key feature of extracting illumination color information lies in the chromaticity feature, the pointwise convolution block is combined with the Loguv of the corresponding color-cast multi-light RAW-RGB image to extract the chromaticity information in the color-cast multispectral image. Among them C out Refers to the number of characteristic channels of the output chromaticity information after combination. Assume that the color cast multispectral image is The output tensor of the pointwise convolution block operation represents the chrominance correlation of each pixel. The specific implementation form is shown in the following formula.

[0064]

[0065] Among them, C in represents the number of channels in the input color-shifted multispectral image, X1(i,j,c) represents the value of the cth channel at position (i,j) in the color-shifted multispectral image, W(1,1,c,k) represents the learned weights connecting the convolution kernels of the cth and kth channels, and Y(i,j,k) represents the output chromaticity feature tensor of the kth channel at position (i,j). The function σ(·) is the ReLU nonlinear activation function, and a total of n repeated chromaticity extraction units are used.

[0066] In order to obtain the 3-channel color-cast multi-light RAW-RGB image corresponding to the color-cast multispectral image Extract chromaticity-related color information First, the color cast multi-light RAW-RGB image is converted to the Loguv color space, and the (I r ,I g ,I b ) into the Loguv color space (I u ,I v ), the specific conversion process is shown in the following formula.

[0067] I u =log(I g / I r );

[0068] I v =log(I g / I b );

[0069] After obtaining these two pieces of information, we need to combine the chromaticity information Y extracted by the pointwise convolution block with the color information Z obtained by the Loguv transform. For this purpose, we have designed an adaptive weighted fusion mechanism to ensure that the two complement each other to enhance the lighting estimation effect. To enable the network to automatically learn how to appropriately combine Y and Z in different scenarios, we introduce two learnable adaptive weights α and β. This allows the final fused chromaticity feature F to be adaptively adjusted according to different lighting conditions. The specific process is shown in the following formula.

[0070]

[0071] α and β are normalized using softmax to ensure that the sum of their weights is 1, thus achieving dynamic fusion.

[0072] After obtaining the fused chromaticity feature F, the AWBU-Net convolutional neural network uses this feature as input to extract low-level features from the two images. Specifically, the AWBU-Net convolutional neural network includes an encoder for feature extraction. This encoder adds a ReLU activation function and batch normalization after each convolution layer. ReLU introduces nonlinear transformations, enabling the network to capture more complex and richer features, avoiding the vanishing gradient problem, and thus improving the network's ability to handle complex lighting conditions.

[0073] The low-level features extracted by the AWBU-Net convolutional neural network are processed through three sputtering blocks, three 3×3 convolutional layers, and two convolutions to obtain low-level features of four scales.

[0074] Step 3: Splatting blocks are used to process the low-level features at the four scales. Starting with the finest features, convolutions with stride 2 are performed on the chromaticity and color context features, with the number of channels doubled. This weighted convolution enables subsequent layers to learn the joint distribution of chromaticity and color context without supervision. The resulting color context features are then added to the chromaticity-aligned features, and convolutions with stride 1 are then used to reduce the dimensionality and select meaningful features.

[0075] Because multispectral images and RGB images are not aligned, based on the aligned chromaticity-color context features in the bilateral space, an affine bilateral grid is learned through the network, which encodes the local transformation at the semantic level and divides the entire network into two asymmetric branches, one is the fully convolutional local branch for learning local color transformation (i.e., local color feature extraction submodule), and the other is the global branch containing convolution and fully connected layers (i.e., illumination information feature extraction submodule), which helps to perform spatial domain regularization of the transformation. Specifically, Figure 1 As shown in the figure, the local color feature extraction submodule L is responsible for extracting local features of the image, which are usually related to the content. These features are combined with the illumination information features generated by the illumination information feature extraction submodule G to generate the final affine bilateral grid Γ. The resulting affine bilateral grid Γ encodes the local transformation at the semantic level, so that the illumination color information predicted from the multispectral image can be used to white balance the RGB image without aligning the semantic representation.

[0076] Step 4: Repeat steps 1-3 above to train the multispectral image-assisted RGB illumination estimation clue module and the multi-scale fusion module. When the loss functions no longer decrease and tend to be stable, stop training.

[0077] Step 5: The single multi-light color cast RAW image to be processed RGB , the size is 3×512×512, and it is sequentially processed through the multi-spectral image assisted RGB to obtain the illumination estimation clue module and multi-scale fusion to obtain the affine bilateral grid, and then the single multi-light color cast RAW image I RGB Input an affine bilateral grid to obtain a white-balanced RGB image with a size of 3×512×512.

[0078] In a public dataset, the accuracy of the multi-light AWB method of the present invention is compared with the existing technology. The results are shown in Table 1. The present invention can effectively improve the accuracy of illumination estimation in multi-light source scenes.

[0079] Table 1

[0080] method average value Median Worst 25% The best 25% SpecUNet 2.514 2.702 10.872 2.083 SpecSlot 1.767 1.832 9.061 1.547 The present invention 1.012 1.037 7.683 1.003

[0081] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0082] In the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Furthermore, a first feature being "above," "above," and "above" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may include the first feature being directly below or obliquely below the second feature, or may simply mean that the first feature is lower in level than the second feature.

[0083] In the present invention, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0084] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multi-light AWB system based on local affine transformation, characterized in that: It includes a multi-spectral image-assisted RGB illumination estimation clue module and a multi-scale fusion module, among which, The multispectral image-assisted RGB illumination estimation clue module includes a depthwise convolution block, a pointwise convolution block, a color space conversion module, an AWBU-Net convolutional neural network, three sputtering blocks, three 3×3 convolutional layers and two 1×1 convolutional layers; The multi-scale fusion module includes a local color feature extraction submodule and an illumination information feature extraction submodule.

2. The multi-light AWB system according to claim 1, characterized in that: The depthwise convolution block is used to extract color context features from the RGB image.

3. The multi-light AWB system according to claim 2, wherein: The pointwise convolution block is used to extract chromaticity information from multispectral images.

4. The multi-light AWB system according to claim 3, wherein: The color space conversion module is used to convert the image from the RGB color space to the Loguv color space to extract color information.

5. The multi-light AWB system according to claim 1, characterized in that: The AWBU-Net convolutional neural network is a UNet network, and ReLU activation function and batch normalization are added after each convolution layer of the encoder.

6. The multi-light AWB system according to claim 4, characterized in that: The AWBU-Net convolutional neural network is used to obtain low-level features based on color context features and chromaticity features, where the chromaticity features are composed of color information and chromaticity information.

7. The multi-light AWB system according to claim 1, wherein: The illumination information feature extraction submodule includes a convolutional layer and a fully connected layer.

8. The multi-light AWB system according to claim 6, wherein: The multi-scale fusion module extracts local color features and illumination information features from low-level features, thereby obtaining an affine bilateral grid.

9. A multi-light AWB method based on local affine transformation, characterized in that: The method is implemented by using the multi-light AWB system according to any one of claims 1 to 8, comprising the following steps: Step 1: Collect multiple multi-light RGB images and multispectral images from different scenes and lighting conditions. From all single-light source images corresponding to the multi-light RGB images, extract the color value of the ColorChecker gray block in each single-light source image as the illumination value of the single-light source image. Use the G channel value of each pixel in each single-light source image as the calculation weight. Multiply the illumination value of each single-light source image by the calculation weight of the corresponding pixel to obtain the true illumination value. Divide the multi-light RGB image by the true illumination value to obtain a true white balance image. The multi-light RGB images, multispectral images, and true white balance images constitute a dataset. Step 2: training a multispectral image-assisted RGB illumination estimation clue module and a multi-scale fusion module in the multi-light AWB system based on the data set; Step 3: Input the single RGB image to be processed into the multi-light AWB system, and obtain the affine bilateral grid through the multi-spectral image-assisted RGB illumination estimation clue module and the multi-scale fusion module; Step 4: The single RGB image to be processed is processed by the affine bilateral grid to obtain a white-balanced RGB image.

10. The multi-light AWB method according to claim 9, wherein: The step 2 specifically includes: Step 2-1: Input the original-size multispectral image and the corresponding RGB image into the multispectral image auxiliary RGB acquisition illumination estimation clue module to extract low-level features at four scales; Step 2-2: Input the extracted low-level features of the four scales into the multi-scale fusion module to generate an affine bilateral grid; Step 2-3: Input the original size RGB image into the affine bilateral grid to obtain the predicted white balance image; Step 2-4: Divide the predicted white balance image and the true white balance image by the original RGB image to obtain the predicted illumination map and the true illumination map respectively, calculate the angular error between the predicted illumination map and the true illumination map, and backpropagate to update the parameters; Step 2-5: Return to step S2-1 until the loss function stops decreasing, and the trained multispectral image-assisted RGB illumination estimation clue module and multi-scale fusion module are obtained.

Images