Spectral imaging chip-based complex light source partition white balance method, system and storage medium

Abstract

The present application is suitable for the technical field of computer photography and image processing, and particularly relates to a complex light source partition white balance method and system based on a spectral imaging chip and a storage medium. The method predicts a pixel-level normalized white point graph through a light-weight model, aggregates the white point coordinates in space to generate a white balance gain, and corrects the overall multi-channel spectral image, thereby achieving accurate estimation and stable correction of the white point in a complex scene such as a mixed light source, effectively avoiding color offset stitching artifacts caused by traditional independent correction, and significantly improving the authenticity and consistency of color restoration in imaging.

Description

Technical Field

[0001] This invention relates to the fields of computational photography and image processing technology, and particularly to a method, system and storage medium for complex light source zoning white balance based on a spectral imaging chip. Background Technology

[0002] White balance technology is a core component in digital imaging systems for eliminating color casts from light sources and restoring the true colors of objects. With the widespread adoption of multispectral imaging chips, using high-dimensional spectral information to replace the traditional RGB three-channel white balance correction has become an important direction for improving color fidelity. This approach can capture richer spectral characteristics of light sources, providing a physical basis for accurate color reproduction in complex lighting scenarios.

[0003] Existing methods generally use deep learning models to process multispectral images to achieve white balance correction. One type of method uses encoder-decoder architectures such as U-Net to reconstruct the corrected RGB image, fuses multi-scale features through skip connections, and uses paired datasets for supervised training. Another type of method uses lightweight backbone networks such as MobileNetV2 to directly regress the global white balance gain, combines dual-source synthetic samples with angle error loss function optimization, and introduces channel attention mechanism to adapt to mobile deployment. Some other methods rely on discrete color temperature classification or lookup table mapping, quantizing the continuous spectral space into a finite number of categories and then matching preset gain parameters.

[0004] However, none of the above methods can solve the problem of balancing accurate prediction of continuous color temperature and computational efficiency under mixed light sources. This is because the reconstruction paradigm requires the complete recovery of the RGB image and then indirect derivation of the white point, introducing a large amount of redundant information unrelated to white balance, resulting in high inference latency and susceptibility to texture noise. While global regression methods reduce computation, they cannot characterize the differences in light sources in different areas of the image, leading to local color shifts in partitioned lighting scenes. Discrete classification methods, on the other hand, suffer from discontinuous color temperature estimation due to quantization intervals, failing to cover arbitrary spectral combinations in real-world environments.

[0005] Therefore, a new partitioned white balance method is urgently needed to solve the above-mentioned technical problems. Summary of the Invention

[0006] This invention provides a method, system, and storage medium for white balance of complex light sources based on a spectral imaging chip, aiming to solve the problem that existing multispectral imaging white balance methods cannot balance accuracy and computational efficiency.

[0007] To address the aforementioned technical problems, in a first aspect, the present invention provides a method for white balance of complex light sources based on a spectral imaging chip, comprising the following steps:

[0008] S101. Acquire the multi-channel spectral image output by the multispectral imaging chip; S102. Input the multi-channel spectral image into the pre-trained lightweight U-Net model, extract the spatial-spectral joint features step by step through the encoder, and fuse multi-scale information through the decoder and cross-layer connection to obtain a pixel-level white point map through regression prediction; wherein, each spatial position in the pixel-level white point map has a corresponding normalized white point coordinate, and the normalized white point coordinate is used to reflect the light source color temperature characteristics of the corresponding spatial position. S103. Spatial partitioning and aggregation are performed on the normalized white point coordinates in the pixel-level white point image to obtain the representative white point coordinates corresponding to each partition. S104. Generate the partition white balance gain corresponding to the partition based on the coordinates of the representative white point; S105. The multi-channel spectral image is white-balanced using the partitioned white-balance gain to obtain a white-balanced corrected image.

[0009] Furthermore, the acquisition of the multi-channel spectral image output by the multispectral imaging chip in step S101 includes the following sub-steps: S1011. Obtain the raw RAW data output by the multispectral imaging chip, and perform dark current subtraction and bad pixel correction on each channel in the raw RAW data to obtain corrected RAW data. S1012. Radiometric calibration of the corrected RAW data is performed using a spectral response matrix pre-calibrated with a standard light source, converting the digital quantization value into an absolute spectral radiance value. S1013. Spatially align and resample the absolute spectral radiance values ​​to generate the multi-channel spectral image with uniform spatial resolution.

[0010] Furthermore, the lightweight U-Net model in step S102 includes a spectral channel preprocessing layer, an encoder, and a decoder, wherein: The spectral channel preprocessing layer is located at the front end of the encoder and is used to rearrange the irregularly arranged multispectral raw data in the multichannel spectral image into a uniform tensor format. The encoder consists of a multi-level cascaded depthwise separable convolutional module and a downsampling layer. The kernel size and number of groups of the depthwise separable convolutional module are set according to the number of channels of the multi-channel spectral image. The depthwise separable convolutional module is used to work with the downsampling layer to extract the low-resolution feature map and spatial-spectral joint features of the input multi-channel spectral image step by step. The decoder consists of multiple upsampling layers, skip connection structures, and an output convolutional layer. The upsampling layer employs a transposed convolution operation to restore the low-resolution feature map to the same spatial resolution as the feature map output by the corresponding level of the encoder. The skip connection structure aligns the spatial-spectral joint features output by the corresponding level of the encoder in the spectral dimension and then concatenates and fuses them with the feature map output by the upsampling layer to obtain the pixel-level white dot map. The output convolutional layer has a 1×1 convolutional kernel, and its number of output channels is equal to the vector dimension of the normalized white dot coordinates, used to map the feature channel number of the pixel-level white dot map to the dimension of the normalized white dot coordinates.

[0011] Furthermore, the normalized white point coordinates in step S102 are two-dimensional coordinate vectors obtained by normalizing the original white point estimates of the spatial positions in the pixel-level white point image.

[0012] Furthermore, step S103, which involves spatially partitioning and aggregating the normalized white point coordinates to obtain representative white point coordinates for each partition, includes the following sub-steps: S1031. Divide the pixel-level white dot image into several spatial partitions according to a preset grid size; S1032. Within each spatial partition, the normalized white point coordinates are weighted according to a preset pixel confidence weight. Abnormal pixels that deviate from the partition mean by more than a preset threshold are removed. The normalized white point coordinates are then recalculated, and the recalculated normalized white point coordinates are used as the representative white point coordinates of the spatial partition.

[0013] Furthermore, in step S104, generating the partition white balance gain corresponding to the partition based on the representative white point coordinates specifically involves: By using a preset white point-gain mapping relationship, the coordinates of the representative white point are converted into R / G gain ratio and B / G gain ratio, which are used as the white balance gain of the partition.

[0014] Furthermore, step S105, which involves using the partitioned white balance gain to perform white balance correction on the multi-channel spectral image to obtain a white balance corrected image, includes the following sub-steps: S1051. Upsample the white balance gain of each partition to the same spatial resolution as the multi-channel spectral image through bilinear interpolation or spline interpolation to generate a pixel-level gain map. S1052. Multiply the pixel-level gain map element-wise with the multi-channel spectral image to obtain the white balance correction image.

[0015] Secondly, the present invention also provides a complex light source zoning white balance system based on a spectral imaging chip, comprising: The data acquisition module is used to acquire multi-channel spectral images output by the multispectral imaging chip; The feature extraction module is used to input the multi-channel spectral image into a pre-trained lightweight U-Net model, extract spatial-spectral joint features step by step through the encoder, and fuse multi-scale information through the decoder and cross-layer connection to obtain a pixel-level white point map through regression prediction; wherein, each spatial position in the pixel-level white point map has a corresponding normalized white point coordinate, and the normalized white point coordinate is used to reflect the light source color temperature characteristics of the corresponding spatial position. The partitioning aggregation module is used to perform spatial partitioning aggregation on the normalized white point coordinates in the pixel-level white point image to obtain the representative white point coordinates corresponding to each partition. The partition gain module is used to generate the partition white balance gain corresponding to the partition based on the coordinates of the representative white point. The correction module is used to perform white balance correction on the multi-channel spectral image using the partitioned white balance gain to obtain a white balance corrected image.

[0016] Thirdly, the present invention also provides a computer device, comprising: a memory, a processor, and a complex light source partition white balance program based on a spectral imaging chip stored in the memory and executable on the processor, wherein when the processor executes the complex light source partition white balance program based on a spectral imaging chip, it implements the steps in the complex light source partition white balance method based on a spectral imaging chip as described in any of the above embodiments.

[0017] Fourthly, the present invention also provides a storage medium storing a complex light source partition white balance program based on a spectral imaging chip, wherein when the complex light source partition white balance program based on a spectral imaging chip is executed by a processor, the program implements the steps of the complex light source partition white balance method based on a spectral imaging chip as described in any of the above embodiments.

[0018] The beneficial effects achieved by this invention are that it provides a white balance method for complex light source partitioning based on a spectral imaging chip. This method predicts and regresses a pixel-level normalized white point map through a lightweight model, and performs spatial partitioning and aggregation of the white point coordinates to generate white balance gain. It also performs overall correction on multi-channel spectral images, achieving accurate estimation and stable correction of white points in complex scenes such as mixed light sources. This effectively avoids color shift stitching artifacts caused by traditional independent partitioning correction, and significantly improves the realism and consistency of image color reproduction. Attached Figure Description

[0019] The present invention will now be described in detail with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and more readily understood through the detailed description following the accompanying drawings. In the drawings: Figure 1 This is a flowchart of the steps of the complex light source partitioning white balance method based on a spectral imaging chip provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the structure of a complex light source partitioning white balance system based on a spectral imaging chip provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] The specific embodiments / examples described herein are specific implementations of the present invention, used to illustrate the concept of the invention, and are illustrative and exemplary, and should not be construed as limiting the implementation methods or scope of the present invention. In addition to the embodiments described herein, those skilled in the art can employ other obvious technical solutions based on the content disclosed in the claims and specification of this application. These technical solutions include those that make any obvious substitutions and modifications to the embodiments described herein, all of which are within the protection scope of the present invention.

[0022] Example 1 Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a complex light source zoning white balance method based on a spectral imaging chip provided in this invention. The complex light source zoning white balance method based on a spectral imaging chip includes the following steps: S101. Acquire the multi-channel spectral image output by the multispectral imaging chip.

[0023] The present invention aims to solve the color shift problem caused by the failure of global assumptions in existing white balance technology in complex mixed light source scenarios. In particular, for high-dimensional spectral data acquired by multispectral imaging chips, a complete technical solution is proposed that can make full use of spectral dimension information and achieve pixel-level accurate white point estimation and zone adaptive correction.

[0024] The multispectral imaging chip refers to an image sensor that integrates multiple narrowband filters or employs a beam-splitting structure. Its output raw data is not in the traditional RGB three-channel format, but rather contains irregularly arranged data with four or more discrete spectral bands. Each pixel location records only the light intensity value of a single band, requiring spatial-spectral joint processing to reconstruct a complete multi-channel spectral image. Because the spectral response characteristics, spatial sampling modes, and noise distribution of such chips differ fundamentally from those of traditional visible light cameras, directly applying existing white balance algorithms not only fails to leverage their spectral resolution advantages but also introduces systematic errors due to neglecting the physical correlation between channels. Therefore, this invention, starting from the physical generation mechanism of multispectral data, constructs a full-link white balance method encompassing data preprocessing, lightweight neural network modeling, spatial partitioning aggregation, and gain mapping.

[0025] Specifically, step S101, acquiring the multi-channel spectral image output by the multispectral imaging chip, includes the following sub-steps: S1011. Obtain the raw RAW data output by the multispectral imaging chip, and perform dark current subtraction and bad pixel correction on each channel in the raw RAW data to obtain corrected RAW data.

[0026] Dark current subtraction and bad pixel correction are necessary because the dark current of each channel of the multispectral chip is inconsistent under long-term exposure or high-temperature conditions, and the position of bad pixels changes with the wavelength. If these fixed-mode noises are not corrected in advance, they will be amplified by subsequent radiometric calibration and contaminate the white point estimation. In this embodiment of the invention, the operation of S1011 is used to subtract the mean of the pre-stored dark frame and replace the interpolation of the neighborhood of bad pixels to ensure that the RAW data only reflects the real incident light signal.

[0027] S1012. The corrected RAW data is radiometrically calibrated using a spectral response matrix pre-calibrated with a standard light source, converting the digital quantization value into an absolute spectral radiance value.

[0028] Radiometric calibration using the spectral response matrix calibrated with a standard light source is a crucial step in converting the dimensionless digital quantization (DN) output of the sensor into a physically meaningful absolute spectral radiance value. This matrix characterizes the linear response relationship of each channel of the chip to a standard light source with a known spectral power distribution. By inverting the matrix or regularizing the inversion, sensitivity differences and non-ideal filtering effects between channels can be eliminated, making data from different bands comparable and providing physically consistent input characteristics for the neural network.

[0029] S1013. Spatially align and resample the absolute spectral radiance values ​​to generate the multi-channel spectral image with uniform spatial resolution.

[0030] Spatial alignment and resampling address the spatial mismatch issues commonly encountered in multispectral chips caused by microlens array offsets or time-division acquisition. Through sub-pixel level registration and unified grid resampling, a multi-channel spectral image tensor with consistent spatial resolution and strictly aligned pixels in each channel is generated. This avoids artifacts or blurring in subsequent U-Net models when processing misaligned spectral features, while also meeting the input format requirements of deep learning frameworks for regular four-dimensional tensors (batch, height, width, channels).

[0031] S102. The multi-channel spectral image is input into a pre-trained lightweight U-Net model. The encoder extracts spatial-spectral joint features step by step, and the decoder fuses multi-scale information with cross-layer connections to obtain a pixel-level white point image through regression prediction. Each spatial position in the pixel-level white point image has a corresponding normalized white point coordinate, which is used to reflect the color temperature characteristics of the light source at the corresponding spatial position.

[0032] The lightweight U-Net model in step S102 includes a spectral channel preprocessing layer, an encoder, and a decoder, wherein: The spectral channel preprocessing layer is located at the front end of the encoder and is used to rearrange the irregularly arranged multispectral raw data in the multichannel spectral image into a uniform tensor format. The encoder consists of a multi-level cascaded depthwise separable convolutional module and a downsampling layer. The kernel size and number of groups of the depthwise separable convolutional module are set according to the number of channels of the multi-channel spectral image. The depthwise separable convolutional module is used to work with the downsampling layer to extract the low-resolution feature map and spatial-spectral joint features of the input multi-channel spectral image step by step. The decoder consists of multiple upsampling layers, skip connection structures, and an output convolutional layer. The upsampling layer employs a transposed convolution operation to restore the low-resolution feature map to the same spatial resolution as the feature map output by the corresponding level of the encoder. The skip connection structure aligns the spatial-spectral joint features output by the corresponding level of the encoder in the spectral dimension and then concatenates and fuses them with the feature map output by the upsampling layer to obtain the pixel-level white dot map. The output convolutional layer has a 1×1 convolutional kernel, and its number of output channels is equal to the vector dimension of the normalized white dot coordinates, used to map the feature channel number of the pixel-level white dot map to the dimension of the normalized white dot coordinates.

[0033] In this embodiment of the invention, the lightweight U-Net model architecture design is key to improving computational efficiency and white balance calculation accuracy. Specifically, unlike existing U-Net models, this embodiment uses the spectral channel preprocessing layer as a dedicated module at the encoder front end, specifically designed to handle the irregular arrangement characteristics of multispectral chip output data, such as Bayer-like mosaic layouts with more than 3 bands, or temporally interleaved sampling of time-multiplexed sensors. This layer reorganizes the original sparse, unaligned multispectral observations into a dense, aligned, unified tensor format by learning or pre-defined rearrangement rules, enabling subsequent convolution operations to effectively extract joint features in both spatial and spectral dimensions, rather than being forced to process a large amount of zero values ​​or misaligned information.

[0034] The kernel size and number of groups in the depth-separable convolution module of the encoder are dynamically configured according to the number of input channels. For example, when the number of channels is 8, a 3×3 convolution kernel is used with 8 groups, which not only preserves the local spectral correlation across channels, but also greatly reduces the number of parameters to adapt to the needs of edge deployment. It works in conjunction with the downsampling layer with a stride of 2 to maintain the spectral channel dimension unchanged while compressing the spatial resolution step by step, thereby generating a series of feature pyramids with decreasing spatial scale but complete spectral information, laying the foundation for multi-scale white point perception.

[0035] In the decoder, the transposed convolutional upsampling layer accurately restores the spatial dimensions, ensuring alignment with the corresponding feature map space of the encoder. The skip connection structure performs spectral dimension alignment before splicing, that is, it eliminates the spectral semantic shift caused by the difference in receptive field in the encoding path through 1×1 convolution or channel permutation, and then splices and fuses it with the upsampling features, which not only supplements high-frequency spatial details but also ensures spectral consistency. Finally, the number of output channels of the 1×1 output convolutional layer is strictly equal to the vector dimension of the normalized white point coordinates (usually 2), directly linearly mapping the high-dimensional fused features to the (r, g) white point space, avoiding redundant intermediate representations.

[0036] In practice, during the training of the lightweight U-Net model, the N-channel multispectral input image is subjected to 99.5 quantile global normalization. The normalization formula is as follows: ; In the formula, The original multispectral image for rendering output, The 99.5th percentile of all pixels in the entire image is used as the threshold, and finally, the pixel value range is constrained to a specific value range through a cropping operation. Normalized multispectral input is obtained Simultaneously, the corresponding pixel-level white point truth map is calculated according to the white point truth generation rules.

[0037] In addition, the input data can be expanded through online data augmentation during the implementation process. The process includes spatial sampling scaling and random noise perturbation: randomly select a region within the original image space for cropping, and then uniformly transform the image to a fixed size through interpolation; then add channel-independent Gaussian noise and truncate the pixel value range. The corresponding white point ground truth values ​​undergo the same cropping and scaling transformations synchronously, and the ground truth values ​​do not introduce noise disturbances, ensuring that the input image and the supervision label maintain an accurate one-to-one spatial correspondence.

[0038] This synchronization enhancement strategy ensures a one-to-one spatial correspondence between the input image and the supervision signal.

[0039] The lightweight U-Net model uses the L1 loss function as the supervision signal for network training. For each pixel... The network outputs the predicted white point value. With the truth value of the white point The L1 loss between them is defined as: ; The L1 loss calculates the absolute value of the error for each channel and each pixel and then averages the results. This effectively constrains the network output to numerically approximate the true value and has good robustness to outliers.

[0040] During training, the network optimizes its model parameters by minimizing this loss function.

[0041] In this embodiment of the invention, angular error (AE) is used as an evaluation index for white balance accuracy. For the first... Each pixel, its white point prediction value with truth value The angle error is defined as: ; This metric measures the angle (in degrees) between the predicted RGB vector and the true RGB vector, comprehensively reflecting the overall error of the three-channel white point prediction. It is a widely used evaluation standard in the field of white balance. Finally, the average AE value of all pixels in the image is taken to obtain the average angular error of the image.

[0042] The entire architecture, while ensuring sufficient expressiveness for white point regression tasks, achieves lightweight design through customized spectral sensing, balancing accuracy and efficiency.

[0043] The normalized white point coordinates in step S102 are two-dimensional coordinate vectors obtained by normalizing the original white point estimates of the spatial positions in the pixel-level white point image.

[0044] The process of generating normalized white point coordinates involves two key steps: regression of the original white point estimate and subsequent normalization processing. The output layer of the lightweight U-Net directly regresses the unconstrained original white point estimate, which may exceed the physically reasonable range (e.g., r or g is less than 0 or greater than 1), or suffer from scale drift due to uneven distribution of training data.

[0045] Therefore, in this embodiment of the invention, an explicit normalization step is added after the model output, which projects the original estimated value to the effective two-dimensional normalized white point coordinates (r, g) within the unit chromaticity triangle, where r = R / (R+G+B) and g = G / (R+G+B), satisfying r>0, g>0, and r+g<1.

[0046] White point coordinate normalization not only ensures that the output conforms to the basic constraints of colorimetry, but also eliminates the interference of absolute intensity on white point estimation, making the coordinates reflect only the color temperature characteristics of the light source. Simultaneously, the normalized coordinate space is compact and bounded, which is beneficial to the statistical stability and numerical robustness of gain mapping during subsequent partitioning and aggregation. Although this process is simple, it is a necessary bridge connecting the output of the neural network's free regression with the physically interpretable white point representation, ensuring the physical consistency and engineering reliability of the entire white balance chain.

[0047] S103. Spatial partitioning and aggregation are performed on the normalized white point coordinates in the pixel-level white point image to obtain the representative white point coordinates corresponding to each partition.

[0048] Step S103, which involves spatially partitioning and aggregating the normalized white point coordinates to obtain representative white point coordinates for each partition, includes the following sub-steps: S1031. Divide the pixel-level white dot image into several spatial partitions according to a preset grid size; S1032. Within each spatial partition, the normalized white point coordinates are weighted according to a preset pixel confidence weight. Abnormal pixels that deviate from the partition mean by more than a preset threshold are removed. The normalized white point coordinates are then recalculated, and the recalculated normalized white point coordinates are used as the representative white point coordinates of the spatial partition.

[0049] The purpose of step S103 is to transform the high-density, noisy local white point estimation in the pixel-level white point image into sparse, robust, and representative white points in the partition, so as to balance spatial adaptability and noise resistance. If single-pixel white points are used directly to drive gain calculation, they are easily misjudged by texture regions or interfered by low signal-to-noise ratio pixels, resulting in patchy color shifts in the corrected image; while global averaging will lose the spatial resolution capability for mixed light sources.

[0050] In this embodiment of the invention, a partitioned aggregation strategy is adopted to compromise the advantages of both single-point driving and global averaging: Step S1031 divides the space into partitions according to a preset grid (e.g., 64×64 pixels). The grid size is set based on a trade-off between the gradient of light source changes in a typical scene and the chip resolution. If the grid is too small, it will be sensitive to noise; if it is too large, local light source information will be lost. For example, the size of the input white point prediction image is... The size of each image patch is The image can then be divided into the following vertical directions: The blocks can be divided horizontally into... There are blocks, and the total number of blocks is [number]. .

[0051] Step S1032 introduces pixel confidence weights within each partition. These weights can be derived from the uncertainty map output by U-Net or from a reliability index calculated based on local gradients and entropy. Pixels with high confidence (such as uniform diffuse regions) are given greater weights, while edge or texture regions are given less weight. After initially suppressing the influence of outliers using weighted averaging, pixels that deviate from the current partition's weighted average by more than a preset threshold (e.g., Euclidean distance > 0.05) are further removed. These pixels typically belong to shadow, highlight, or misclassified regions. After removal, the weighted average is recalculated, and the result is used as the representative white point coordinates for that partition. For example, within each partition, the mean and standard deviation of the R, G, and B channels are calculated separately, and pixels deviating from the mean by more than 3σ in any channel are identified as outliers. Specifically, for the... Image blocks ,Include Each pixel has an RGB white point value of [number] pixels. , .

[0052] Calculate the mean and standard deviation of each channel within the image patch: ; ; Abnormal pixel detection criteria: pixel The deviation from the corresponding channel mean in any channel exceeds ,Right now: ; After removing all abnormal pixels, the set of remaining valid pixels is denoted as . : ; This strategy can effectively eliminate unreliable white point predictions in overexposed highlight areas, specular reflection areas, and dark shadow areas.

[0053] Finally, local aggregation is performed. The average value of the remaining valid white points after removing abnormal pixels is calculated as the representative white point of the image block. Specifically, the average value of the remaining valid pixels after removing abnormal pixels is calculated. Calculate the average value to obtain the first... Representative white points of each image patch : ; in For the first The number of valid pixels in each image block.

[0054] This two-stage robust aggregation mechanism effectively filters outliers while preserving the true color temperature information of the dominant light source within the partition, providing a clean and reliable input for subsequent gain generation.

[0055] S104. Generate the partition white balance gain corresponding to the partition based on the representative white point coordinates.

[0056] Step S104, which generates the partition white balance gain based on the representative white point coordinates, specifically involves: By using a preset white point-gain mapping relationship, the coordinates of the representative white point are converted into R / G gain ratio and B / G gain ratio, which are used as the white balance gain of the partition.

[0057] The white-point-gain mapping is the core step in converting the representative white-point coordinates in the color space into the R / G and B / G gain ratios required for actual correction. During implementation, this mapping can be achieved through a lookup table or analytical formula. During the system calibration phase, the multispectral chip is illuminated with a standard light source of known color temperature. The coordinates of the representative white point under each standard light source and its corresponding ideal gain value (so that the neutral gray area after correction is R=G=B) are recorded. A mapping table or fitting polynomial from the white point coordinates to the gain ratio is constructed. During step S104, the representative white point coordinates obtained in S103 are substituted into this mapping, and interpolation yields accurate R / G and B / G gain ratios. This mapping relationship implicitly incorporates comprehensive compensation for chip spectral response, lens transmittance, and human eye color adaptation characteristics, ensuring that the corrected image is not only neutral at the sensor level but also conforms to visually perceived white balance.

[0058] Since the gain ratio depends only on the normalized white point coordinates and is independent of the absolute brightness, this mapping has good generalization ability and extremely low computational cost, making it suitable for real-time processing.

[0059] S105. The multi-channel spectral image is white-balanced using the partitioned white-balance gain to obtain a white-balanced corrected image.

[0060] Step S105, which involves performing white balance correction on the multi-channel spectral image using the partitioned white balance gain to obtain a white balance corrected image, includes the following sub-steps: S1051. Upsample the white balance gain of each partition to the same spatial resolution as the multi-channel spectral image through bilinear interpolation or spline interpolation to generate a pixel-level gain map. S1052. Multiply the pixel-level gain map element-wise with the multi-channel spectral image to obtain the white balance correction image.

[0061] The purpose of step S105 is to smoothly extend the discrete partition white balance gain to the entire image and complete the final correction by element-wise multiplication, so as to avoid visual artifacts caused by abrupt gain changes at the partition boundaries.

[0062] The S1051 uses bilinear interpolation or cubic spline interpolation to upsample the representative gain at the center of the partition to the original image resolution, generating a continuous pixel-level gain map. Bilinear interpolation is computationally efficient and ensures a natural gain transition, making it suitable for most scenes. For areas with drastic changes in light source, spline interpolation can be used to obtain higher-order continuity. The interpolation process essentially performs low-pass filtering on the partition gain field, suppressing high-frequency noise while preserving the main spatial trends.

[0063] S1052 multiplies the pixel-level gain map element-wise with the original multi-channel spectral image, that is, it multiplies the R, G, B (or each channel of the multispectral image) of each pixel by the gain value at the corresponding position to obtain the white balance correction image. During the implementation, this operation is performed in the linear optical domain, which conforms to the physical definition of white balance. Since the gain map has been smoothed, the correction result has no block effect or ringing.

[0064] The key implementation details of the method proposed in this invention are concentrated in the deep adaptation to the characteristics of multispectral data and the guarantee of end-to-end physical consistency: First, the three-step preprocessing in S101—dark current subtraction, radiometric calibration, and spatial resampling—ensures that the data input to the neural network has physical interpretability and spatial regularity, which is a prerequisite for the effectiveness of all subsequent algorithms.

[0065] Secondly, the lightweight U-Net is not a simple transfer of a general model. Instead, it achieves efficient extraction and fusion of spatial-spectral joint features through exclusive designs such as spectral channel preprocessing layers, channel-aware depth-separable convolutions, and spectral dimension-aligned skip connections, making pixel-level white point regression both accurate and lightweight.

[0066] Furthermore, the explicit constraint of normalized white point coordinates and the partitioned robust aggregation mechanism effectively connect the neural network output and physical correction parameters, balancing estimation accuracy and noise robustness. Finally, the white point-gain mapping and smooth interpolation correction form a closed-loop execution path, ensuring that the final output image meets white balance requirements at both the visual and physical levels.

[0067] The beneficial effects achieved by this invention are that it provides a white balance method for complex light source partitioning based on a spectral imaging chip. This method predicts and regresses a pixel-level normalized white point map through a lightweight model, and performs spatial partitioning and aggregation of the white point coordinates to generate white balance gain. It also performs overall correction on multi-channel spectral images, achieving accurate estimation and stable correction of white points in complex scenes such as mixed light sources. This effectively avoids color shift stitching artifacts caused by traditional independent partitioning correction, and significantly improves the realism and consistency of image color reproduction.

[0068] Example 2 This invention also provides a complex light source zoning white balance system 200 based on a spectral imaging chip, please refer to... Figure 2 , Figure 2 This is a schematic diagram of the structure of a complex light source zoning white balance system based on a spectral imaging chip provided in an embodiment of the present invention, which includes: Data acquisition module 201 is used to acquire multi-channel spectral images output by the multispectral imaging chip; The feature extraction module 202 is used to input the multi-channel spectral image into a pre-trained lightweight U-Net model, extract spatial-spectral joint features step by step through the encoder, and fuse multi-scale information through the decoder and cross-layer connection to obtain a pixel-level white point map through regression prediction; wherein, each spatial position in the pixel-level white point map has a corresponding normalized white point coordinate, and the normalized white point coordinate is used to reflect the light source color temperature characteristics of the corresponding spatial position. The partition aggregation module 203 is used to perform spatial partition aggregation on the normalized white point coordinates in the pixel-level white point image to obtain the representative white point coordinates corresponding to each partition. The partition gain module 204 is used to generate the partition white balance gain corresponding to the partition based on the coordinates of the representative white point. The correction module 205 is used to perform white balance correction on the multi-channel spectral image using the partitioned white balance gain to obtain a white balance corrected image.

[0069] The complex light source zoning white balance system 200 based on the spectral imaging chip can implement the steps in the complex light source zoning white balance method based on the spectral imaging chip in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0070] Example 3 This invention also provides a computer device, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a complex light source partition white balance program based on a spectral imaging chip stored in the memory 302 and capable of running on the processor 301.

[0071] The processor 301 calls the complex light source zoning white balance program based on the spectral imaging chip stored in the memory 302, and executes the steps in the complex light source zoning white balance method based on the spectral imaging chip provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps: S101. Acquire the multi-channel spectral image output by the multispectral imaging chip.

[0072] Step S101, acquiring the multi-channel spectral image output by the multispectral imaging chip, includes the following sub-steps: S1011. Obtain the raw RAW data output by the multispectral imaging chip, and perform dark current subtraction and bad pixel correction on each channel in the raw RAW data to obtain corrected RAW data. S1012. Radiometric calibration of the corrected RAW data is performed using a spectral response matrix pre-calibrated with a standard light source, converting the digital quantization value into an absolute spectral radiance value. S1013. Spatially align and resample the absolute spectral radiance values ​​to generate the multi-channel spectral image with uniform spatial resolution.

[0073] S102. The multi-channel spectral image is input into a pre-trained lightweight U-Net model. The encoder extracts spatial-spectral joint features step by step, and the decoder fuses multi-scale information with cross-layer connections to obtain a pixel-level white point image through regression prediction. Each spatial position in the pixel-level white point image has a corresponding normalized white point coordinate, which is used to reflect the color temperature characteristics of the light source at the corresponding spatial position.

[0074] The lightweight U-Net model in step S102 includes a spectral channel preprocessing layer, an encoder, and a decoder, wherein: The spectral channel preprocessing layer is located at the front end of the encoder and is used to rearrange the irregularly arranged multispectral raw data in the multichannel spectral image into a uniform tensor format. The encoder consists of a multi-level cascaded depthwise separable convolutional module and a downsampling layer. The kernel size and number of groups of the depthwise separable convolutional module are set according to the number of channels of the multi-channel spectral image. The depthwise separable convolutional module is used to work with the downsampling layer to extract the low-resolution feature map and spatial-spectral joint features of the input multi-channel spectral image step by step. The decoder consists of multiple upsampling layers, skip connection structures, and an output convolutional layer. The upsampling layer employs a transposed convolution operation to restore the low-resolution feature map to the same spatial resolution as the feature map output by the corresponding level of the encoder. The skip connection structure aligns the spatial-spectral joint features output by the corresponding level of the encoder in the spectral dimension and then concatenates and fuses them with the feature map output by the upsampling layer to obtain the pixel-level white dot map. The output convolutional layer has a 1×1 convolutional kernel, and its number of output channels is equal to the vector dimension of the normalized white dot coordinates, used to map the feature channel number of the pixel-level white dot map to the dimension of the normalized white dot coordinates.

[0075] The normalized white point coordinates in step S102 are two-dimensional coordinate vectors obtained by normalizing the original white point estimates of the spatial positions in the pixel-level white point image.

[0076] S103. Spatial partitioning and aggregation are performed on the normalized white point coordinates in the pixel-level white point image to obtain the representative white point coordinates corresponding to each partition.

[0077] Step S103, which involves spatially partitioning and aggregating the normalized white point coordinates to obtain representative white point coordinates for each partition, includes the following sub-steps: S1031. Divide the pixel-level white dot image into several spatial partitions according to a preset grid size; S1032. Within each spatial partition, the normalized white point coordinates are weighted according to a preset pixel confidence weight. Abnormal pixels that deviate from the partition mean by more than a preset threshold are removed. The normalized white point coordinates are then recalculated, and the recalculated normalized white point coordinates are used as the representative white point coordinates of the spatial partition.

[0078] S104. Generate the partition white balance gain corresponding to the partition based on the representative white point coordinates.

[0079] Step S104, which generates the partition white balance gain based on the representative white point coordinates, specifically involves: By using a preset white point-gain mapping relationship, the coordinates of the representative white point are converted into R / G gain ratio and B / G gain ratio, which are used as the white balance gain of the partition.

[0080] S105. The multi-channel spectral image is white-balanced using the partitioned white-balance gain to obtain a white-balanced corrected image.

[0081] Step S105, which involves performing white balance correction on the multi-channel spectral image using the partitioned white balance gain to obtain a white balance corrected image, includes the following sub-steps: S1051. Upsample the white balance gain of each partition to the same spatial resolution as the multi-channel spectral image through bilinear interpolation or spline interpolation to generate a pixel-level gain map. S1052. Multiply the pixel-level gain map element-wise with the multi-channel spectral image to obtain the white balance correction image.

[0082] The computer device 300 provided in this embodiment of the invention can implement the steps in the complex light source partitioning white balance method based on spectral imaging chip as described in the above embodiments, and can achieve the same technical effect. Referring to the description in the above embodiments, it will not be repeated here.

[0083] Example 4 This invention also provides a storage medium storing a complex light source zoning white balance program based on a spectral imaging chip. When the processor executes this complex light source zoning white balance program based on a spectral imaging chip, it implements each process and step in the complex light source zoning white balance method based on a spectral imaging chip provided in this invention, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0084] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to a complex light source zoning white balance program or instructions based on a spectral imaging chip. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0085] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0087] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form under the guidance of the present invention without departing from the spirit and scope of the claims. All such changes are within the protection scope of the present invention.

Claims

1. A method for zonal white balance of complex light sources based on a spectral imaging chip, characterized in that, Includes the following steps: S101. Acquire the multi-channel spectral image output by the multispectral imaging chip; S102. Input the multi-channel spectral image into the pre-trained lightweight U-Net model, extract the spatial-spectral joint features step by step through the encoder, and fuse multi-scale information through the decoder and cross-layer connection to obtain a pixel-level white point map through regression prediction; wherein, each spatial position in the pixel-level white point map has a corresponding normalized white point coordinate, and the normalized white point coordinate is used to reflect the light source color temperature characteristics of the corresponding spatial position. S103. Spatial partitioning and aggregation are performed on the normalized white point coordinates in the pixel-level white point image to obtain the representative white point coordinates corresponding to each partition. S104. Generate the partition white balance gain corresponding to the partition based on the coordinates of the representative white point; S105. The multi-channel spectral image is white-balanced using the partitioned white-balance gain to obtain a white-balanced image. The lightweight U-Net model in step S102 includes a spectral channel preprocessing layer, an encoder, and a decoder, wherein: The spectral channel preprocessing layer is located at the front end of the encoder and is used to rearrange the irregularly arranged multispectral raw data in the multichannel spectral image into a uniform tensor format. The encoder consists of a multi-level cascaded depthwise separable convolutional module and a downsampling layer. The kernel size and number of groups of the depthwise separable convolutional module are set according to the number of channels of the multi-channel spectral image. The depthwise separable convolutional module is used to work with the downsampling layer to extract the low-resolution feature map and spatial-spectral joint features of the input multi-channel spectral image step by step. The decoder consists of multiple upsampling layers, skip connection structures, and an output convolutional layer. The upsampling layer employs a transposed convolution operation to restore the low-resolution feature map to the same spatial resolution as the feature map output by the corresponding level of the encoder. The skip connection structure aligns the spatial-spectral joint features output by the corresponding level of the encoder in the spectral dimension and then concatenates and fuses them with the feature map output by the upsampling layer to obtain the pixel-level white dot map. The output convolutional layer has a 1×1 convolutional kernel, and its number of output channels is equal to the vector dimension of the normalized white dot coordinates, used to map the feature channel number of the pixel-level white dot map to the dimension of the normalized white dot coordinates.

2. The complex light source zoning white balance method based on a spectral imaging chip according to claim 1, characterized in that, Step S101, acquiring the multi-channel spectral image output by the multispectral imaging chip, includes the following sub-steps: S1011. Obtain the raw RAW data output by the multispectral imaging chip, and perform dark current subtraction and bad pixel correction on each channel in the raw RAW data to obtain corrected RAW data. S1012. Radiometric calibration of the corrected RAW data is performed using a spectral response matrix pre-calibrated with a standard light source, converting the digital quantization value into an absolute spectral radiance value. S1013. Spatially align and resample the absolute spectral radiance values ​​to generate the multi-channel spectral image with uniform spatial resolution.

3. The complex light source zoning white balance method based on a spectral imaging chip according to claim 1, characterized in that, The normalized white point coordinates in step S102 are two-dimensional coordinate vectors obtained by normalizing the original white point estimates of the spatial positions in the pixel-level white point image.

4. The complex light source zoning white balance method based on a spectral imaging chip according to claim 1, characterized in that, Step S103, which involves spatially partitioning and aggregating the normalized white point coordinates to obtain representative white point coordinates for each partition, includes the following sub-steps: S1031. Divide the pixel-level white dot image into several spatial partitions according to a preset grid size; S1032. Within each spatial partition, the normalized white point coordinates are weighted according to a preset pixel confidence weight. Abnormal pixels that deviate from the partition mean by more than a preset threshold are removed. The normalized white point coordinates are then recalculated, and the recalculated normalized white point coordinates are used as the representative white point coordinates of the spatial partition.

5. The complex light source zoning white balance method based on a spectral imaging chip according to claim 1, characterized in that, Step S104, which generates the partition white balance gain based on the representative white point coordinates, specifically involves: By using a preset white point-gain mapping relationship, the coordinates of the representative white point are converted into R / G gain ratio and B / G gain ratio, which are used as the white balance gain of the partition.

6. The complex light source zoning white balance method based on a spectral imaging chip according to claim 1, characterized in that, Step S105, which involves performing white balance correction on the multi-channel spectral image using the partitioned white balance gain to obtain a white balance corrected image, includes the following sub-steps: S1051. Upsample the white balance gain of each partition to the same spatial resolution as the multi-channel spectral image through bilinear interpolation or spline interpolation to generate a pixel-level gain map. S1052. Multiply the pixel-level gain map element-wise with the multi-channel spectral image to obtain the white balance correction image.

7. A complex light source zoning white balance system based on a spectral imaging chip, characterized in that, include: The data acquisition module is used to acquire multi-channel spectral images output by the multispectral imaging chip; The feature extraction module is used to input the multi-channel spectral image into a pre-trained lightweight U-Net model, extract spatial-spectral joint features step by step through the encoder, and fuse multi-scale information through the decoder and cross-layer connection to obtain a pixel-level white point map through regression prediction; wherein, each spatial position in the pixel-level white point map has a corresponding normalized white point coordinate, and the normalized white point coordinate is used to reflect the light source color temperature characteristics of the corresponding spatial position. The partitioning aggregation module is used to perform spatial partitioning aggregation on the normalized white point coordinates in the pixel-level white point image to obtain the representative white point coordinates corresponding to each partition. The partition gain module is used to generate the partition white balance gain corresponding to the partition based on the coordinates of the representative white point. The correction module is used to perform white balance correction on the multi-channel spectral image using the partitioned white balance gain to obtain a white balance corrected image; The lightweight U-Net model includes a spectral channel preprocessing layer, an encoder, and a decoder, wherein: The spectral channel preprocessing layer is located at the front end of the encoder and is used to rearrange the irregularly arranged multispectral raw data in the multichannel spectral image into a uniform tensor format. The encoder consists of a multi-level cascaded depthwise separable convolutional module and a downsampling layer. The kernel size and number of groups of the depthwise separable convolutional module are set according to the number of channels of the multi-channel spectral image. The depthwise separable convolutional module is used to work with the downsampling layer to extract the low-resolution feature map and spatial-spectral joint features of the input multi-channel spectral image step by step. The decoder consists of multiple upsampling layers, skip connection structures, and an output convolutional layer. The upsampling layer employs a transposed convolution operation to restore the low-resolution feature map to the same spatial resolution as the feature map output by the corresponding level of the encoder. The skip connection structure aligns the spatial-spectral joint features output by the corresponding level of the encoder in the spectral dimension and then concatenates and fuses them with the feature map output by the upsampling layer to obtain the pixel-level white dot map. The output convolutional layer has a 1×1 convolutional kernel, and its number of output channels is equal to the vector dimension of the normalized white dot coordinates, used to map the feature channel number of the pixel-level white dot map to the dimension of the normalized white dot coordinates.

8. A computer device, characterized in that, include: The system includes a memory, a processor, and a complex light source zoning white balance program based on a spectral imaging chip, stored in the memory and executable on the processor. When the processor executes the complex light source zoning white balance program based on a spectral imaging chip, it implements the steps of the complex light source zoning white balance method based on a spectral imaging chip as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a complex light source zoning white balance program based on a spectral imaging chip. When the processor executes the complex light source zoning white balance program based on a spectral imaging chip, it implements the steps in the complex light source zoning white balance method based on a spectral imaging chip as described in any one of claims 1-6.

Images