Multi-spectral white balance method and system under complex light source and storage medium

Abstract

This invention relates to the field of multispectral computational imaging technology, and particularly to a multispectral white balance method, system, and storage medium under complex light sources. The multispectral white balance method includes: acquiring hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data; weighting and superimposing the spectra of each light source with the corresponding pixel's mixing weight pixel by pixel to obtain a mixed light source spectrum; performing spectral integration by combining hyperspectral reflectance and the spectral response function of a multispectral camera to generate a multi-channel multispectral simulation image; generating pixel-level white point ground truth values ​​based on the integration and normalization of the mixed light source spectrum and the RGB camera spectral response function; and supervising the training of a network model using the simulation image and the white point ground truth values ​​to obtain a multispectral white balance correction model for correcting the multispectral image to be corrected. Compared with existing technologies, this invention significantly improves the generalization ability and correction accuracy under complex mixed light sources.

Description

Technical Field

[0001] This invention relates to the field of multispectral computational imaging technology, and in particular to a multispectral white balance method, system and storage medium under complex light sources. Background Technology

[0002] Hyperspectral and multispectral imaging technologies can acquire the imaging response of a target scene across multiple spectral channels. Unlike traditional RGB imaging technology, which only records information in three broad bands (red, green, and blue), hyperspectral or multispectral imaging technologies can acquire richer spectral information, thereby reflecting the reflection, absorption, and radiation characteristics of different substances in the spectral dimension. They are widely used in fields such as industrial inspection, remote sensing identification, medical imaging, agricultural monitoring, and computational imaging.

[0003] In the field of multispectral computational imaging technology, multispectral images can also be used to estimate the spectral characteristics or chromaticity information of scene light sources, thereby achieving white balance correction. The core of white balance processing lies in determining the illumination color corresponding to each pixel in the image to eliminate the interference caused by different light source color temperatures and spectral distributions on the true color of objects. In recent years, deep learning-based white balance estimation methods have gradually become mainstream. These methods establish a mapping relationship between multispectral images and light source chromaticity by training neural networks.

[0004] However, the performance of deep learning models is highly dependent on the scale and distribution of training data. For ordinary RGB vision tasks, large-scale training data can be obtained through mobile terminal shooting, online image collection, and manual annotation; but for multispectral white balance tasks, the acquisition of training data is significantly limited. On the one hand, hyperspectral or multispectral imaging equipment is expensive, and some equipment requires line-by-line or band-by-band scanning, resulting in low data acquisition efficiency; on the other hand, multispectral data needs to undergo dark current correction, whiteboard correction, spectral response calibration, and radiometric calibration, and the data acquisition and annotation process is highly dependent on specialized equipment and operators.

[0005] More importantly, real-world application environments are typically not single, constant light source scenarios. For example, in indoor scenes, natural light from outside a window and artificial light sources such as indoor LED lights and halogen lamps may simultaneously illuminate the same object; in photography studios, industrial workshops, agricultural greenhouses, and urban night scenes, multiple color temperatures and types of light sources may also coexist. Due to differences in light source location, propagation distance, object geometry, occlusion relationships, and surface reflection properties, the contribution ratio of various light sources at different pixel locations in an image usually exhibits a continuous, gradually changing two-dimensional spatial distribution.

[0006] Existing synthesis methods for augmenting training data with mixed light sources typically employ geometric segmentation to divide the image into different regions and assign different types of light source spectra to each region. For example, an image might be divided into a left and right half, with the left half using the first light source spectrum and the right half using the second, followed by simple linear interpolation near the segmentation boundaries. While such methods can generate simulated images with characteristics of both light sources, their light source weights are primarily determined by manually set geometric boundaries, failing to reflect the continuous illumination variations in real mixed-light scenes that are jointly determined by light source position, distance attenuation, spatial occlusion, and the three-dimensional structure of objects.

[0007] Furthermore, some existing simulation methods employ the assumption of a globally constant light source, which sets the light source spectrum and illumination intensity for all pixels in the entire image to the same value. This assumption ignores the illumination distance attenuation, surface normal changes, and local occlusion shadows during the actual imaging process, resulting in simulated images lacking the illumination gradients and shadow variations found in real scenes.

[0008] Therefore, there is a significant distribution difference between existing hybrid light source simulation data and real-world acquired data. Deep learning models trained using such simulation data are prone to learning unrealistic features such as artificially segmented boundaries or globally constant illumination, leading to a decrease in the accuracy of white point estimation in real-world complex lighting scenarios. How to generate multispectral training data that conforms to real optical imaging laws, has continuous spatial illumination variation characteristics, and includes accurate white point annotations is a pressing technical problem to be solved in this field. Summary of the Invention

[0009] This invention provides a multispectral white balance method, system, and storage medium for complex light sources, aiming to solve the problems of existing geometric segmentation light source data synthesis methods that deviate from the laws of optical physics, have distributional offsets between synthesized data and real scenes, lack scene diversity, and have high costs for multispectral training data acquisition and difficulties in pixel-level ground truth labeling.

[0010] In a first aspect, the present invention provides a multispectral white balance method under complex light sources, the multispectral white balance method under complex light sources comprising the following steps:

[0011] S1. Acquire hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data; wherein, the hyperspectral reflectance data is used to characterize the spectral reflectance characteristics of an object surface to incident light of different wavelengths; the spectral power distribution data of multiple light sources is used to characterize the relative energy distribution of different light sources at different wavelengths; and the true weight distribution map data is used to characterize the spatial mixing weight of each light source at each pixel position in the image. S2. Based on the hyperspectral reflectance data, the spectral power distribution data of the multiple light sources, the real weight distribution map data, and the spectral response function of the multispectral camera, spectral integration is performed to obtain a multi-channel multispectral simulation image. S3. Based on the spectral power distribution data of the multiple light sources and the real weight distribution map data, and based on the spectral response function of the RGB camera, perform spectral integration to obtain the true value of the white point corresponding to the multi-channel multispectral simulation image. S4. Supervised training of the network model is performed based on the multi-channel multispectral simulation image and the white point ground value to obtain the multispectral white balance correction model. S5. Obtain the multispectral image to be corrected, and use the multispectral image to be corrected as the input of the multispectral white balance correction model to obtain the white balance corrected multispectral image.

[0012] Preferably, step S2 includes the following sub-steps: S21. By performing pixel-by-pixel weighted superposition of the spectral power distribution of each light source in the multi-type light source spectral power distribution data and the mixing weight of each light source in the corresponding pixel in the real weight distribution map data, pixel-level mixed light source spectral data is obtained. S22. Based on the pixel-level mixed light source spectral data, the hyperspectral reflectance data, and the multispectral camera spectral response function, perform spectral integration operations pixel by pixel and spectral channel by traversing all pixel positions and all spectral channels to obtain multiple imaging response values. S23. All the imaging response values ​​are arranged and integrated according to the two-dimensional spatial coordinates of the image and the dimensions of the multispectral channels to obtain the multi-channel multispectral simulation image.

[0013] Preferably, the pixel-level hybrid light source spectral data satisfies the following conditions: ; in, Indicates at wavelength Next pixel The corresponding pixel-level hybrid light source spectral data, Indicates the first i Each light source at wavelength The following is the spectral power distribution data of various light sources. Represents pixels In the i Spatial mixing weights corresponding to each light source Indicates the number of light sources.

[0014] Preferably, the imaging response value satisfies the following condition: ; in, Represents pixels In the c Imaging response values ​​of multiple spectral channels, Represents the spectral integral operator. Indicates the first i Spectral power distribution data of the various types of light sources, for each light source. Represents pixels The corresponding spectral reflectance, This represents the spectral response function of the multispectral camera.

[0015] Preferably, step S3 includes the following sub-steps: S31. Perform channel-by-channel spectral integration processing based on the pixel-level mixed light source spectral data and the RGB camera spectral response function to obtain pixel-level raw RGB response data; S32. Normalize the pixel-level raw RGB response data to obtain the true value of the white point.

[0016] Preferably, the pixel-level raw RGB response data satisfies the following conditions: ; in, Represents pixels The pixel-level raw RGB response data, This represents the spectral response function of an RGB camera.

[0017] Preferably, the hyperspectral reflectance data includes multiple two-dimensional spatial images, each of the two-dimensional spatial images includes multiple pixels, and each pixel includes reflectance values ​​for multiple spectral bands; The spectral power distribution data of the multiple light sources includes multiple continuous spectral curves with different color temperatures and different light source types. The continuous spectral curves are used to characterize the relative energy distribution of the light source at different wavelengths. The real weight distribution map data includes multiple weight distribution maps. Each pixel position in each weight distribution map corresponds to a mixed weight coefficient that matches each light source. For any pixel position, the mixed weight coefficients corresponding to each light source are between 0 and 1, and the sum of the mixed weight coefficients corresponding to each light source is equal to 1.

[0018] Secondly, the present invention also provides a multispectral white balance system under complex light sources, the multispectral white balance system under complex light sources comprising: The data acquisition module is used to acquire hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data. The hyperspectral reflectance data characterizes the spectral reflectance properties of an object surface to incident light of different wavelengths. The spectral power distribution data of multiple light sources characterizes the relative energy distribution of different light sources at different wavelengths. The true weight distribution map data characterizes the spatial mixing weights of each light source at each pixel location in the image. The simulation image module is used to perform spectral integration calculations based on the hyperspectral reflectance data, the spectral power distribution data of the multiple light sources, the real weight distribution map data, and the spectral response function of the multispectral camera to obtain a multi-channel multispectral simulation image. The white point truth module is used to obtain the white point truth value corresponding to the multi-channel multispectral simulation image by performing spectral integration calculation based on the spectral power distribution data of the multi-type light sources and the real weight distribution map data, and based on the RGB camera spectral response function. The model training module is used to supervise the training of the network model based on the multi-channel multispectral simulation image and the white point ground truth, so as to obtain a multispectral white balance correction model. The white balance module is used to acquire the multispectral image to be corrected, and to use the multispectral image to be corrected as the input of the multispectral white balance correction model to obtain the white balance corrected multispectral image.

[0019] Thirdly, the present invention also provides a computer device, comprising: a memory, a processor, and a multispectral white balance program for complex light sources stored in the memory and executable on the processor, wherein when the processor executes the multispectral white balance program for complex light sources, it implements the steps in the multispectral white balance method for complex light sources as described in any of the above embodiments.

[0020] Fourthly, the present invention also provides a computer-readable storage medium storing a multispectral white balance program under complex light sources, wherein when the multispectral white balance program under complex light sources is executed by a processor, the steps in the multispectral white balance method under complex light sources as described in any of the above embodiments are implemented.

[0021] Compared with existing technologies, this invention utilizes a true weight distribution map to represent the spatial contribution ratio of each light source at different pixel locations, and generates multi-channel multispectral simulation images through pixel-by-pixel weighted spectral integration. This avoids unnatural light source boundaries formed by geometric stitching, making the illumination changes in the simulation data more consistent with real imaging laws and improving the realism of the simulation data. This invention synchronously generates simulation images and pixel-level white point ground truth values ​​based on the same multi-source light spectral power distribution data and weight distribution map, ensuring a strict correspondence between the two in spatial location and illumination mechanism. No manual pixel-by-pixel annotation is required, guaranteeing the physical consistency between the image and the label. This invention supports mixed simulations of dual, three, four, and any number of light sources. It can be expanded by simply adding corresponding multi-source light spectral power distribution data and weight channels, generating richer and more complex illumination training samples, and improving the adaptability to complex lighting scenes. Attached Figure Description

[0022] The present invention will now be described in detail with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and more readily understood through the detailed description following the accompanying drawings. In the drawings: Figure 1 This is a flowchart of the multispectral white balance method under complex light sources provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the structure of the multispectral white balance system under complex light sources provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Example 1 Please refer to Figure 1 This invention provides a multispectral white balance method under complex light sources, which includes the following steps: S1. Acquire hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data; wherein, the hyperspectral reflectance data is used to characterize the spectral reflectance characteristics of an object surface to incident light of different wavelengths; the spectral power distribution data of multiple light sources is used to characterize the relative energy distribution of different light sources at different wavelengths; and the true weight distribution map data is used to characterize the spatial mixing weight of each light source at each pixel position in the image.

[0025] In this embodiment of the invention, the hyperspectral reflectance data includes multiple two-dimensional spatial images, each of which includes multiple pixels, and each pixel includes reflectance values ​​for multiple spectral bands. The hyperspectral reflectance data may be derived from the KAUST dataset, CAVE dataset, or a self-collected dataset. The size of each two-dimensional spatial image is H×W, where H is the height of the image and W is the width of the image. Each pixel contains reflectance values ​​for M spectral bands. The reflectance is an inherent physical property of the object and does not change with the lighting environment. The hyperspectral reflectance data serves as the basis for material information in data synthesis.

[0026] The multi-source spectral power distribution data includes multiple continuous spectral curves with different color temperatures and different light source types. The continuous spectral curves are used to characterize the relative energy distribution of the light source at different wavelengths. The multi-source spectral power distribution data is spectral power distribution (SPD) data obtained by actual measurement using a spectroradiometer.

[0027] The real weight distribution map data includes multiple weight distribution maps. Each pixel position in each weight distribution map corresponds to a mixing weight coefficient that matches each light source. For any given pixel position, the mixing weight coefficients for each light source are between 0 and 1, and the sum of the mixing weight coefficients for all light sources equals 1. The real weight distribution map data can be derived from the LSMI dataset. Each weight distribution map has a size of H×W×N and records the mixing weights of N light sources at each pixel position in the scene. The mixing weights are continuous floating-point values, and their spatial distribution naturally incorporates multiple physical influencing factors such as the object's three-dimensional geometry, spatial occlusion relationships, the object's surface material reflection characteristics, the spatial position of the light source, and illumination attenuation, reflecting the physical law that the mixing weights of light sources continuously and gradually change with spatial position in the real world. This invention introduces pixel-by-pixel continuous light source weight distribution collected from real scenes to replace the traditional manual geometric hard segmentation method. This ensures that the two-dimensional spatial changes of the mixing weights of multiple light sources within the synthesized multispectral image completely conform to the physical laws of real optical imaging, eliminating false light source boundaries caused by geometric segmentation and improving the consistency between the synthesized data and the distribution of the real scene.

[0028] S2. Based on the hyperspectral reflectance data, the spectral power distribution data of the various light sources, the real weight distribution map data, and the spectral response function of the multispectral camera, spectral integration is performed to obtain a multi-channel multispectral simulation image.

[0029] In this embodiment of the invention, the spectral response function of the multispectral camera is used to accurately match the photoelectric conversion characteristics of a real camera; optionally, the multispectral camera is a 14-channel multispectral camera, and correspondingly, the multi-channel multispectral simulation image includes 14 spectral channels.

[0030] Step S2 includes the following sub-steps: S21. By performing pixel-by-pixel weighted superposition of the spectral power distribution of each light source in the multi-type light source spectral power distribution data and the mixing weight of each light source in the corresponding pixel in the real weight distribution map data, pixel-level mixed light source spectral data is obtained. S22. Based on the pixel-level mixed light source spectral data, the hyperspectral reflectance data, and the multispectral camera spectral response function, perform spectral integration operations pixel by pixel and spectral channel by traversing all pixel positions and all spectral channels to obtain multiple imaging response values. S23. All the imaging response values ​​are arranged and integrated according to the two-dimensional spatial coordinates of the image and the dimensions of the multispectral channels to obtain the multi-channel multispectral simulation image.

[0031] In this embodiment of the invention, the pixel-level hybrid light source spectral data satisfies the following conditions: ; in, Indicates at wavelength Next pixel The corresponding pixel-level hybrid light source spectral data, Indicates the first i Each light source at wavelength The following is the spectral power distribution data of various light sources. Represents pixels In the i Spatial mixing weights corresponding to each light source Indicates the number of light sources.

[0032] In this embodiment of the invention, the imaging response value satisfies the following condition: ; in, Represents pixels In the c Imaging response values ​​of multiple spectral channels, Represents the spectral integral operator. Indicates the first i Spectral power distribution data of the various types of light sources, for each light source. Represents pixels The corresponding spectral reflectance, This represents the spectral response function of the multispectral camera.

[0033] In this embodiment of the invention, a dual-light source mixed scene is taken as a preferred implementation example, i.e., N=2, at which time the imaging response value satisfies: ; And the weights satisfy the normalization constraint. When the simulation scenario involves a mixed three-source lighting condition, a third source spectrum is added. and the corresponding spatial weights The overall imaging model is extended to a three-channel spectral integral superposition form, with the weights satisfying the normalization constraint. Similarly, a fourth light source parameter and weighting factor can be added to the four-light source scene. The number of weight channels corresponds one-to-one with the number of light sources participating in the mixing. The number of input light sources and the dimension of weight channels can be dynamically matched through the multi-light source extension adaptation control module, and the global weight normalization constraint conditions can be updated to realize the expansion of mixed lighting simulation without limiting the number of light sources. This invention constructs the imaging response of each pixel as a continuous weighted superposition of the spectra of multiple light sources. The weight of a single light source changes continuously in the range of [0,1], which can realistically reproduce the smooth and gradual transition characteristics of natural light and completely solve the defects of false light source boundaries and data distribution offset caused by geometric segmentation schemes. At the same time, the general weighted integral architecture is not limited by the number of light sources and can simulate complex real scenes such as indoor multi-light mixing, outdoor multi-light superposition, and multi-color temperature supplementary lighting in a photography studio, enriching the diversity of lighting conditions in training data.

[0034] S3. Based on the spectral power distribution data of the multiple light sources and the real weight distribution map data, and by performing spectral integration based on the RGB camera spectral response function, the ground truth (GT) of the white point corresponding to the multi-channel multispectral simulation image is obtained.

[0035] In this embodiment of the invention, step S3 includes the following sub-steps: S31. Perform channel-by-channel spectral integration processing based on the pixel-level mixed light source spectral data and the RGB camera spectral response function to obtain pixel-level raw RGB response data; S32. Normalize the pixel-level raw RGB response data to obtain the true value of the white point.

[0036] In this embodiment of the invention, the pixel-level raw RGB response data satisfies the following conditions: ; in, Represents pixels The pixel-level raw RGB response data, This represents the spectral response function of an RGB camera.

[0037] In this embodiment of the invention, the normalization process is a pixel-level normalization process, which maps the three-channel response values ​​to the 0~1 range and satisfies that the sum of the three channel values ​​is 1. The true value of the white point satisfies the following condition: ; Decomposed into a single-channel independent expression: ; ; ; in, Represents pixels The corresponding white point ground truth value. This invention eliminates overall light intensity interference and preserves the color temperature and color characteristics of the light source through normalization processing, meeting the normalized supervision label requirements for white balance network training; furthermore, the white point ground truth labeling process and the multispectral image synthesis process are based on unified light source, weights, and spectral integral physical logic, sharing the same source and physical benchmark, with no human error or distribution shift, achieving a one-to-one accurate correspondence between the synthesized image and the ground truth label, and synchronous output in pairs, without any manual labeling cost.

[0038] S4. Supervised training of the network model is performed based on the multi-channel multispectral simulation image and the true value of the white point to obtain the multispectral white balance correction model.

[0039] In this embodiment of the invention, the network model is a U-Net regression network. During training, the multi-channel multispectral simulated image is used as the input to the network model, and the ground truth white point values ​​are used as supervision labels to supervise the training of the network model, thereby obtaining the multispectral white balance correction model. During the training process, the hyperspectral reflectance data, light source spectrum pairing, and weight distribution map are randomly combined in each iteration to dynamically and in batches generate multispectral training samples that conform to the real illumination distribution law and corresponding pixel-level ground truth white point labels. This allows for the realization of infinite diversity of training data without any manual annotation, fundamentally solving the data scarcity problem of hyperspectral deep learning, and improving the prediction robustness and cross-scene generalization ability of the white balance network under complex mixed light sources from the data source.

[0040] S5. Obtain the multispectral image to be corrected, and use the multispectral image to be corrected as the input of the multispectral white balance correction model to obtain the white balance corrected multispectral image.

[0041] In this embodiment of the invention, the multispectral image to be corrected is a multispectral image acquired under a real complex light source scene; after the multispectral image to be corrected is input into the multispectral white balance correction model, the multispectral white balance correction model outputs pixel-level white point prediction results, and the white balance is corrected based on the white point prediction results to obtain the multispectral image after white balance correction.

[0042] In summary, this invention introduces pixel-by-pixel continuous light source weight distribution from real-world scene acquisition and constructs a complete multi-light source linear hybrid imaging physical model. It uses pixel-by-pixel continuous floating-point weights to complete the linear superposition of multi-light source spectra, fully reproducing the optical characteristics of the continuous and gradual change of light source mixing weights with spatial position in real-world scenes. It outputs physically reliable, spatially distributed multispectral simulation images of hybrid light sources and pixel-level white point ground truth values. Furthermore, the synthesis process and annotation process share the same origin and physical benchmark and can be freely extended to any multi-light source working condition.

[0043] Through controlled variable experiments, under identical conditions of hyperspectral reflectance library, light source spectral library, U-Net regression network architecture, and training hyperparameters, the model trained using the true weight distribution synthesis strategy of this invention significantly outperforms the model trained using the geometric stitching synthesis strategy in four indicators: mean angle error, median angle error, best 25th percentile, and worst 25th percentile. The results are shown in the table below:

[0044] As shown in the table above, using the multispectral white balance method for complex light sources proposed in this invention, the average angle error was reduced from 4.51° to 2.26°, the median angle error from 4.16° to 1.97°, the best 25th percentile error from 1.97° to 1.08°, and the worst 25th percentile error from 7.91° to 3.99°. This demonstrates that this invention can provide downstream white balance regression networks with training data that more closely approximates real-world illumination distributions, fundamentally improving the model's generalization ability and prediction accuracy in complex mixed light source scenarios. It effectively solves the problems of existing synthesis schemes deviating from optical physics, simulation samples deviating from real-world scene distributions, insufficient scene diversity, and scarcity of multispectral training data.

[0045] Example 2 This invention also provides a multispectral white balance system under complex light sources; please refer to... Figure 2 , Figure 2 This is a schematic diagram of the structure of a multispectral white balance system 200 under complex light sources provided in an embodiment of the present invention, which includes: The data acquisition module 201 is used to acquire hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data; wherein, the hyperspectral reflectance data is used to characterize the spectral reflectance characteristics of an object surface to incident light of different wavelengths; the spectral power distribution data of multiple light sources is used to characterize the relative energy distribution of different light sources at different wavelengths; and the true weight distribution map data is used to characterize the spatial mixing weight of each light source at each pixel position in the image. The simulation image module 202 is used to perform spectral integration calculations based on the hyperspectral reflectance data, the spectral power distribution data of the multiple light sources, the real weight distribution map data, and the spectral response function of the multispectral camera to obtain a multi-channel multispectral simulation image. The white point truth module 203 is used to obtain the white point truth value corresponding to the multi-channel multispectral simulation image by performing spectral integration calculation based on the spectral power distribution data of the multi-type light sources and the real weight distribution map data, and based on the RGB camera spectral response function. Model training module 204 is used to supervise the training of the network model based on the multi-channel multispectral simulation image and the white point ground truth, so as to obtain a multispectral white balance correction model. The white balance module 205 is used to acquire the multispectral image to be corrected, and use the multispectral image to be corrected as the input of the multispectral white balance correction model to obtain the white balance corrected multispectral image.

[0046] The multispectral white balance system 200 under complex light sources can implement the steps in the multispectral white balance method under complex light sources in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0047] Example 3 This invention also provides a computer device, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a multispectral white balance program for complex light sources stored in the memory 302 and capable of running on the processor 301.

[0048] The processor 301 calls the multispectral white balance program under complex light sources stored in the memory 302, and executes the steps in the multispectral white balance method under complex light sources provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps: S1. Acquire hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data; wherein, the hyperspectral reflectance data is used to characterize the spectral reflectance characteristics of an object surface to incident light of different wavelengths; the spectral power distribution data of multiple light sources is used to characterize the relative energy distribution of different light sources at different wavelengths; and the true weight distribution map data is used to characterize the spatial mixing weight of each light source at each pixel position in the image.

[0049] In this embodiment of the invention, the hyperspectral reflectance data includes multiple two-dimensional spatial images, each of which includes multiple pixels, and each pixel includes reflectance values ​​for multiple spectral bands. The hyperspectral reflectance data may be derived from the KAUST dataset, CAVE dataset, or a self-collected dataset. The size of each two-dimensional spatial image is H×W, where H is the height of the image and W is the width of the image. Each pixel contains reflectance values ​​for M spectral bands. The reflectance is an inherent physical property of the object and does not change with the lighting environment. The hyperspectral reflectance data serves as the basis for material information in data synthesis.

[0050] The multi-source spectral power distribution data includes multiple continuous spectral curves with different color temperatures and different light source types. The continuous spectral curves are used to characterize the relative energy distribution of the light source at different wavelengths. The multi-source spectral power distribution data is spectral power distribution (SPD) data obtained by actual measurement using a spectroradiometer.

[0051] The real weight distribution map data includes multiple weight distribution maps. Each pixel position in each weight distribution map corresponds to a mixing weight coefficient that matches each light source. For any given pixel position, the mixing weight coefficients for each light source are between 0 and 1, and the sum of the mixing weight coefficients for all light sources equals 1. The real weight distribution map data can be derived from the LSMI dataset. Each weight distribution map has a size of H×W×N and records the mixing weights of N light sources at each pixel position in the scene. The mixing weights are continuous floating-point values, and their spatial distribution naturally incorporates multiple physical influencing factors such as the object's three-dimensional geometry, spatial occlusion relationships, the object's surface material reflection characteristics, the spatial position of the light source, and illumination attenuation, reflecting the physical law that the mixing weights of light sources continuously and gradually change with spatial position in the real world. This invention introduces pixel-by-pixel continuous light source weight distribution collected from real scenes to replace the traditional manual geometric hard segmentation method. This ensures that the two-dimensional spatial changes of the mixing weights of multiple light sources within the synthesized multispectral image completely conform to the physical laws of real optical imaging, eliminating false light source boundaries caused by geometric segmentation and improving the consistency between the synthesized data and the distribution of the real scene.

[0052] S2. Based on the hyperspectral reflectance data, the spectral power distribution data of the various light sources, the real weight distribution map data, and the spectral response function of the multispectral camera, spectral integration is performed to obtain a multi-channel multispectral simulation image.

[0053] In this embodiment of the invention, the spectral response function of the multispectral camera is used to accurately match the photoelectric conversion characteristics of a real camera; optionally, the multispectral camera is a 14-channel multispectral camera, and correspondingly, the multi-channel multispectral simulation image includes 14 spectral channels.

[0054] Step S2 includes the following sub-steps: S21. By performing pixel-by-pixel weighted superposition of the spectral power distribution of each light source in the multi-type light source spectral power distribution data and the mixing weight of each light source in the corresponding pixel in the real weight distribution map data, pixel-level mixed light source spectral data is obtained. S22. Based on the pixel-level mixed light source spectral data, the hyperspectral reflectance data, and the multispectral camera spectral response function, perform spectral integration operations pixel by pixel and spectral channel by traversing all pixel positions and all spectral channels to obtain multiple imaging response values. S23. All the imaging response values ​​are arranged and integrated according to the two-dimensional spatial coordinates of the image and the dimensions of the multispectral channels to obtain the multi-channel multispectral simulation image.

[0055] In this embodiment of the invention, the pixel-level hybrid light source spectral data satisfies the following conditions: ; in, Indicates at wavelength Next pixel The corresponding pixel-level hybrid light source spectral data, Indicates the first i Each light source at wavelength The following is the spectral power distribution data of various light sources. Represents pixels In the i Spatial mixing weights corresponding to each light source Indicates the number of light sources.

[0056] In this embodiment of the invention, the imaging response value satisfies the following condition: ; in, Represents pixels In the c Imaging response values ​​of multiple spectral channels, Represents the spectral integral operator. Indicates the firsti Spectral power distribution data of the various types of light sources, for each light source. Represents pixels The corresponding spectral reflectance, This represents the spectral response function of the multispectral camera.

[0057] In this embodiment of the invention, a dual-light source mixed scene is taken as a preferred implementation example, i.e., N=2, at which time the imaging response value satisfies: ; And the weights satisfy the normalization constraint. When the simulation scenario involves a mixed three-source lighting condition, a third source spectrum is added. and the corresponding spatial weights The overall imaging model is extended to a three-channel spectral integral superposition form, with the weights satisfying the normalization constraint. Similarly, a fourth light source parameter and weighting factor can be added to the four-light source scene. The number of weight channels corresponds one-to-one with the number of light sources participating in the mixing. The number of input light sources and the dimension of weight channels can be dynamically matched through the multi-light source extension adaptation control module, and the global weight normalization constraint conditions can be updated to realize the expansion of mixed lighting simulation without limiting the number of light sources. This invention constructs the imaging response of each pixel as a continuous weighted superposition of the spectra of multiple light sources. The weight of a single light source changes continuously in the range of [0,1], which can realistically reproduce the smooth and gradual transition characteristics of natural light and completely solve the defects of false light source boundaries and data distribution offset caused by geometric segmentation schemes. At the same time, the general weighted integral architecture is not limited by the number of light sources and can simulate complex real scenes such as indoor multi-light mixing, outdoor multi-light superposition, and multi-color temperature supplementary lighting in a photography studio, enriching the diversity of lighting conditions in training data.

[0058] S3. Based on the spectral power distribution data of the multiple light sources and the real weight distribution map data, and by performing spectral integration based on the RGB camera spectral response function, the ground truth (GT) of the white point corresponding to the multi-channel multispectral simulation image is obtained.

[0059] In this embodiment of the invention, step S3 includes the following sub-steps: S31. Perform channel-by-channel spectral integration processing based on the pixel-level mixed light source spectral data and the RGB camera spectral response function to obtain pixel-level raw RGB response data; S32. Normalize the pixel-level raw RGB response data to obtain the true value of the white point.

[0060] In this embodiment of the invention, the pixel-level raw RGB response data satisfies the following conditions: ; in, Represents pixels The pixel-level raw RGB response data, This represents the spectral response function of an RGB camera.

[0061] In this embodiment of the invention, the normalization process is a pixel-level normalization process, which maps the three-channel response values ​​to the 0~1 range and satisfies that the sum of the three channel values ​​is 1. The true value of the white point satisfies the following condition: ; Decomposed into a single-channel independent expression: ; ; ; in, Represents pixels The corresponding white point ground truth value. This invention eliminates overall light intensity interference and preserves the color temperature and color characteristics of the light source through normalization processing, meeting the normalized supervision label requirements for white balance network training; furthermore, the white point ground truth labeling process and the multispectral image synthesis process are based on unified light source, weights, and spectral integral physical logic, sharing the same source and physical benchmark, with no human error or distribution shift, achieving a one-to-one accurate correspondence between the synthesized image and the ground truth label, and synchronous output in pairs, without any manual labeling cost.

[0062] S4. Supervised training of the network model is performed based on the multi-channel multispectral simulation image and the true value of the white point to obtain the multispectral white balance correction model.

[0063] In this embodiment of the invention, the network model is a U-Net regression network. During training, the multi-channel multispectral simulated image is used as the input to the network model, and the ground truth white point values ​​are used as supervision labels to supervise the training of the network model, thereby obtaining the multispectral white balance correction model. During the training process, the hyperspectral reflectance data, light source spectrum pairing, and weight distribution map are randomly combined in each iteration to dynamically and in batches generate multispectral training samples that conform to the real illumination distribution law and corresponding pixel-level ground truth white point labels. This allows for the realization of infinite diversity of training data without any manual annotation, fundamentally solving the data scarcity problem of hyperspectral deep learning, and improving the prediction robustness and cross-scene generalization ability of the white balance network under complex mixed light sources from the data source.

[0064] S5. Obtain the multispectral image to be corrected, and use the multispectral image to be corrected as the input of the multispectral white balance correction model to obtain the white balance corrected multispectral image.

[0065] In this embodiment of the invention, the multispectral image to be corrected is a multispectral image acquired under a real complex light source scene; after the multispectral image to be corrected is input into the multispectral white balance correction model, the multispectral white balance correction model outputs pixel-level white point prediction results, and the white balance is corrected based on the white point prediction results to obtain the multispectral image after white balance correction.

[0066] The computer device 300 provided in this embodiment of the invention can implement the steps in the multispectral white balance method under complex light sources as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0067] Example 4 This invention also provides a computer-readable storage medium storing a multispectral white balance program under complex light sources. When executed by a processor, the multispectral white balance program under complex light sources implements the various processes and steps of the multispectral white balance method under complex light sources provided in this invention, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0068] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to computer programs or instructions. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0069] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0070] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0071] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form under the guidance of the present invention without departing from the spirit and scope of the claims. All such changes are within the protection scope of the present invention.

Claims

1. A multispectral white balance method under complex light sources, characterized in that, The multispectral white balance method under complex light sources includes the following steps: S1. Acquire hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data; wherein, the hyperspectral reflectance data is used to characterize the spectral reflectance characteristics of an object surface to incident light of different wavelengths; the spectral power distribution data of multiple light sources is used to characterize the relative energy distribution of different light sources at different wavelengths; and the true weight distribution map data is used to characterize the spatial mixing weight of each light source at each pixel position in the image. S2. Based on the hyperspectral reflectance data, the spectral power distribution data of the multiple light sources, the real weight distribution map data, and the spectral response function of the multispectral camera, spectral integration is performed to obtain a multi-channel multispectral simulation image. S3. Based on the spectral power distribution data of the multiple light sources and the real weight distribution map data, and based on the spectral response function of the RGB camera, perform spectral integration to obtain the true value of the white point corresponding to the multi-channel multispectral simulation image. S4. Supervised training of the network model is performed based on the multi-channel multispectral simulation image and the white point ground value to obtain the multispectral white balance correction model. S5. Obtain the multispectral image to be corrected, and use the multispectral image to be corrected as the input of the multispectral white balance correction model to obtain the white balance corrected multispectral image.

2. The multispectral white balance method under complex light sources as described in claim 1, characterized in that, Step S2 includes the following sub-steps: S21. By performing pixel-by-pixel weighted superposition of the spectral power distribution of each light source in the multi-type light source spectral power distribution data and the mixing weight of each light source in the corresponding pixel in the real weight distribution map data, pixel-level mixed light source spectral data is obtained. S22. Based on the pixel-level mixed light source spectral data, the hyperspectral reflectance data, and the multispectral camera spectral response function, perform spectral integration operations pixel by pixel and spectral channel by traversing all pixel positions and all spectral channels to obtain multiple imaging response values. S23. All the imaging response values ​​are arranged and integrated according to the two-dimensional spatial coordinates of the image and the dimensions of the multispectral channels to obtain the multi-channel multispectral simulation image.

3. The multispectral white balance method under complex light sources as described in claim 2, characterized in that, The pixel-level hybrid light source spectral data satisfies the following conditions: ； in, Indicates at wavelength Next pixel The corresponding pixel-level hybrid light source spectral data, Indicates the first i Each light source at wavelength The following is the spectral power distribution data of various light sources. Represents pixels In the i Spatial mixing weights corresponding to each light source Indicates the number of light sources.

4. The multispectral white balance method under complex light sources as described in claim 3, characterized in that, The imaging response value satisfies the following condition: ； in, Represents pixels In the c Imaging response values ​​of multiple spectral channels, Represents the spectral integral operator. Indicates the first i Spectral power distribution data of the various types of light sources, for each light source. Represents pixels The corresponding spectral reflectance, This represents the spectral response function of the multispectral camera.

5. The multispectral white balance method under complex light sources as described in claim 4, characterized in that, Step S3 includes the following sub-steps: S31. Perform channel-by-channel spectral integration processing based on the pixel-level mixed light source spectral data and the RGB camera spectral response function to obtain pixel-level raw RGB response data; S32. Normalize the pixel-level raw RGB response data to obtain the true value of the white point.

6. The multispectral white balance method under complex light sources as described in claim 5, characterized in that, The pixel-level raw RGB response data meets the following conditions: ； in, Represents pixels The pixel-level raw RGB response data, This represents the spectral response function of an RGB camera.

7. The multispectral white balance method under complex light sources as described in claim 1, characterized in that, The hyperspectral reflectance data includes multiple two-dimensional spatial images, each of which includes multiple pixels, and each pixel includes reflectance values ​​for multiple spectral bands. The spectral power distribution data of the multiple light sources includes multiple continuous spectral curves with different color temperatures and different light source types. The continuous spectral curves are used to characterize the relative energy distribution of the light source at different wavelengths. The real weight distribution map data includes multiple weight distribution maps. Each pixel position in each weight distribution map corresponds to a mixed weight coefficient that matches each light source. For any pixel position, the mixed weight coefficients corresponding to each light source are between 0 and 1, and the sum of the mixed weight coefficients corresponding to each light source is equal to 1.

8. A multispectral white balance system under complex light sources, characterized in that, The multispectral white balance system under complex light sources includes: The data acquisition module is used to acquire hyperspectral reflectance data, spectral power distribution data of multiple light sources, and true weight distribution map data. The hyperspectral reflectance data characterizes the spectral reflectance properties of an object surface to incident light of different wavelengths. The spectral power distribution data of multiple light sources characterizes the relative energy distribution of different light sources at different wavelengths. The true weight distribution map data characterizes the spatial mixing weights of each light source at each pixel location in the image. The simulation image module is used to perform spectral integration calculations based on the hyperspectral reflectance data, the spectral power distribution data of the multiple light sources, the real weight distribution map data, and the spectral response function of the multispectral camera to obtain a multi-channel multispectral simulation image. The white point truth module is used to obtain the white point truth value corresponding to the multi-channel multispectral simulation image by performing spectral integration calculation based on the spectral power distribution data of the multi-type light sources and the real weight distribution map data, and based on the RGB camera spectral response function. The model training module is used to supervise the training of the network model based on the multi-channel multispectral simulation image and the white point ground truth, so as to obtain a multispectral white balance correction model. The white balance module is used to acquire the multispectral image to be corrected, and to use the multispectral image to be corrected as the input of the multispectral white balance correction model to obtain the white balance corrected multispectral image.

9. A computer device, characterized in that, include: The system includes a memory, a processor, and a multispectral white balance program for complex light sources stored in the memory and executable on the processor. When the processor executes the multispectral white balance program for complex light sources, it implements the steps in the multispectral white balance method for complex light sources as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multispectral white balance program for complex light sources, which, when executed by a processor, implements the steps of the multispectral white balance method for complex light sources as described in any one of claims 1-7.

Images