Method and apparatus for constructing cross-camera hyperspectral image simulation datasets for AWB

Abstract

This invention relates to the fields of automatic white balance and spectral imaging technology in computer vision and computational imaging, specifically to a method and apparatus for constructing a cross-camera hyperspectral image simulation dataset for AWB (Automatic White Balance). The method includes: each camera acquiring multiple sample entries based on mixed-light RAW RGB images and corresponding mixed-light RGB illumination labels to establish a light source database; generating an XYZ white balance reference image for the first camera; each camera generating a synthetic mixed-light scene XYZ image based on samples extracted from the light source database; and performing hyperspectral image reconstruction based on the XYZ white balance reference image and the synthetic mixed-light scene XYZ image to establish a mixed-light hyperspectral image simulation dataset. This invention can establish a simulation dataset with spatial continuity and realism for cross-camera illumination.

Description

Technical Field

[0001] This invention relates to the fields of automatic white balance and spectral imaging technology in computer vision and computational imaging, specifically to a method and apparatus for constructing a cross-camera hyperspectral image simulation dataset for AWB. Background Technology

[0002] Automatic white balance (AWB) is a fundamental and crucial research task in computer vision. Its goal is to eliminate the influence of light source color temperature variations on imaging results under complex lighting conditions, restoring the colors of objects in the image to a stable representation under standard lighting conditions, thereby simulating the adaptive ability of the human visual system to changes in illumination. In real-world applications, ambient lighting often exhibits significant spatial non-uniformity, especially under mixed lighting conditions where multiple light sources with different color temperatures act together. Different regions in the image may be affected by different light sources, making the modeling and inference of automatic white balance algorithms more challenging. Therefore, constructing high-quality mixed-light datasets with realistic illumination distribution characteristics is of great significance for developing and evaluating automatic white balance algorithms suitable for mixed-light scenarios.

[0003] While capturing real-world scene data through live-action shooting can preserve the authenticity of lighting and scenes to some extent, this method typically requires complex lighting control and expensive acquisition equipment, making it difficult to achieve large-scale data acquisition while ensuring lighting diversity. Furthermore, acquiring simultaneously and strictly aligned multimodal data (such as RGB and multispectral images) under live-action conditions presents inherent challenges. Therefore, current research tends to construct controllable datasets through simulation to meet the needs of automatic white balance algorithm training and evaluation. However, existing mixed-light simulation datasets still have the following shortcomings in their construction process: (1) Insufficient realism of illumination. This problem has already affected the quality of simulation data under single-camera conditions. Under multi-camera or cross-device conditions, due to the differences in imaging responses of different cameras to the same illumination distribution, the aforementioned unrealistic illumination distribution phenomenon will be further amplified if a unified and continuous illumination modeling mechanism is lacking, thereby significantly weakening the credibility and comparability of simulation data in cross-camera scenarios. Although the construction process of this type of simulation data with insufficient illumination realism is relatively simple, there are obvious distribution differences between it and the real imaging results, which limits the generalization ability of models trained based on such data in complex real-world scenarios and cross-camera applications.

[0004] (2) The lack of multispectral data and the difficulty in alignment are particularly prominent under multi-camera conditions. The differences in imaging perspective and internal parameters between different cameras make it almost impossible to acquire cross-camera, cross-modal and strictly aligned data in practice. At the same time, most existing mixed light simulation datasets still generally only contain RGB images and lack corresponding multispectral or hyperspectral data, making it difficult to support the simultaneous use of RGB and spectral information and further carry out related research on cross-camera joint modeling.

[0005] (3) RGB-based spectral reconstruction lacks a unified physical modeling foundation. Under single-camera conditions, this camera correlation can be implicitly ignored or mitigated through empirical methods. However, in cross-camera scenarios, if spectral modeling or data simulation is still performed directly based on the RGB space, it is easy to introduce systematic biases that are highly correlated with specific devices, making it difficult for the generated data to have a unified physical interpretation foundation. From the physical essence of spectral imaging, the spectral distribution in a real scene should be an objective attribute independent of specific imaging devices. Therefore, there is an urgent need for a unified intermediate representation and modeling framework that can connect the camera-dependent RGB space and the camera-independent spectral space and is applicable to multi-camera conditions.

[0006] (4) Insufficient ability to construct mixed-light data across cameras: Most existing mixed-light simulation datasets are constructed for single cameras or fixed imaging devices. Their illumination modeling, color mapping, and data distribution are highly coupled with specific camera parameters, making it difficult to directly transfer them to other camera conditions. In cross-camera application scenarios, different cameras have significant differences in sensor response, color calibration parameters, and imaging characteristics. Simulation data lacking a unified physical modeling foundation cannot guarantee color consistency and illumination comparability under cross-camera conditions. Therefore, further meeting the requirements of mixed-light modeling, hyperspectral data generation, and cross-camera consistency on the basis of existing datasets faces significant technical challenges.

[0007] In summary, there is an urgent need for a simulation data generation scheme for mixed lighting scenes under cross-camera conditions. This scheme should be able to consistently model the imaging response and lighting parameters of different cameras within a unified physical modeling framework. Under the premise of ensuring the scene spatial structure remains unchanged, it should be able to construct cross-camera simulation data that is spatially continuous in lighting, physically realistic, and strictly aligned with RGB and hyperspectral data. This would provide more reliable, universal, and physically consistent training and evaluation data for automatic white balance research under mixed lighting conditions and related cross-camera vision tasks. Summary of the Invention

[0008] In view of the above problems, the present invention provides a method and apparatus for constructing a cross-camera hyperspectral image simulation dataset for AWB, which solves the technical problem in the prior art that it is impossible to consistently model the imaging response and illumination parameters of different cameras.

[0009] On the one hand, the present invention provides a method for constructing a cross-camera hyperspectral image simulation dataset for AWB, comprising the following steps: Step S1: For each of the multiple cameras: obtain multiple sample entries based on the mixed-light RGB illumination labels and camera calibration parameters; establish a light source database from the multiple sample entries; For the first camera: obtain the RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label; obtain the light source coefficient mapping map based on the mixed-light RGB illumination label; generate the XYZ white balance reference image based on the RGB white balance reference image of the first camera and the color temperature related camera calibration parameters of each entry in the light source database; the first camera is one of multiple cameras; Step S2, for each camera in the second camera: An RGB white balance reference image is generated based on the XYZ white balance reference image and standard white light camera calibration parameters; the second camera is one of the multiple cameras other than the first camera. Step S3: For each of the multiple cameras: An RGB relighting illumination map is obtained based on the sample entries in the light source database and the light source coefficient mapping map; Generate a composite mixed-light scene RGB image based on the RGB white balance reference image and the RGB relighting illumination map; Obtain the XYZ relighting map based on the RGB relighting map; A composite mixed-light scene XYZ image is generated based on the XYZ white balance reference image and the XYZ relight illumination map. Step S4: Perform hyperspectral image reconstruction based on the XYZ white balance reference image to generate a hyperspectral reference image with multiple spectral bands. Perform hyperspectral image reconstruction based on the composite mixed-light scene XYZ image of each camera in multiple cameras to generate a composite mixed-light scene hyperspectral image with multiple spectral bands. Construct a mixed-light hyperspectral image simulation dataset.

[0010] Preferably, in step S1, the step of obtaining multiple sample entries based on the mixed-light RGB illumination label and camera calibration parameters specifically includes: Based on camera color matrix The illumination-color temperature consistency iterative algorithm is executed on the illumination value of each pixel in the mixed light RGB illumination label to obtain the correlation color temperature and XYZ illumination value corresponding to the illumination value of each pixel; Based on the aforementioned correlated color temperature and camera Camera color matrix With the camera's forward color matrix Obtain color temperature-related camera calibration parameters; Finally, the camera For each pixel, the RGB illumination value, XYZ illumination value, correlated color temperature, and color temperature-related camera calibration parameters are taken as a sample entry, and multiple sample entries are obtained.

[0011] Preferably, the step based on the correlated color temperature and camera... Camera color matrix With the camera's forward color matrix The expression for obtaining color temperature-related camera calibration parameters is:

[0012]

[0013]

[0014] in, and They represent cameras exist The color temperature-dependent color matrix and color temperature-dependent forward color matrix corresponding to the correlated color temperature of each pixel. These are the interpolation coefficients. express The correlated color temperature corresponding to a pixel. and Respectively represent and The corresponding calibrated color temperature.

[0015] Preferably, the illumination-color temperature consistency iterative algorithm specifically includes: (1) Initialize chromaticity coordinates, including setting the chromaticity coordinates to standard white point chromaticity coordinates, wherein the standard white point can be selected as the chromaticity coordinates corresponding to CIE standard illuminant D65; (2) Calculate the correlated color temperature corresponding to the current chromaticity coordinates, and adjust the camera color matrix based on the correlated color temperature. Perform linear interpolation to obtain the color temperature-related color matrix corresponding to the relevant color temperature; (3) Multiply the mixed light RGB illumination label with the inverse matrix of the color temperature related color matrix to obtain the XYZ illumination value; (4) Normalize the X and Y components in the XYZ illumination values ​​to obtain new chromaticity coordinates; (5) If the difference between the new chromaticity coordinates and the current chromaticity coordinates is greater than a preset threshold, then update the current chromaticity coordinates to the new chromaticity coordinates and return to step (2). If the difference between the new chromaticity coordinates and the current chromaticity coordinates is less than a preset threshold, then the current correlated color temperature and XYZ illumination label will be used as the final output correlated color temperature and XYZ illumination values.

[0016] Preferably, in step S1, the specific steps for obtaining the RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label include: The RGB white balance reference image of the first camera is obtained by performing element-wise division calculation on the mixed light RAW RGB image and the mixed light RGB illumination label. The steps for obtaining the light source coefficient mapping map based on the mixed-light RGB illumination label specifically include: Get RGB lighting tag The proportion of light source A and light source B components corresponding to a pixel. and , by the and Construct a light source coefficient mapping diagram; The expression for generating the XYZ white balance reference image based on the RGB white balance reference image of the first camera and the color temperature-related camera calibration parameters of each entry in the light source database is as follows:

[0017] in, Indicates camera exist The color temperature-dependent forward color matrix corresponding to the correlated color temperature of a pixel. This represents the XYZ space white balance reference image obtained through pixel-by-pixel mapping. Indicates camera exist RGB white balance reference image corresponding to each pixel.

[0018] Preferably, step S2 specifically includes: for each camera in the second camera: Search the light source databases of cameras other than Camera 1, and use the color temperature-correlated forward color matrix of the sample entry whose correlated color temperature is closest to that of standard white light conditions as the reference forward color mapping matrix. The expression is:

[0019] in, Indicates camera The reference forward color mapping matrix, Indicates to Optimize to obtain the minimum value; This indicates the correlated color temperature corresponding to standard white light. This indicates the calculation of absolute value; Invert the reference forward color mapping matrix to obtain the corresponding inverse reference color mapping matrix, and then apply the camera... The inverse reference color mapping matrix is ​​multiplied by the XYZ space white balance reference image to obtain the camera's... The RGB space white balance reference image is expressed as:

[0020] in, Indicates camera RGB space white balance reference image.

[0021] Preferably, in step S3, the expression for obtaining the RGB relighting mapping map based on the sample entries in the light source database and the light source coefficient mapping map is:

[0022] in, Indicates camera exist RGB relighting map at the pixel location and They represent from the camera The RGB illumination values ​​corresponding to two sample entries extracted from the light source database; In step S3, the step of generating a composite mixed-light scene RGB image based on the RGB white balance reference image and the RGB relight illumination map specifically includes: multiplying the RGB white balance reference image and the RGB relight illumination map element by element to generate a composite mixed-light scene RGB image. In step S3, the step of obtaining the XYZ relighting illumination map based on the RGB relighting illumination map specifically includes: performing a lighting-color temperature consistency iterative algorithm on the RGB relighting illumination map pixel by pixel to obtain the XYZ relighting illumination map; In step S3, the step of generating a composite mixed-light scene XYZ image based on the XYZ white balance reference image and the XYZ relight illumination map specifically includes: multiplying the XYZ white balance reference image and the XYZ relight illumination map element by element to generate a composite mixed-light scene XYZ image.

[0023] Preferably, step S4 specifically includes: The MST++ hyperspectral reconstruction model was retrained on XYZ-hyperspectral image pairs constructed from the ARAD-1K dataset; The XYZ space white balance reference image and the XYZ image of the composite mixed scene from each of the multiple cameras are respectively input into the retrained hyperspectral image reconstruction model to obtain the hyperspectral reference image and the corresponding composite mixed scene hyperspectral image for each camera. The dimensions of both the hyperspectral reference image and the synthesized mixed-light scene hyperspectral image are [missing information]. ,in, These represent the image height and width, respectively. For each camera: combine the image pairs consisting of the RGB space white balance reference image and the hyperspectral reference image, and the image pairs consisting of the synthesized mixed-light scene RGB image and the synthesized mixed-light scene hyperspectral image, and combine the above image pairs to form a mixed-light hyperspectral image simulation dataset.

[0024] On one hand, the present invention provides a device for constructing a cross-camera hyperspectral image simulation dataset for AWB, comprising: A multi-camera database construction module is used to: obtain multiple sample entries for each of multiple cameras based on mixed-light RGB illumination labels and camera calibration parameters; and build a light source database from the multiple sample entries. The spatial mapping module is used for the following purposes: for the first camera, it obtains an RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label; obtains a light source coefficient mapping map based on the mixed-light RGB illumination label; and generates an XYZ white balance reference image based on the RGB white balance reference image of the first camera and the color temperature-related camera calibration parameters of each entry in the light source database; wherein the first camera is one of a plurality of cameras; for each of the second cameras, it generates an RGB white balance reference image based on the XYZ white balance reference image and the standard white light camera calibration parameters; wherein the second cameras are other cameras besides the first camera among the plurality of cameras. A multi-camera relighting module is used for each of multiple cameras to: obtain an RGB relighting illumination map based on sample entries in the light source database and the light source coefficient mapping map; generate a composite mixed-light scene RGB image based on an RGB white balance reference image and the RGB relighting illumination map; obtain an XYZ relighting illumination map based on the RGB relighting illumination map; and generate a composite mixed-light scene XYZ image based on the XYZ white balance reference image and the XYZ relighting illumination map. The hyperspectral reconstruction module is used to reconstruct hyperspectral images based on XYZ white balance reference images, generating hyperspectral reference images with multiple spectral bands. It also reconstructs hyperspectral images based on the composite mixed-light scene XYZ images from each of the multiple cameras, generating composite mixed-light scene hyperspectral images with multiple spectral bands, and constructs a mixed-light hyperspectral image simulation dataset.

[0025] Compared with the prior art, the present invention has at least the following beneficial effects: (1) Physically consistent modeling capability for mixed lighting under multi-camera conditions Unlike existing methods that model only a single camera's imaging model, this invention extends the illumination-color temperature consistency modeling mechanism to multi-camera scenes under mixed lighting conditions. Illumination is no longer considered a local parameter bound to the camera, but rather a pixel-by-pixel approach that independently establishes the correspondence between RGB illumination values, correlated color temperature, and XYZ illumination representation at each spatial location. By introducing camera factory calibration parameters and performing unified interpolation on the color mapping matrix and forward color mapping matrix in the color temperature dimension, this invention can construct a light source database with consistent physical meaning for different cameras. This maintains the continuity and consistency of the mixed lighting modeling process under multi-camera conditions, laying a unified physical foundation for subsequent cross-camera relighting and data generation.

[0026] (2) Cross-camera color consistency constraint mechanism based on standard white light reference and common color space To address the significant device-dependent differences in the RGB imaging space across different cameras, this invention uses the CIE XYZ color space as a camera-independent common intermediate representation. Furthermore, it selects a reference forward color mapping matrix under standard white light conditions among multiple cameras to define the color coordinate system of each camera under a unified reference imaging condition. This mechanism not only enables reversible mapping of the same scene between different camera RGB spaces but also allows for consistent modeling of the same physical illumination across camera conditions, effectively avoiding the systematic biases caused by directly performing illumination migration and spectral modeling in the camera-dependent RGB space.

[0027] (3) Strict alignment generation capability of RGB-hyperspectral multimodal data under simulation framework with cross-camera conditions By completing the white balance reference definition, mixed illumination relighting, and spectral reconstruction processes in a unified XYZ space, this invention can simultaneously generate RGB images corresponding to multiple cameras and their strictly spatially aligned hyperspectral images without relying on real multi-device synchronous acquisition. Since all modal data originate from the same white balance reference representation and unified illumination modeling process, the generation mechanism guarantees a strict pixel-level correspondence between the RGB images and the hyperspectral images, effectively overcoming the multimodal registration error problem commonly found in existing real-world or semi-simulated datasets.

[0028] (4) Scalable simulation data generation capability for any target camera This invention, requiring only known illumination labels and factory calibration parameters of the target camera, can simulate and generate RGB imaging results and their corresponding hyperspectral data representations under mixed illumination conditions, while maintaining known scene spatial structure, illumination mixing ratios, and physical modeling consistency. This feature allows the invention to overcome the limitations of real-world acquisition conditions on the number of devices and acquisition costs, providing a universal, controllable, and physically consistent high-quality data generation method for tasks such as cross-camera white balance modeling, illumination transfer, and multi-device color consistency analysis. Attached Figure Description

[0029] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] Figure 1 The flowchart shows the overall process of constructing a cross-camera hyperspectral image simulation dataset for AWB provided by this invention.

[0031] Figure 2 The flowchart of the multi-camera database construction and spatial mapping module provided by the present invention.

[0032] Figure 3 The flowchart of the multi-camera relighting and hyperspectral reconstruction module provided by the present invention is shown.

[0033] Figure 4 The flowchart of the single-camera light source database construction module provided by the present invention.

[0034] Figure 5 The flowchart of the single-camera scene relighting module provided by the present invention.

[0035] Figure 6 The flowchart of the illumination-color temperature consistency iterative algorithm provided by the present invention.

[0036] Figure 7 The flowchart below shows the module-by-module process of the device for constructing a cross-camera hyperspectral image simulation dataset for AWB provided by the present invention.

[0037] Figure 8 The flowchart shows a submodule of the device for constructing a cross-camera hyperspectral image simulation dataset for AWB provided by the present invention.

[0038] Figure 9 A schematic diagram of the RAW RGB image and related parameters of the input camera 1 provided by the present invention.

[0039] Figure 10 This is a schematic diagram of the XYZ space white balance reference image provided by the present invention.

[0040] Figure 11 This is a schematic diagram of the RGB white balance reference image of camera 2 provided by the present invention.

[0041] Figure 12 This is a schematic diagram of the pixel-by-pixel scene relighting result of the camera 1 provided by the present invention.

[0042] Figure 13 This is a schematic diagram of the pixel-by-pixel scene relighting result of the camera 2 provided by the present invention.

[0043] Figure 14 This is a schematic diagram of a hyperspectral reference image provided by the present invention.

[0044] Figure 15 A schematic diagram of a composite mixed-light scene hyperspectral image provided by camera 1 for the present invention.

[0045] Figure 16 A schematic diagram of a composite mixed-light scene hyperspectral image provided by camera 2 for the present invention. Detailed Implementation

[0046] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0047] This invention provides a method for controlling relighting simulation of multi-camera scenes and reconstructing corresponding hyperspectral images under multi-camera, multi-lighting conditions. It uses a raw RAW RGB image captured by a designated camera in a specific mixed lighting scene as the basic scene input, combined with the lighting label information and calibration parameters of that camera, and incorporates the lighting labels and calibration parameters of several other cameras. While maintaining the scene spatial structure and lighting mixing ratio, it constructs independent light source databases for each camera and performs cross-camera spatial mapping under a unified color space. This allows for controllable relighting simulation of multi-camera scenes and reconstruction of corresponding hyperspectral images. This embodiment is applicable to both single-light source scenes and multi-light source mixed lighting scenes, and can achieve color space alignment and lighting condition transfer between different cameras while maintaining physical consistency.

[0048] like Figure 7 , Figure 8 As shown, acquire camera Raw RAW RGB images captured in a specific scene. and the lighting label information corresponding to the image. And the camera's factory calibration parameters. Based on the aforementioned illumination label information and calibration parameters, an RGB space white balance reference image can be obtained. Simultaneously build a camera Light source database This is used to characterize the color mapping relationship of the camera under different correlated color temperatures. Based on this, through the construction of a multi-camera database and a space mapping module, white balance reference images from different cameras are mapped to a unified CIE XYZ color space to generate camera-independent XYZ space white balance reference images. Furthermore, through the multi-camera relighting and hyperspectral reconstruction module, pixel-by-pixel lighting relighting is first performed in the RGB and XYZ spaces of each camera, and the corresponding composite mixed-light scene RGB image is generated. XYZ images of the composite mixed lighting scene Then, using a hyperspectral image reconstruction model, spectral reconstruction is performed on the XYZ spatial white balance reference image and the synthetic mixed-light scene XYZ image, respectively, to output a hyperspectral reference image. and synthesized hyperspectral images of mixed light scenes .

[0049] Optionally, the dimensions of the original RAW RGB image, the RGB space white balance reference image, the composite mixed-light scene RGB image, the XYZ space white balance reference image, the composite mixed-light scene XYZ image, the hyperspectral reference image, and the composite mixed-light scene hyperspectral image can all be expressed as follows: ,in, Indicates the height of the image. Indicates the width of the image. The number of channels in the image is represented; RGB lighting labels and XYZ lighting representations are consistent with the spatial resolution of the image; the size of the RGB relighting map can be represented as... ,in The corresponding number of RGB color channels; the size of the light source coefficient map can be represented as ,in This indicates the number of light sources involved in the blending process in the scene.

[0050] Optionally, in this embodiment, the original RAW RGB image, RGB white balance reference image, XYZ white balance reference image, composite mixed-light scene RGB image, and composite mixed-light scene XYZ image are all the same size. ,in These correspond to the number of three color channels in the RGB and XYZ color spaces, respectively. Correspondingly, the dimensions of the RGB lighting mix label and the RGB relighting map are both... The dimensions of the hyperspectral reference image and the synthesized mixed-light scene hyperspectral image are both... ,in This corresponds to 31 spectral channels sampled at 10 nm intervals within the 400 nm to 700 nm wavelength range; furthermore, the number of light sources participating in the mixing in the application scenario is set to 2, therefore the size of the corresponding light source coefficient map is... .

[0051] To illustrate the effectiveness of the method proposed in this invention, a specific embodiment is provided below to describe the above-mentioned technical solution of this invention in detail. A method for constructing a cross-camera hyperspectral image simulation dataset for AWB is disclosed, and the specific implementation steps are as follows: Step S1: For each of the multiple cameras: obtain multiple sample entries based on the mixed-light RGB illumination labels and camera calibration parameters; establish a light source database from the multiple sample entries; For the first camera: obtain the RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label; obtain the light source coefficient mapping map based on the mixed-light RGB illumination label; generate the XYZ white balance reference image based on the RGB white balance reference image of the first camera and the color temperature related camera calibration parameters of each entry in the light source database; the first camera is one of multiple cameras; Step S2, for each camera in the second camera: An RGB white balance reference image is generated based on the XYZ white balance reference image and standard white light camera calibration parameters; the second camera is one of the multiple cameras other than the first camera. Step S3: For each of the multiple cameras: An RGB relighting illumination map is obtained based on the sample entries in the light source database and the light source coefficient mapping map; Generate a composite mixed-light scene RGB image based on the RGB white balance reference image and the RGB relighting illumination map; Obtain the XYZ relighting map based on the RGB relighting map; A composite mixed-light scene XYZ image is generated based on the XYZ white balance reference image and the XYZ relight illumination map. Step S4: Perform hyperspectral image reconstruction based on the XYZ white balance reference image to generate a hyperspectral reference image with multiple spectral bands. Perform hyperspectral image reconstruction based on the composite mixed-light scene XYZ image of each camera in multiple cameras to generate a composite mixed-light scene hyperspectral image with multiple spectral bands. Construct a mixed-light hyperspectral image simulation dataset.

[0052] The various steps of the present invention will be described in detail below.

[0053] (1) Construction of multi-camera database and spatial mapping In this step, the present invention, based on a single real mixed-light scene, combines the light source calibration information of multiple cameras to construct a light source database corresponding to each camera, and on this basis defines a unified standard white light reference imaging coordinate system, thereby achieving consistent spatial mapping and intermediate representation construction across cameras. For example... Figure 7 As shown, the process includes two main stages: single-camera light source database construction and multi-camera spatial mapping and standard white light reference definition. For details on the single-camera light source database construction process, please refer to [link to relevant documentation]. Figure 8 .

[0054] (1-1) Unified Modeling and White Balance Reference Definition for Mixed Light Imaging Under mixed lighting conditions, the camera The captured raw mixed-light RAW RGB image can be modeled as a pixel-by-pixel combination of a white balance reference image and the corresponding lighting. Specifically, for any pixel position... ,camera Mixed light RAWRGB image It can be represented as its RGB space white balance reference image. With RGB lighting labels The element-wise product, i.e.:

[0055] Among them, symbols This indicates an element-wise multiplication operation. Indicates camera At pixel position RGB lighting values ​​at that location , They represent cameras Mixed RAW RGB image and RGB white balance reference image at pixel position (x, y).

[0056] Based on the above expression, for the camera The RGB white balance reference image is obtained by performing element-wise division calculations on the mixed-light RGB illumination labels from the mixed-light RAW RGB image. The RGB white balance reference image is used to characterize the scene reflection information after removing the influence of illumination and serves as the basic input for subsequent cross-camera spatial mapping.

[0057] Based on the proportions of the two light source components in the RGB lighting label, a pixel-level light source coefficient mapping map can be obtained. The light source coefficient mapping map is represented by the symbol... express.

[0058] Specifically, the light source coefficient mapping diagram is in The pixel location it points to has a weighting coefficient with dual reference light sources. and , representing the proportions of light source A and light source B in the RGB lighting label, respectively, are used to describe the relative proportions of the contributions of the two light sources to the overall lighting at a specific spatial location, and satisfy the relationship... This weighting information reflects the non-uniform spatial distribution characteristics of the mixed illumination.

[0059] (1-2) Construction of camera light source database Based on the above imaging model, this invention constructs a light source database for each camera and utilizes the camera... Using the illumination label and factory calibration parameters as input, a light source database corresponding to the camera is constructed. This process is executed independently on each camera, and its workflow is as follows: Figure 8 As shown. Specifically, input camera. Mixed light RGB lighting label and its corresponding two sets of color mapping matrices For RGB lighting labels The illumination value of each pixel in the image is used to perform an iterative algorithm for illumination-color temperature consistency. The process iteratively solves for the correlated color temperature and XYZ illumination value corresponding to the illumination value of each pixel, and the relationship can be expressed as:

[0060] in, Indicates camera At pixel position Correlated color temperature at that location Indicates camera At pixel position XYZ illumination values ​​at the location, Indicates camera The camera color matrix.

[0061] The following is a detailed description of the illumination-color temperature consistency iterative algorithm, such as... Figure 6 As shown, the illumination-color temperature consistency iterative algorithm includes: Step 6001: Initialize chromaticity coordinates, including setting the chromaticity coordinates to standard white point chromaticity coordinates, wherein the standard white point can be selected as the chromaticity coordinates corresponding to CIE standard illuminant D65; Step 6002: Calculate the correlated color temperature corresponding to the current chromaticity coordinates, and adjust the camera color matrix based on the correlated color temperature. Perform linear interpolation to obtain the color temperature-related color matrix corresponding to the relevant color temperature; Step 6003: Multiply the mixed light RGB illumination label with the inverse matrix of the color temperature related color matrix to obtain the XYZ illumination value; Step 6004: Normalize the X and Y components of the XYZ illumination values ​​to obtain new chromaticity coordinates; Step 6005: If the difference between the new chromaticity coordinates and the current chromaticity coordinates is greater than a preset threshold, then update the current chromaticity coordinates to the new chromaticity coordinates and return to step 6002; If the difference between the new chromaticity coordinates and the current chromaticity coordinates is less than a preset threshold, then the current correlated color temperature and XYZ illumination label will be used as the final output correlated color temperature and XYZ illumination values.

[0062] Based on the known correlated color temperature, further based on the camera The camera calibration parameters include two sets of matrices: the camera color matrix. With the camera's forward color matrix The camera's color temperature is calculated using a linear interpolation method. Color mapping matrix at correlated color temperature With forward color mapping matrix The expression is:

[0063]

[0064] in, and They represent cameras exist The color temperature-dependent color matrix and color temperature-dependent forward color matrix corresponding to the pixel's correlated color temperature, and the interpolation coefficients. Defined as:

[0065] in, express The correlated color temperature corresponding to a pixel. and Respectively represent and The corresponding calibrated color temperature.

[0066] In this step, camera 1 is processed separately to obtain its XYZ space white balance reference image, which serves as a camera-independent common intermediate representation. A detailed description follows: For the RGB space white balance reference image of camera 1 Here, the color temperature-dependent forward color matrix of camera 1 is used. The RGB white balance reference image is mapped pixel-by-pixel to the XYZ space to construct a camera-independent common intermediate representation, providing a data foundation for subsequent cross-camera space mapping and scene relighting steps. The specific mapping process is as follows:

[0067] in, Indicates camera exist The color temperature-dependent forward color matrix corresponding to the correlated color temperature of a pixel. This represents the XYZ space white balance reference image obtained through pixel-by-pixel mapping. Indicates camera exist RGB white balance reference image corresponding to each pixel.

[0068] The above processing procedure is performed on the camera. This process iterates through each pixel in the mixed-light RAW RGB image. For each unique RGB pixel value, a sample entry can be constructed, expressed as follows: ,in, Indicates camera The i-th entry, They represent cameras The i-th entry contains the correlated color temperature, RGB illumination value, XYZ illumination value, color matrix, and forward color matrix.

[0069] By aggregating multiple sample entries constructed from different illumination samples, a camera light source database can be formed, denoted as:

[0070] in, Indicates camera The number of light source samples contained in the light source database. When the database is large, quantitative sampling can be performed on the total number of sample entries, therefore... It doesn't have to be a camera. The total number of pixels in the RAW RGB image. The quantitative sampling may include: extracting a certain number of data entries from the data of each scene at fixed intervals of color temperature values, then adding them to the total light source database and saving them in real time. This sampling operation is an optional step, used to control the database size and reduce storage and computing overhead while ensuring the coverage of illumination distribution.

[0071] (1-3) Definition of standard white light reference forward mapping and cross-camera space mapping After completing the construction of the light source database and XYZ space white balance reference image for each camera, this step further introduces a unified reference imaging standard among multiple cameras.

[0072] Specifically, to minimize the differences in imaging space between different cameras, this embodiment selects standard white light conditions (D65) as a unified reference, and retrieves the light source databases of cameras other than camera 1, selecting the light source record that is closest to the standard white light color temperature for each camera. A reference forward color mapping matrix for that camera under standard white light conditions is defined, expressed as:

[0073] in, Indicates camera The reference forward color mapping matrix, Indicates to Optimize to obtain the minimum value; This indicates the correlated color temperature corresponding to standard white light. This indicates the calculation of the absolute value. The above expression is for processing cameras other than camera 1, therefore... The value range is 2 to N, where N is the total number of cameras.

[0074] Next, based on the XYZ space white balance reference image mentioned above... For any other camera For the reference forward color mapping matrix Inverting the color map yields the corresponding inverse reference color mapping matrix. This inverse reference color mapping matrix is ​​used to backmap the white balance reference image from the XYZ space to the camera's RGB space. The expression is:

[0075] in, Indicates camera RGB space white balance reference image.

[0076] Through the above processing, this module, while maintaining the consistency of the scene space structure, completed the color space unification and mapping between multiple cameras, and obtained the white balance reference images in the RGB space of each of cameras 1 to N. The common representation of the white balance reference image in XYZ space This provides a physically consistent data foundation for subsequent multi-camera scene relighting and hyperspectral image reconstruction modules.

[0077] (2) Multi-camera relighting and hyperspectral reconstruction In this step, using white balance reference images in RGB and XYZ spaces as intermediate representations, and constrained by the light source coefficient mapping map, pixel-by-pixel mixed illumination relighting simulation is performed on the imaging results of multiple cameras, and further hyperspectral reconstruction is performed on the generated XYZ image. Figure 7As shown, the process includes two main stages: multi-camera scene relighting and hyperspectral image reconstruction. For a detailed explanation of the single-camera scene relighting implementation, please refer to [link to documentation]. Figure 8 .

[0078] (2-1) Simulation of relighting in multi-camera scenes In the process of relighting in multi-camera scenes, the mixed lighting conditions are first modeled pixel by pixel based on the light source database corresponding to each camera and the light source coefficient mapping map of the scene.

[0079] Specifically, for any camera It can select several light source records from its light source database according to preset rules as reference light source parameters to participate in the mixing.

[0080] In this embodiment, taking dual-light source mixing as an example, from the camera Light source database Select two unique light source records, for example, corresponding to relevant color temperatures. and Combined at pixel position Light source coefficient at the location and Cameras can be built The RGB relighting map is expressed as:

[0081] in, Indicates camera exist RGB relighting map at the pixel location and They represent from the camera The RGB illumination values ​​corresponding to two sample entries extracted from the light source database.

[0082] After that, regarding the camera Based on the RGB white balance reference image and the RGB relighting map, element-wise multiplication is performed to generate a composite mixed-light scene RGB image, expressed as:

[0083] in, Indicates camera exist RGB white balance reference image at the pixel location.

[0084] To maintain consistency in lighting and color temperature in the XYZ space, the RGB relighting map needs to be mapped pixel-by-pixel to the XYZ space. Specifically, the aforementioned... Input as follows Figure 6The illumination-color temperature consistency iterative algorithm shown And combined with the camera's calibration color mapping matrix It can simultaneously solve the corresponding XYZ relighting map and its associated color temperature pixel by pixel, with the expression being:

[0085] Indicates camera exist The XYZ re-illuminated light map at the pixel location, with the same size as the input RGB re-illuminated light map.

[0086] Similarly, XYZ can be re-illuminated. With a unified XYZ space white balance reference image The camera is generated by combining elements one by one. The XYZ image of the composite mixed-light scene is expressed as:

[0087] in, express Composite mixed-light scene XYZ image at pixel location express XYZ white balance reference image at the pixel location.

[0088] Through the above steps, this invention generates corresponding RGB and XYZ images of the synthetic mixed lighting scene for each camera while maintaining the consistency of the scene structure, thereby achieving physical consistency simulation of mixed lighting under multi-camera conditions.

[0089] (2-2) Hyperspectral image reconstruction Obtaining the XYZ space white balance reference image And the XYZ images of the composite lighting scene corresponding to each camera. Subsequently, hyperspectral reconstruction is performed on the aforementioned XYZ images to expand their spectral dimensions. Specifically, this invention uses a hyperspectral image reconstruction model to complete the above process, denoted as the function... Its input is an XYZ spatial image. The output is a hyperspectral image. This corresponds to 31 spectral bands sampled at equal intervals of 10 nm within the visible light band from 400 nm to 700 nm.

[0090] The hyperspectral image reconstruction model can be a model provided by existing technology. In this embodiment, the MST++ hyperspectral reconstruction model can be used and retrained on the XYZ-hyperspectral image pairs constructed from the ARAD-1K dataset, so that the retrained MST++ hyperspectral reconstruction model can process input images in XYZ space.

[0091] Specifically, the XYZ space white balance reference image is first input into the hyperspectral image reconstruction model to obtain the hyperspectral reference image, which is represented as follows:

[0092] in, This represents a hyperspectral reference image.

[0093] Furthermore, for any camera The corresponding synthetic mixed-light scene XYZ image By inputting the same hyperspectral image reconstruction model, a synthetic hyperspectral image of the mixed-light scene can be obtained, which is represented as follows:

[0094] in, express Synthetic mixed-light scene hyperspectral image; hyperspectral reconstruction results, all sizes are .

[0095] Through the above processing, spatially strictly aligned RGB–hyperspectral image pairs can be obtained, including those from the camera. RGB space white balance reference image With hyperspectral reference image and camera RGB images of composite mixed lighting scenes Hyperspectral images of composite mixed light scenes Thus, a technical solution for batch construction of RGB-hyperspectral paired data was realized under multiple camera and mixed lighting conditions.

[0096] This allows us to obtain two spatially aligned pairs of RGB-hyperspectral images, including those from the camera. RGB space white balance reference image With hyperspectral reference image and camera RGB images of composite mixed lighting scenes Hyperspectral images of composite mixed light scenes Furthermore, this embodiment realizes a cross-camera simulation data construction method with a unified physical modeling foundation for multi-camera conditions. Specifically, given only the illumination label and factory calibration parameters of the target camera, this embodiment can generate a mixed illumination RGB image corresponding to any camera and its strictly aligned hyperspectral image while maintaining scene spatial structure, illumination mixing ratio, and color physical consistency. This simultaneously meets the requirements of real-world scene illumination modeling, accurate alignment of RGB and spectral data, and cross-camera data consistency, providing a general, controllable, and physically consistent data generation method for cross-camera white balance modeling, illumination transfer, and related visual tasks.

[0097] The steps described in the embodiments of the present invention are for the same mixed-light RAW RGB image. The processing procedure can be understood as follows: different mixed-light RAW RGB images can be processed in the same way as steps S1-S4 of this invention to obtain different mixed-light RAW RGB image simulation datasets.

[0098] This invention provides the implementation process of the overall technical solution as follows: Figures 1-6 As shown, a detailed description follows.

[0099] Figure 1 The overall flow of the present invention is shown, wherein: 101. Input the original RAW RGB image captured by camera 1 in a specific scene, along with the corresponding lighting label and the factory calibration parameters of the camera used for shooting. Here, it specifically refers to two types of calibration matrices: the color matrix (CM) and the forward color matrix (FM).

[0100] 102. Input the corresponding lighting labels of cameras 2~i in any scene, as well as the factory calibration parameters of each camera, namely the color matrix and the forward color matrix.

[0101] 103. The input passes through the multi-camera database construction and spatial mapping module. The purpose is to establish a one-to-one correspondence between the relevant color temperature, RGB illumination value and XYZ illumination value in each camera space by utilizing the consistency of illumination and color temperature, and to realize the mapping and conversion of the white balance reference image between the RGB space and XYZ space of multiple cameras.

[0102] 104. The single-camera light source database construction module is a core sub-module of the multi-camera database construction and spatial mapping module, which mainly implements the construction of the light source database for a single camera.

[0103] 105. RGB spatial white balance reference images for all cameras constructed using the multi-camera database construction and spatial mapping module.

[0104] 106. Utilizing the multi-camera database construction and spatial mapping module's output multi-camera light source database, white balance reference images in RGB camera space and XYZ common space, the multi-camera relighting and hyperspectral reconstruction module simulates real mixed lighting conditions and the imaging process of multiple cameras. Random illumination is then re-superimposed onto the white balance reference image of the input scene. Specifically, when the input scene is a single-light scene, global single illumination is directly superimposed; when the input scene is a multi-light scene, mixed illumination is superimposed according to the distribution of multiple light sources, ultimately obtaining relit RGB and relit XYZ images in multiple camera spaces. Then, a hyperspectral reconstruction model is introduced to perform spectral reconstruction on the white balance reference image and relit image in XYZ space, enriching and expanding the image information in terms of channel dimension and device diversity.

[0105] 107. The single-camera scene relighting module is a core sub-module of the multi-camera relighting and hyperspectral reconstruction module. It mainly implements the relighting steps of a single camera and outputs the composite relighting XYZ image and composite relighting RGB image of a single camera.

[0106] 108, the RGB images of all cameras relit by the multi-camera relit and hyperspectral reconstruction module.

[0107] 109~110, hyperspectral reference images predicted by the multi-camera relighting and hyperspectral reconstruction module and composite relighting hyperspectral images from all cameras.

[0108] Figure 2 The flowchart of the multi-camera database construction and spatial mapping module of the present invention is shown, wherein: 201, Input the original RAW RGB image captured by camera 1 in a specific scene.

[0109] 202. Input the illumination label information corresponding to the camera 1, as well as the color matrix and forward color matrix obtained when the camera was calibrated at the factory.

[0110] 203. Input the lighting label information corresponding to cameras 2~i under any scene conditions, as well as the color matrix and forward color matrix obtained by each camera at the factory calibration.

[0111] 204. Based on the illumination labels of input cameras 1~i and the factory calibration parameters of each camera, the module enters the single-camera light source database construction module. This module establishes a consistent mapping relationship of illumination-related parameters in the space of each camera through the illumination-color temperature consistency iterative algorithm. The constructed light source database maintains consistency in data structure and parameter type.

[0112] 205. Based on the input original RAW RGB image of camera 1, and combined with its corresponding illumination label information, generate the RGB space white balance reference image of camera 1.

[0113] Sections 206-208 construct light source databases for cameras 1 through 1 respectively using the single-camera light source database construction module. Each light source database contains the RGB illumination representation of the camera under different correlated color temperatures, the corresponding XYZ illumination representation, and CM' and FM' obtained by interpolation from the factory calibration parameters, thus providing a physically consistent and controllable parameter basis for subsequent multi-camera mixed illumination simulations.

[0114] 209. Based on the light source database of camera 1, the color temperature-dependent forward color mapping matrix FM′ corresponding to the input lighting conditions is called. In single-lighting scenarios, a single mapping is performed. In multi-lighting scenarios, the corresponding FM′ is called pixel by pixel according to the light source color temperature at each pixel position to map the RGB space white balance reference image of camera 1 to a unified CIEXYZ color space.

[0115] 210-211, based on the light source databases of cameras 2-i, records whose correlated color temperature is closest to the preset standard white light color temperature are selected to obtain the reference forward color mapping matrix corresponding to cameras 2-i under standard white light conditions. The reference forward color mapping matrix is ​​used to define the reference imaging coordinate system of each camera under standard white light conditions, rather than to describe the actual illumination distribution in a specific scene.

[0116] Steps 212-213 involve performing matrix inversion operations on the standard white light reference forward color mapping matrices of cameras 2-i obtained in the previous step to obtain the corresponding inverse mapping matrices, which are used to realize the inverse mapping of the white balance reference image from the CIE XYZ color space to the RGB space of each camera.

[0117] 214. Using the color temperature-dependent forward color mapping matrix of camera 1, its RGB space white balance reference image is mapped to the CIE XYZ color space to generate an XYZ space white balance reference image with the same spatial resolution as the input RGB space white balance reference image. In single-light scenes, this mapping process is performed only once; in multi-light scenes, the mapping process is performed pixel-by-pixel according to pixel position, thereby obtaining an XYZ space white balance reference image as a common intermediate representation.

[0118] 215~216, respectively, using the inverse mapping matrix under standard white light, the XYZ space white balance reference image is mapped back to the RGB space of each camera 2~i, generating the RGB space white balance reference image corresponding to each camera, representing a consistent reference representation of the same scene in different camera imaging systems under standard white light conditions.

[0119] Figure 3 The multi-camera relighting and hyperspectral reconstruction module flow of the present invention is shown, wherein: 301. Input the original RAW RGB image captured by camera 1 in a specific scene, the corresponding illumination label information, and the color matrix and forward color matrix calibrated by the camera at the factory.

[0120] 302. Input the lighting label information corresponding to cameras 2~i under any scene conditions, as well as the color matrix and forward color matrix obtained by each camera at the factory calibration.

[0121] 303. Based on the proportion of each light source in the illumination label of the RAW RGB image, a pixel-level light source coefficient mapping map can be obtained. This map is used to describe the relative proportion of the contribution of multiple light sources to the overall illumination at a specific spatial location. The number of coefficients at each pixel location depends on the number of light sources in the scene.

[0122] 304. The input passes through the multi-camera database construction and spatial mapping module. The purpose is to establish a one-to-one correspondence between the relevant color temperature, RGB illumination value and XYZ illumination value in each camera space by utilizing the consistency of illumination and color temperature, and to realize the mapping and conversion of the white balance reference image between the RGB space and XYZ space of multiple cameras.

[0123] 305. The single-camera light source database construction module is a core sub-module of the multi-camera database construction and spatial mapping module, which mainly implements the construction of the light source database for a single camera.

[0124] Images 306-308, constructed through a multi-camera database construction and spatial mapping module, correspond to the XYZ spatial white balance reference image of the RAW RGB image input, the RGB spatial white balance reference image of each camera, and the light source database, serving as the data foundation for the subsequent relighting simulation stage.

[0125] 309, the single-camera scene relighting module is a core sub-module of the multi-camera relighting and hyperspectral reconstruction module. It uses the pixel-level light source coefficient mapping map and XYZ space white balance reference image of the RAW RGB input image, as well as the RGB space white balance reference image and light source database of each camera as input. It mainly implements the relighting step of a single camera and outputs the composite relighting XYZ image and composite relighting RGB image of a single camera.

[0126] 310~311, by repeatedly using the single-camera scene relighting module, the composite relighting RGB image and the corresponding composite relighting XYZ image of each camera are obtained respectively.

[0127] 312. Use any effective hyperspectral image reconstruction model to perform spectral reconstruction on the XYZ image input.

[0128] 313. The synthesized re-illuminated XYZ images of cameras 1~i are processed by the hyperspectral image reconstruction model to obtain the re-illuminated hyperspectral image of each camera.

[0129] The hyperspectral reference image is obtained by reconstructing the white balance reference image in the XYZ space using a hyperspectral image model.

[0130] Figure 4 The flowchart of the single-camera light source database construction module of the present invention is shown, wherein: 401, the factory calibration parameters of camera k, specifically referring to two types of calibration matrices: the color matrix and the forward color matrix.

[0131] 402, the RGB lighting label for camera k. When the scene being labeled is a single-light scene, this label is usually a single RGB color vector; when the scene being labeled is a multi-light scene, this label is usually an RGB lighting map with the same spatial resolution as the RAW RGB image.

[0132] 403~404 are two forward color matrices FM1 and FM2 from the camera calibration parameters, and two color matrices CM1 and CM2.

[0133] 405. Input the RGB illumination labels and two color matrices into the illumination-color temperature consistency iterative algorithm. The specific process of the algorithm module is as follows: Figure 6 As shown, the algorithm is used to iteratively calculate the correlated color temperature and XYZ illumination value that correspond one-to-one with the RGB illumination value pixel by pixel. The number of times the iterative algorithm is called depends on the number of different RGB illumination values ​​in the illumination label.

[0134] 406. Through the calculation of the illumination-color temperature consistency iterative algorithm, the output is the correlated color temperature (CCT) corresponding to the RGB illumination label. When the scene type is a multi-illumination scene, the number of correlated color temperatures obtained depends on the number of different RGB illumination values ​​input to the iterative algorithm.

[0135] 407~408. Using correlated color temperature (CCT), interpolate the two forward color matrices FM1 and FM2 and the two color matrices CM1 and CM2 respectively to obtain the color temperature correlated forward color mapping matrix FM' and the color temperature correlated color mapping matrix CM'. When the scene type is a multi-light scene, the number of matrices depends on the number of different correlated color temperatures obtained by iteration.

[0136] 409. Through the calculation of the illumination-color temperature consistency iterative algorithm, the output is the XYZ illumination reference label corresponding to the RGB illumination label. Its size depends on the size of the input RGB illumination label.

[0137] 410. When the generated light source database is too large, sample and save the entries. This step is optional.

[0138] 411 is a camera k-light source database consisting of five types of data: correlated color temperature (CCT), RGB illumination labels, XYZ illumination labels, color temperature correlated forward color mapping matrix FM', and color temperature correlated color mapping matrix CM'.

[0139] Figure 5 The process of the single-camera scene relighting module of the present invention is shown, wherein: 501~503 are the XYZ spatial white balance reference images, the light source database of a certain camera k, and its RGB spatial white balance reference images obtained by the multi-camera database construction and spatial mapping module.

[0140] 504. Here, the number of mixed illuminations during the simulation is limited to two. Therefore, the total light source database is randomly sampled twice without overlap, and then the data entries of the dual reference light sources are obtained.

[0141] 505~506, by sampling from two random reference light sources, two sets of parameters for reference light source A and reference light source B are obtained.

[0142] 507, the pixel-level light source coefficient mapping map obtained in the main module process 303.

[0143] 508 combines the parameters of reference light source A and reference light source B with a pixel-level light source coefficient mapping map to obtain a pixel-level mixed RGB relighting mapping map, which simulates the real fusion and superposition of multiple light sources in a scene.

[0144] 509. The simulated pixel-level mixed RGB re-illumination map is input into the illumination-color temperature consistency iterative algorithm. The specific process of the algorithm module is as follows: Figure 6 As shown, the algorithm is used to iteratively calculate the correlated color temperature and XYZ illumination value that correspond one-to-one with the RGB illumination value pixel by pixel. The number of times the iterative algorithm is called depends on the number of different RGB illumination values ​​in the illumination map.

[0145] 510. After pixel-by-pixel iterative calculation, a pixel-level blended XYZ relighting map is obtained. It extends the simulation results of multiple light sources in the scene from the RGB domain to the XYZ domain. Its size depends on the size of the pixel-level lighting coefficient map.

[0146] 511. By combining the pixel-level blended XYZ relighting illumination map with the white balance reference image in XYZ space, a synthetic relighting XYZ image of camera k that approximates the real lighting conditions is obtained.

[0147] 512. By combining the pixel-level blended RGB relighting map with the RGB space white balance reference image of camera k, a synthetic mixed-light scene RGB image of camera k that approximates the real lighting conditions is obtained.

[0148] Figure 6 The flow of the illumination-color temperature consistency iterative algorithm of the present invention is shown, wherein: 601~602, input the RGB lighting labels, and the two color matrices CM1 and CM2 corresponding to the camera.

[0149] 603. Initialize the chromaticity coordinates to the preset standard white point chromaticity coordinates, wherein the standard white point can be selected as the chromaticity coordinates corresponding to the CIE standard illuminant D65.

[0150] 604. Calculate the corresponding correlated color temperature (CCT) based on the current chromaticity coordinates.

[0151] 605. Using the correlated color temperature (CCT), perform linear interpolation on the input color matrices CM1 and CM2.

[0152] 606. After linear interpolation, the color temperature-dependent color mapping matrix CM′ corresponding to the current color temperature is obtained.

[0153] 607. Perform an inverse operation on the color temperature-related color mapping matrix CM′ to obtain the inverse of the color mapping matrix.

[0154] 608. Perform matrix multiplication between the mixed-light RGB illumination label and the inverse color mapping matrix to obtain the corresponding XYZ illumination value, the size of which is consistent with the RGB illumination label.

[0155] 609. Normalize the X and Y components of the XYZ illumination values ​​in the result.

[0156] 610, the normalized result is used as the new chromaticity coordinates, denoted as xy_new.

[0157] 611. Determine whether the difference between the newly calculated chromaticity coordinates xy_new and the chromaticity coordinates xy used in the previous iteration is less than a preset threshold (e.g., 1e-6).

[0158] 612~613, if the judgment result is yes, then output the XYZ illumination value corresponding to the current iteration as the XYZ illumination reference label, and the corresponding correlated color temperature (CCT), and end the iteration process.

[0159] 614. If the result is negative, update the currently calculated chromaticity coordinates xy_new to the chromaticity coordinates xy of the new iteration, and return to step 604 to continue the iteration steps.

[0160] This invention provides a device for constructing a mixed-light hyperspectral image simulation dataset for AWB, such as... Figure 7 As shown, it includes: A multi-camera database construction module is used to: obtain multiple sample entries for each of multiple cameras based on mixed-light RGB illumination labels and camera calibration parameters; and build a light source database from the multiple sample entries. The spatial mapping module is used for the following purposes: for the first camera, it obtains an RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label; obtains a light source coefficient mapping map based on the mixed-light RGB illumination label; and generates an XYZ white balance reference image based on the RGB white balance reference image of the first camera and the color temperature-related camera calibration parameters of each entry in the light source database; wherein the first camera is one of a plurality of cameras; for each of the second cameras, it generates an RGB white balance reference image based on the XYZ white balance reference image and the standard white light camera calibration parameters; wherein the second cameras are other cameras besides the first camera among the plurality of cameras. A multi-camera relighting module is used for each of multiple cameras to: obtain an RGB relighting illumination map based on sample entries in the light source database and the light source coefficient mapping map; generate a composite mixed-light scene RGB image based on an RGB white balance reference image and the RGB relighting illumination map; obtain an XYZ relighting illumination map based on the RGB relighting illumination map; and generate a composite mixed-light scene XYZ image based on the XYZ white balance reference image and the XYZ relighting illumination map. The hyperspectral reconstruction module is used to reconstruct hyperspectral images based on XYZ white balance reference images, generating hyperspectral reference images with multiple spectral bands. It also reconstructs hyperspectral images based on the composite mixed-light scene XYZ images from each of the multiple cameras, generating composite mixed-light scene hyperspectral images with multiple spectral bands, and constructs a mixed-light hyperspectral image simulation dataset.

[0161] The 9- of the present invention Figure 16 The experimental visualization results of this invention are shown.

[0162] like Figure 9As shown, from left to right, the images displayed are: the mixed-light RAW RGB images input from camera 1. Pixel-level light source coefficient mapping Its corresponding RGB lighting label RGB space white balance reference image Then, in the multi-camera database construction and spatial mapping module, the camera light source database is constructed, and the XYZ white balance reference image can be obtained simultaneously. As a representation of public space, such as Figure 10 As shown.

[0163] For the other cameras, inverting their standard white light reference forward color mapping matrix yields the corresponding inverse reference color mapping matrix. This allows the white balance reference image to be backmapped from the XYZ space to the RGB space of each camera. Taking camera 2 as an example... Figure 11 This shows the RGB white balance reference image of the camera. .

[0164] Next, using the multi-camera relighting and hyperspectral reconstruction module, pixel-by-pixel relighting mixtures in RGB and XYZ spaces are constructed respectively. These mixtures are then fused pixel-by-pixel with the white balance reference image, and finally, the hyperspectral image is reconstructed. Taking dual-reference light source relighting from camera 1 and camera 2 as an example, first, two non-repeating light source records are selected from the light source database of camera 1, and pixel-level relighting results are gradually constructed, such as... Figure 12 As shown, from left to right, the RGB relighting distribution of camera 1 is displayed. Composite RGB images of mixed lighting scenes XYZ images of the composite mixed lighting scene .

[0165] Similarly, the pixel-by-pixel relighting results from camera 2 can be seen, such as... Figure 13 As shown, from left to right, the RGB relighting distribution of camera 2 is displayed. Composite RGB images of mixed lighting scenes XYZ images of the synthesized mixed-light scene .

[0166] Finally, the white balance reference image in XYZ space... The XYZ images of the scene synthesized and mixed by the camera. For hyperspectral reconstruction, the MST++ hyperspectral reconstruction model is introduced here and retrained on XYZ-hyperspectral image pairs constructed from the ARAD-1K dataset, enabling it to process input images in XYZ space and output images as follows. This corresponds to 31 spectral bands sampled at 10 nm intervals within the visible light range of 400 nm to 700 nm, resulting in the reconstructed hyperspectral reference image. like Figure 14 As shown (single-band visualization).

[0167] Using the same method, the XYZ images of the composite lighting scene from both cameras were processed separately. and Hyperspectral reconstruction was performed, and the results are as follows: Figure 15 and Figure 16 As shown.

[0168] While the specific embodiments of the present invention depict actions or steps in a particular order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in sequential order, or requiring all illustrated actions or steps to be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0169] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing an AWB-oriented cross-camera hyperspectral image simulation dataset, characterized in that, Includes the following steps: Step S1: For each of the multiple cameras: obtain multiple sample entries based on the mixed-light RGB illumination labels and camera calibration parameters; establish a light source database from the multiple sample entries; For the first camera: Obtain the RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label; A light source coefficient mapping map is obtained based on the mixed-light RGB illumination label; an XYZ white balance reference image is generated based on the RGB white balance reference image of the first camera and the color temperature-related camera calibration parameters of each entry in the light source database; The first camera is one of a plurality of cameras; Step S2, for each camera in the second camera: An RGB white balance reference image is generated based on the XYZ white balance reference image and standard white light camera calibration parameters; the second camera is one of the multiple cameras other than the first camera. Step S3: For each of the multiple cameras: An RGB relighting illumination map is obtained based on the sample entries in the light source database and the light source coefficient mapping map; Generate a composite mixed-light scene RGB image based on the RGB white balance reference image and the RGB relighting illumination map; Obtain the XYZ relighting map based on the RGB relighting map; A composite mixed-light scene XYZ image is generated based on the XYZ white balance reference image and the XYZ relight illumination map. Step S4: Perform hyperspectral image reconstruction based on the XYZ white balance reference image to generate a hyperspectral reference image with multiple spectral bands. Perform hyperspectral image reconstruction based on the composite mixed-light scene XYZ image of each camera in multiple cameras to generate a composite mixed-light scene hyperspectral image with multiple spectral bands. Construct a mixed-light hyperspectral image simulation dataset.

2. The method for constructing a cross-camera hyperspectral image simulation dataset for AWB according to claim 1, characterized in that, In step S1, the step of obtaining multiple sample entries based on the mixed-light RGB illumination label and camera calibration parameters specifically includes: Based on camera color matrix The illumination-color temperature consistency iterative algorithm is executed on the illumination value of each pixel in the mixed light RGB illumination label to obtain the correlation color temperature and XYZ illumination value corresponding to the illumination value of each pixel; Based on the aforementioned correlated color temperature and camera Camera color matrix With the camera's forward color matrix Obtain color temperature-related camera calibration parameters; Finally, the camera For each pixel, the RGB illumination value, XYZ illumination value, correlated color temperature, and color temperature-related camera calibration parameters are taken as a sample entry, and multiple sample entries are obtained.

3. The method for constructing a cross-camera hyperspectral image simulation dataset for AWB according to claim 2, characterized in that, The basis of the correlated color temperature and camera Camera color matrix With the camera's forward color matrix The expression for obtaining color temperature-related camera calibration parameters is: in, and They represent cameras respectively. exist The color temperature-dependent color matrix and color temperature-dependent forward color matrix corresponding to the correlated color temperature of each pixel. These are the interpolation coefficients. express The correlated color temperature corresponding to a pixel. and Respectively represent and The corresponding calibrated color temperature.

4. The method for constructing a cross-camera hyperspectral image simulation dataset for AWB according to claim 3, characterized in that, The illumination-color temperature consistency iterative algorithm specifically includes: (1) Initialize chromaticity coordinates, including setting the chromaticity coordinates to standard white point chromaticity coordinates, wherein the standard white point can be selected as the chromaticity coordinates corresponding to CIE standard illuminant D65; (2) Calculate the correlated color temperature corresponding to the current chromaticity coordinates, and adjust the camera color matrix based on the correlated color temperature. Perform linear interpolation to obtain the color temperature-related color matrix corresponding to the relevant color temperature; (3) Multiply the mixed light RGB illumination label with the inverse matrix of the color temperature related color matrix to obtain the XYZ illumination value; (4) Normalize the X and Y components in the XYZ illumination values ​​to obtain new chromaticity coordinates; (5) If the difference between the new chromaticity coordinates and the current chromaticity coordinates is greater than a preset threshold, then update the current chromaticity coordinates to the new chromaticity coordinates and return to step (2). If the difference between the new chromaticity coordinates and the current chromaticity coordinates is less than a preset threshold, then the current correlated color temperature and XYZ illumination label will be used as the final output correlated color temperature and XYZ illumination values.

5. The method for constructing a cross-camera hyperspectral image simulation dataset for AWB according to claim 4, characterized in that, In step S1, the specific steps for obtaining the RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label include: The RGB white balance reference image of the first camera is obtained by performing element-wise division calculation on the mixed light RAW RGB image and the mixed light RGB illumination label. The steps for obtaining the light source coefficient mapping map based on the mixed-light RGB illumination label specifically include: Get RGB lighting tag The proportion of light source A and light source B components corresponding to a pixel. and , by the and Construct a light source coefficient mapping diagram; The expression for generating the XYZ white balance reference image based on the RGB white balance reference image of the first camera and the color temperature-related camera calibration parameters of each entry in the light source database is as follows: in, Indicates camera exist The color temperature-dependent forward color matrix corresponding to the correlated color temperature of a pixel. This represents the XYZ space white balance reference image obtained through pixel-by-pixel mapping. Indicates camera exist RGB white balance reference image corresponding to each pixel.

6. The method for constructing a cross-camera hyperspectral image simulation dataset for AWB according to claim 5, characterized in that, Step S2 specifically includes: for each camera in the second camera: Search the light source databases of cameras other than Camera 1, and use the color temperature-correlated forward color matrix of the sample entry whose correlated color temperature is closest to that of standard white light conditions as the reference forward color mapping matrix. The expression is: in, Indicates camera The reference forward color mapping matrix, Indicates to Optimize to obtain the minimum value; This indicates the correlated color temperature corresponding to standard white light. This indicates the calculation of absolute value; Invert the reference forward color mapping matrix to obtain the corresponding inverse reference color mapping matrix, and then apply the camera... The inverse reference color mapping matrix is ​​multiplied by the XYZ space white balance reference image to obtain the camera's... The RGB space white balance reference image is expressed as: in, Indicates camera RGB space white balance reference image.

7. The method for constructing a cross-camera hyperspectral image simulation dataset for AWB according to claim 6, characterized in that, In step S3, the expression for obtaining the RGB relighting mapping map based on the sample entries in the light source database and the light source coefficient mapping map is as follows: in, Indicates camera exist RGB relighting map at the pixel location and They represent from the camera The RGB illumination values ​​corresponding to two sample entries extracted from the light source database; In step S3, the step of generating a composite mixed-light scene RGB image based on the RGB white balance reference image and the RGB relight illumination map specifically includes: multiplying the RGB white balance reference image and the RGB relight illumination map element by element to generate a composite mixed-light scene RGB image. In step S3, the step of obtaining the XYZ relighting illumination map based on the RGB relighting illumination map specifically includes: performing a lighting-color temperature consistency iterative algorithm on the RGB relighting illumination map pixel by pixel to obtain the XYZ relighting illumination map; In step S3, the step of generating a composite mixed-light scene XYZ image based on the XYZ white balance reference image and the XYZ relight illumination map specifically includes: multiplying the XYZ white balance reference image and the XYZ relight illumination map element by element to generate a composite mixed-light scene XYZ image.

8. The method for constructing a cross-camera hyperspectral image simulation dataset for AWB according to claim 7, characterized in that, Step S4 specifically includes: The MST++ hyperspectral reconstruction model was retrained on XYZ-hyperspectral image pairs constructed from the ARAD-1K dataset; The XYZ space white balance reference image and the XYZ image of the composite mixed scene from each of the multiple cameras are respectively input into the retrained hyperspectral image reconstruction model to obtain the hyperspectral reference image and the corresponding composite mixed scene hyperspectral image for each camera. The dimensions of both the hyperspectral reference image and the synthesized mixed-light scene hyperspectral image are [missing information]. ,in, These represent the image height and width, respectively. For each camera: combine the RGB space white balance reference image and the hyperspectral reference image into an image pair, combine the synthesized mixed-light scene RGB image and the synthesized mixed-light scene hyperspectral image into an image pair, and combine the above image pairs into a mixed-light hyperspectral image simulation dataset.

9. A device for constructing a cross-camera hyperspectral image simulation dataset for AWB, characterized in that, include: A multi-camera database construction module is used to obtain multiple sample entries for each of multiple cameras based on mixed RGB lighting labels and camera calibration parameters. A light source database is established from the aforementioned sample entries; The spatial mapping module is used to obtain the RGB white balance reference image of the first camera based on the mixed-light RAW RGB image and the corresponding mixed-light RGB illumination label; A light source coefficient mapping map is obtained based on the mixed-light RGB illumination label; an XYZ white balance reference image is generated based on the RGB white balance reference image of the first camera and the color temperature-related camera calibration parameters of each entry in the light source database; The first camera is one of a plurality of cameras; for each of the second cameras: an RGB white balance reference image is generated based on the XYZ white balance reference image and standard white light camera calibration parameters; the second camera is any camera other than the first camera among the plurality of cameras; A multi-camera relighting module is used to obtain an RGB relighting illumination map for each of the multiple cameras based on sample entries in the light source database and the light source coefficient mapping map. Generate a composite mixed-light scene RGB image based on the RGB white balance reference image and the RGB relighting illumination map; Obtain the XYZ relighting map based on the RGB relighting map; A composite mixed-light scene XYZ image is generated based on the XYZ white balance reference image and the XYZ relight illumination map. The hyperspectral reconstruction module is used to reconstruct hyperspectral images based on XYZ white balance reference images, generating hyperspectral reference images with multiple spectral bands. It also reconstructs hyperspectral images based on the composite mixed-light scene XYZ images from each of the multiple cameras, generating composite mixed-light scene hyperspectral images with multiple spectral bands, and constructs a mixed-light hyperspectral image simulation dataset.

Images