Illumination Spectrum Restoration

A neural network with three-dimensional convolutions addresses the challenge of recovering illumination spectra in uncontrolled environments by directly estimating from hyperspectral images, providing accurate results without a white reference panel.

JP2024542330AInactive Publication Date: 2024-11-15COMMONWEALTH SCI & IND RES ORG
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
JP2023534025
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-24
Filing Date
2022-11-24
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Recovering the illumination spectrum in uncontrolled real-world images is challenging, especially for moving sensor platforms, as using a white reference panel is time-consuming and impractical.

Method used

A method involving a neural network with three-dimensional convolutions is used to estimate the illumination spectrum directly from hyperspectral or multispectral images, utilizing a smoothing function and training with white patches to minimize error, and applying it to convolutional layers for accurate intensity values.

Benefits of technology

This approach efficiently recovers accurate illumination spectra without the need for a white reference panel, enabling real-time processing and improved image analysis under various lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542330000001_ABST
    Figure 2024542330000001_ABST
Patent Text Reader

Abstract

The present disclosure relates to processing electronic images, such as hyperspectral or multispectral images. In particular, the present disclosure provides methods, software, and computer systems for estimating the illumination spectrum of a digital image. A processor applies a neural network to the digital image by computing three-dimensional convolutions in one or more convolution layers of the neural network. The three-dimensional convolutions comprise convolutions along the spectral dimension. The processor then evaluates an output layer connected to the one or more convolution layers in the neural network. The output layer has a plurality of output values, each of which provides an intensity value for a respective band of the illumination spectrum of the digital image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims priority from Australian Provisional Patent Application No. 2021903790, filed on November 24, 2021, the contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to the processing of electronic images, such as hyperspectral or multispectral images, and in particular provides methods, software, and computer systems for estimating the illumination spectrum of an image. [Background technology]

[0003] The appearance of objects in a scene depends heavily on the color of the light source. Hence, the recovery of the light power spectrum has applications in recognition, surveillance, and visual tracking. Despite its importance, the recovery and identification of the light source color in a scene has proven to be a difficult task in uncontrolled real-world imagery.

[0004] Traditionally, the illumination spectrum of a hyperspectral image is restored by measuring the illumination reflected off a white reference target (or Spectralon) placed in the scene. However, placing a white reference panel in the scene and estimating the illumination spectrum is a time-consuming endeavor and is not always practical. For example, when images are captured from a moving sensor platform (e.g., a vehicle), it is not feasible to use a white reference panel for every image.

[0005] 1 shows an example scene 100 with a mountain 102 illuminated by a sun 104 having a particular illumination spectrum. When capturing the scene 100, conventional cameras perform white balancing, which is often difficult without a white reference.

[0006] 2 shows the example scene 100 in more detail. The sun (light source) 104 has a light source spectrum 204. The mountain 102 has a reflectance spectrum 210. For simplicity only one reflectance spectrum is shown, but of course there can be many different reflectance spectra for many different materials.

[0007] When light from light source 104 hits mountain 102, the light source spectrum 204 is multiplied by the reflectance spectrum 210 and the resulting spectrum reaches sensor 212 as radiance spectrum 214. Sensor 212 has a large number of pixels, e.g., a million, and for each pixel location, captures a separate sampled version of the radiance spectrum.

[0008] 3 shows a data structure 300 for multispectral image data. The data structure 300 comprises one layer for each wavelength. Each layer represents the radiance values ​​for one wavelength and all the pixels, with an example pixel 302 highlighted. The values ​​of pixel 302 for different wavelengths, i.e. the radiance values ​​from the underlying layers at the same location as pixel 302, represent the radiance spectrum, also called the image spectrum or input spectrum.

[0009] Any discussion of documents, acts, materials, devices, articles or the like contained in this specification is not to be construed as an admission that any or all of such matters form part of the prior art document or were common general knowledge in the relevant field of the present disclosure existing prior to the priority date of each claim of this application.

[0010] Throughout this specification the word "comprise" or variations such as "comprises" or "comprising" will be understood to mean the inclusion of stated elements, integers, or steps, or groups of elements, integers, or steps, but not the exclusion of other elements, integers, or steps, or groups of elements, integers, or steps. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," International Conference on Learning Representations, 201 [Non-Patent Document 2] K. He, X. Zhang, S. Ren and J. Sun,"Deep Residual Learning for Image Recognition", IEEE Conference on Computer Vision and Pattern Recognition, 2016 [Non-Patent Document 3] K. He, X. Zhang, S. Ren and J. Sun,"Deep Residual Learning for Image Recognition", IEEE Conference on Computer Vision and Pattern Recognition, 2016 Summary of the Invention [Means for solving the problem]

[0012] The method for determining an illumination spectrum in a digital image comprises: computing a 3D convolution in one or more convolutional layers of the neural network, the 3D convolution comprising a convolution along a spectral dimension; evaluating an output layer connected to one or more convolutional layers in the neural network, each output layer having a plurality of output values ​​providing intensity values ​​for a respective band of the illumination spectrum of the digital image; This involves applying a neural network to a digital image.

[0013] Advantageously, the neural network comprises a 3-dimensional convolution along the spectral dimension, so that the spectral information can be efficiently trained, and further advantageously, the output values ​​provide intensity values ​​for each illumination band, so that the method directly produces an accurate illumination spectrum.

[0014] In some embodiments, the method further includes training the neural network by applying a smoothing function to the output values ​​of the output layer to calculate a cost value to be minimized during training.

[0015] In some embodiments, the smoothing function comprises a cubic spline approximation to the output values ​​of the output layer.

[0016] In some embodiments, the method further comprises downsampling bands of the digital image.

[0017] In some embodiments, the one or more convolution layers are configured to downsample bands of the digital image.

[0018] In some embodiments, the method further includes upsampling the result of the convolution layer.

[0019] In some embodiments, the method further includes training the neural network with a plurality of training images.

[0020] In some embodiments, training involves extracting, from a number of training images, the illumination spectrum observed from a white patch in the image.

[0021] In some embodiments, the training further includes generating a plurality of sub-images from the plurality of training images and minimizing an error between the determined illumination spectrum for the plurality of sub-images and the observed illumination spectrum.

[0022] In some embodiments, the error is based on a cubic smoothing spline function.

[0023] In some embodiments, the error is represented by an error function with a first summand based on the mean squared error and a second summand representing a roughness penalty.

[0024] In some embodiments, the roughness penalty is based on a forward difference of the output values.

[0025] In some embodiments, the neural network is based on ResNet.

[0026] In some embodiments, the output layer is a fully connected layer.

[0027] In some embodiments, the method further includes processing the hyperspectral image based on the illumination spectrum.

[0028] The software, when executed by a computer, causes the computer to carry out the above-described method.

[0029] A computer system for determining an illumination spectrum in a digital image includes: computing a 3D convolution in one or more convolutional layers of the neural network, the 3D convolution comprising a convolution along a spectral dimension; evaluating an output layer connected to one or more convolutional layers in the neural network, each output layer having a plurality of output values ​​providing intensity values ​​for a respective band of the illumination spectrum of the digital image; The method includes: providing a processor configured to apply a neural network to the digital image by

[0030] In some embodiments, the computer system further comprises an image sensor that generates a digital image, and a storage medium that stores the digital image and the illumination spectrum.

[0031] The optional features provided in reference to the above methods are likewise optional features for the computer system.

[0032] Examples will now be described with reference to the following drawings: [Brief description of the drawings]

[0033] [Figure 1] 1 shows an example scene according to the prior art. [Diagram 2] 2 shows the scene of FIG. 1 in more detail. [Diagram 3] 1 shows a data structure of multispectral image data according to the prior art. [Figure 4] 1 illustrates a computer system for estimating an illumination spectrum. [Diagram 5] 1 illustrates a computer-implemented method for determining the illumination spectrum in a digital image. [Figure 6] 4 illustrates a 3D convolution applied to the data structure of FIG. 3. [Figure 7] The fully connected output layer is shown. [Figure 8] 1 shows a training and testing workflow for illumination spectrum restoration. [Figure 9] Training error on the IllumNet dataset. Cubic smoothing spline error, mean squared error, and roughness error are for the validation dataset. [Figure 10] 4 shows predicted and measured illumination spectra for an outdoor, overcast scene. [Figure 11] 4 shows predicted and measured illumination spectra for an outdoor, sunny scene. [Figure 12] 4 shows predicted and measured illumination spectra for an outdoor, overcast scene. [Figure 13] 1 shows predicted and measured illumination spectra for an indoor LED-illuminated scene. [Figure 14]4 shows predicted and measured illumination spectra for an indoor halogen-illuminated scene. [Figure 15] 1 shows predicted and measured illumination spectra for a scene illuminated indoors with a mix of halogen and fluorescent lights. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The present disclosure provides a method, including a neural network, for recovering an illumination spectrum from a hyperspectral or multispectral image. The image is typically a digital image having a plurality of image pixels. Each image pixel is associated with a plurality of intensity values ​​related to the light intensity at a plurality of respective frequencies. These pixel values ​​may be calculated by a debayering method. In the following description, the term "pixel" may be replaced with "image point" to indicate that an individually addressable image element may be calculated based on a plurality of pixels. For example, the image resolution may be reduced by combining pixels, and the method 500 is performed on a low-resolution image having a plurality of points instead of pixels. Unless otherwise stated, when the word "pixel" is used, this may be equally applicable to "image point".

[0035] Computer Systems 4 shows a computer system 400 for estimating the illumination spectrum of an image of a scene 100. The computer system 400 comprises a sensor 402 and a computer 404. In this example, the sensor 402 is a hyperspectral or multispectral sensor capable of capturing an image of the scene 100 illuminated by three light sources as described with reference to FIG.

[0036] In one example, the computer system 400 is integrated into a handheld device, such as a consumer camera, and the scene 100 can be any scene on Earth, such as a tourist attraction, a person, an engineered structure, or an agricultural area. The sensor 402 can have multiple bands that balance computational cost and accuracy. The sensor 402 can have as low as three bands (e.g., RGB) to as high as hundreds of bands.

[0037] The computer 404 receives images from the sensor 402 via a data port 406, which are stored in local memory 408(b) by the processor 410. The processor 410 uses software stored in the memory 408(a) to execute the method shown in Figure 5. The program memory 408(b) is a non-transitory computer readable medium, such as, for example, a hard drive, solid state disk, or cloud storage.

[0038] The processor 410 performs a method for estimating the illumination spectrum of an image by applying a trained neural network to the image. The processor 410 may perform white balance or other image processing on the image using the illumination spectrum and store an updated version of the image in the data store 408(b). In other examples, the processor 410 stores white balance data and / or the determined illumination spectrum in the data store 408(b).

[0039] The software may provide a user interface that may be presented to a user on a monitor 412. The user interface may be capable of accepting input from a user (i.e., a touch screen). User input is provided by the monitor 412 to the input / output port 406. Images are stored in memory 408(b) by the processor 410. In this example, memory 408(b) is local to the computer 404, but may also be remote to the computer 404.

[0040] The processor 410 may receive data, e.g., image data, from the data memory 408(b) and from the communication port 406. In one example, the processor 410 receives image data from the sensor 402 via the communication port 406 using a Wi-Fi network, e.g., IEEE 802.11 compliant. The Wi-Fi network may be a decentralized ad-hoc network such that no dedicated management infrastructure, e.g., a router, is required, or a centralized network with a router or access point that manages the network.

[0041] In one example, the processor 410 receives and processes image data in real-time, meaning that the processor 410 determines the illumination spectrum every time it receives image data from the sensor 402 and completes this calculation before the sensor 402 sends the next image data update, which can be useful for live video processing.

[0042] Although communication port 406 is shown as a single entity, it should be understood that any type of data port may be used to receive data, such as a network connection, a memory interface, a pin on the chip package of processor 410, or an IP socket, or a logical port such as parameters of a function stored in program memory 408(a) and executed by processor 410. These parameters may be stored in data memory 408(b) and treated in the source code as values ​​or as references, i.e., pointers.

[0043] The processor 410 may receive data through all of these interfaces, including memory access of volatile memory, such as cache or RAM, or non-volatile memory, such as an optical disk drive, a hard disk drive, a storage server, or cloud storage. The computer system 404 may further be implemented in a cloud computing environment, such as a group of managed interconnected servers hosting a dynamic number of virtual machines.

[0044] It should be understood that any of the receiving steps may be preceded by the processor 410 determining or calculating the data to be subsequently received. For example, the processor 410 determines image data, for example by filtering or debayering (i.e., demosaicing) raw data from the sensor 402, and stores the image data in a data memory 408(b), for example a RAM or a processor register. The processor 410 then requests the data from the data memory 408(b), for example by providing a read signal along with a memory address. The data memory 408(b) provides the data as voltage signals on physical bit lines, and the processor 410 receives the image data via the memory interface.

[0045] method 5 illustrates a computer-implemented method 500 for determining the illumination spectrum in a digital image, executed by the processor 410. In other words, the method 500 may function as a blueprint or pseudocode for software implemented in a particular programming language, such as, for example, C++, and stored in the program memory 408(a) as compiled machine-readable code. The image is composed of points of wavelength-indexed spectral data, such as, for example, multispectral image data.

[0046] A digital image is a data structure that comprises multiple color values ​​for each of multiple points (i.e., pixels) of an image, as shown in Figure 3. Each color value is represented by a digitally coded value, e.g., a number from 0 to 255, indicating the intensity of that pixel in that color (i.e., band). Digital images may be stored in any of the available image formats, e.g., ENVI, jpg, tiff, png, bmp, gif, etc. Digital images may be transformed or modified, including resizing / rescaling, normalization, or other image pre-processing or transformation operations, before being processed by the processor 410.

[0047] Instead of analyzing a white reference region or fitting a reflectance model to the image, the processor 410 applies a neural network to the hyperspectral image 501. FIG.

[0048] The processor 410 does this by first computing 502 a three-dimensional convolution 601 in one or more convolution layers of the neural network. The three-dimensional convolution comprises a convolution along the spectral dimension. The processor 410 then evaluates an output layer 602 connected to one or more convolution layers in the neural network. The output layer has a number of output values, each of which provides an intensity value for a respective band of the illumination spectrum of the hyperspectral image.

[0049] Convolution Mathematically, a convolution is an integral function that describes the amount of overlap when one function g is shifted over another function f. Intuitively, a convolution acts as a mixer that mixes one function with another, reducing the data space while preserving information. From the perspective of neural networks and deep learning, convolutions are filters (matrices / vectors) with learnable parameters that are used to extract low-dimensional features from input data. They have the property of preserving spatial or positional relationships between input data points. Convolutional neural networks exploit spatial local correlations by imposing local connection patterns between neurons in adjacent layers.

[0050] Intuitively, convolution is the step of applying the concept of a sliding window (a filter with learnable weights) to the input and producing a weighted sum (of the weights and the input) as output, which is the feature space that is used as the input for the next layer.

[0051] For example, in a face recognition problem, the first few convolutional layers learn the presence of keypoints in the input image, the deeper convolutional layers learn edges and shapes, and the final convolutional layer learns faces. In this example, the input space is first reduced to a low dimensional space (representing information about points / pixels), then to another space (containing edges / shapes), and finally reduced to classify faces in the image. Convolution is applicable to N dimensions.

[0052] Here, convolution is applied in three dimensions, and Figure 6 shows 3D convolution applied to the data structure of Figure 3. 3D convolution applies a three-dimensional filter to a dataset, which moves in three directions (x, y, z) to compute a low-level feature representation, whose output shape is a three-dimensional volume space, for example a cube or a cuboid.

[0053] In FIG. 6, filter 601, a 3×3 filter in this example, calculates a weighted sum of pixel values. There are 9 weights, which are the coefficients of the filter. Thus, the filter starts at an initial position 602 in the data structure 300, multiplies each pixel value by its respective filter coefficient, and adds the results. Finally, filter 601 stores the resulting numerical value in the output pixel 603. The filter then moves pixel by pixel along one direction in the data structure 300 and repeats the calculation for the next pixel of the output image. That direction may be the x-dimension 604 or the y-dimension 605. Importantly, filter 601 may move along the spectral dimension 606. That is, the 3×3×3 dimensional filter 601 adds pixel values ​​along the spectral dimension 606 and also moves along the spectral dimension 606. Hence, this operation is referred to as convolution along the spectral dimension.

[0054] Output layer FIG. 7 shows a fully connected output layer 700. The output layer has an input 701 and an output 702. The input 701 is the result of a previous convolutional layer. That is, a convolutional layer, for example as shown in FIG. 6, is connected to the output layer 700 to provide the result to the output layer 700. However, the convolutional layer is not necessarily directly connected to the output layer 700 in the sense that there may be further non-convolutional layers between the convolutional layer and the output layer 700. For example, there may be a max pooling layer between the convolutional layer and the output layer 700.

[0055] The output layer has outputs 702, each providing a value of an estimated illumination spectrum at a respective wavelength. Thus, for example, a first output 703 provides as an output value a first intensity 704 of the illumination spectrum. Collectively, the outputs provide an overall illumination spectrum 705. In other words, the outputs collectively provide discrete samples of the illumination spectrum 705. As will be described below, these output samples may be used for spline interpolation.

[0056] In one example, the association of power 703 with particular wavelengths in the illumination spectrum may be predetermined, an even distribution of power along the spectrum, or in another example, the association may be trained to provide improved spectral resolution in particular bands.

[0057] Finally, note that FIG. 7 shows a simple example of only four inputs 702 and four outputs 703; in many practical applications the number of inputs and outputs will be significantly larger (see example values ​​below).

[0058] Considering Figures 6 and 7 together, the present disclosure provides a deep learning network for estimating the illumination spectrum of a hyperspectral image under various lighting conditions.

[0059] composition In one example, the input data size to the network is (B, C, D, H, W), where B is the batch size, C is the number of channels, D is the depth, H is the height, and W is the width. For RGB images (e.g., tracking, video segmentation), C is set to 3, corresponding to the number of channels in the image. In another example, C is set to 1 and D is the number of bands / s. For one setting, the number of bands is 204 and s is 4 (i.e., D=51), which is mainly due to memory constraints of the computer used to perform the network training.

[0060] The 3D convolution kernel (d, h, w) has both spatial and spectral range, but since the goal is to estimate the illumination spectrum, the depth of the kernel may be made longer than its width and height, i.e., d>(h, w) and h=w.

[0061] The 3D max pooling kernel can be smaller than the 3D convolution kernel so that fine details in the illumination spectrum, such as "peaks" caused by absorption bands, are not lost.

[0062] The output of the network is a vector with a size corresponding to the number of bands in the hyperspectral image. Because the input image is depth subsampled, the network also has the ability to perform interpolation, or spectral "super-resolution", of the input signal. In one example, the input image is depth subsampled to 51 and the output has 204 output values.

[0063] To train the network, a training dataset is generated or an existing dataset, such as IllumNet, can be used. Images can be captured with Specim IQ cameras or other cameras under various lighting conditions, both indoors and outdoors. Outdoor images can be captured under sunny, cloudy, and shaded conditions at various times of the day. For indoor images, halogen and LED light sources, as well as a mixture of halogen or LED and fluorescent light sources, can be used. A ResNet18 network is available, but the 2D kernel is modified to a 3D kernel to fit the spectral characteristics of the data. Along with a good fit to the actual illumination spectrum, the predicted illumination spectrum also needs to be smooth, which is achieved by a cubic smoothing spline error cost function. Experimental results show that the trained model is able to infer accurate estimates of the illumination spectrum.

[0064] problem definition The radiance or raw image captured by the camera is converted to a reflectance image to investigate the material composition of the scene. For each band, the reflectance intensity at pixel (x,y) can be found as follows:

number

[0065] The white reference comprises a material with nearly 100% reflectance with no spectral features. If the white reference is measured with the same illumination and measurement geometry and distance as the rest of the scene, the signal from the white reference target can be assumed to contain only signals from the illumination. That is, during training, measurements from the white target can be used as the illumination spectrum 705, and the network parameters are optimized so that the output 702 provides values ​​as close as possible to the illumination spectrum 705 from the white reference target. In that sense, the spectrum from the white reference target is similar to a landmark in supervised learning. The white reference target also contains information about the spectral response of the hyperspectral camera, i.e., how the camera affects the measured spectrum.

[0066] The goal of automatic illumination restoration is to restore the illumination using deep learning instead of a white reference target, i.e., after training, the trained network can be applied to input images without a white reference target.

[0067] Data Augmentation The illumination restoration dataset consisted of 1004 images captured with various illumination sources. In one example, the images have a pixel size of 512x512. Of these images, 80% were set for training and validation (70% for training and 10% for validation), and 20% for testing. For the training dataset, the white reference target (e.g., Spectralon) was cropped to avoid bias during training, and n 256x256 sub-images were randomly selected from each image. Each of the cropped images was then rotated three times (i.e., 90°, 180°, and 360°). To avoid bias toward either indoor or outdoor images, the training dataset contained an approximately equal number of indoor and outdoor images. As a result, the training dataset contained approximately 40,000 images.

[0068] Network Design The present disclosure can be implemented using several convolutional neural networks (CNNs) to recover the illumination spectrum. These include: -VGG16 (K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition", International Conference on Learning Representations, 201), ResNet18 (K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition", IEEE Conference on Computer Vision and Pattern Recognition, 2016), and ·ResNet101 (described in K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition", IEEE Conference on Computer Vision and Pattern Recognition, 2016) Includes.

[0069] The above references are hereby incorporated by reference in their entireties.

[0070] VGG16 generates large model files with a size of about 1GB, which may not be suitable for some portable applications. In some experiments, validation and testing results show that ResNet18 outperforms ResNet101, which is probably because ResNet18 has a shorter network and the shorter network is more suitable for the example dataset.

[0071] The main application of ResNet is object detection in RGB images. Since the method disclosed herein incorporates spectral features to recover the illumination of hyperspectral images, the original ResNet is modified to use 3D convolutions instead of 2D convolutions. It should be noted that other CNNs can be modified similarly.

[0072] Experimental results demonstrate that the use of 3D convolutions yields significantly better results. We refer to the modified ResNet network as ResNet3D18. The ResNet3D18 architecture for IllumNet is shown in Table 1. The structural blocks of the network are listed in the third column, along with the number of nested blocks. It should be noted that conv1 has an input channel of 1 and a depth of 51, which is the number of image bands downsampled (by nearest neighbor) by a factor of 4. This was done to reduce GPU memory usage. Downsampling was performed by conv31, conv41, and conv51, with a stride of 2. The last layer is a fully connected layer with 204 outputs, which correspond to the number of bands in the image. The last layer upsamples the spectrum back to the number of bands in the input image.

[0073] [Table 1]

[0074] Although specific examples of layer configurations are described above, it should be noted that a wide range of different configurations are equally applicable: the number of input channels is generally N, and the value of 51 above is used for a particular hardware setup, other examples may be 3-1000 or outside that range, and the number of output channels at 204 is only one example, and this number may vary widely, for example, between 3-1000 output channels, or outside that range.

[0075] Neural networks may have more layers (i.e., "deeper") or have different size filters or other different parameters. For example, there may be multiple combinations of convolutional and maxpooling layers before the data reaches the output layer. Neural networks may include one or more of convolutional layers, pooling layers, recurrent layers, preprocessing layers, normalization layers, regularization layers, attention layers, reshaping layers, merging layers, local connection layers, or activation layers. Possible neural networks may include Xception, EfficientNet B0-B7, VGG16 and VGG19, ResNet and ResNetV2, MobileNet and MobileNetV2, DenseNet, NasNetLarge and NasNetMobile, InceptionV3, InceptionResNetV2.

[0076] Additionally, the neural network may be implemented in Keras (https: / / keras.io / ) or other software tools or implementation frameworks.

[0077] implementation In one example, weights are trained from scratch rather than initialized with a pre-trained network. Stochastic Gradient Descent (SGD) was used with a mini-batch size of 4. Experimental results showed that lower mini-batch values ​​yielded better results. The learning rate was set to 0.005, momentum to 0.9, and the model was trained for 100 iterations.

[0078] The training and testing workflow of our illumination restoration method is shown in FIG.

[0079] Loss function The CNN is trained using an optimization process that uses a loss function to calculate the model error. The illumination spectrum recovery problem can be a regression problem. Examples of loss functions for regression problems include mean squared error (MSE) and mean absolute error (MAE). MSE and MAE are calculated as follows:

number

number

number

[0080] To obtain a predicted spectrum that is smooth and at the same time a good fit to the ground truth spectrum, the processor 410 uses a cubic smoothing spline function [3][4]. The smoothing spline is

number

number

number

[0081] A cubic smoothing spline estimate of the function f

number

number

number

number

number

[0082] Using the above formula, a Cubic Smoothing Spline Error (CSSE) function is fitted as the loss function for ResNet3D18 as follows:

number

number

number

number

number

number

[0083] The value of α is chosen so that the predicted spectrum is not noisy and at the same time not overly smooth: it is undesirable for the predicted spectrum to be less smooth than the actual spectrum, since this can attenuate significant absorption bands that are usually sharp.

[0084] Experimental Results Figure 9 shows the training error along with the MSE, roughness, and CSSE errors for various α values ​​on the validation data. An interesting observation from the plot is that the roughness converges quickly when α=0.6 and α=0.8. When α=1.0, the roughness error still converges, even though we are not minimizing roughness here. Not surprisingly, the roughness value for α=1.0 is always high.

[0085] Table 2 shows the test data results for various metrics and α after 50 epochs. The test set contains 398 full-size images, including both indoor and outdoor images. It is interesting to note that α=0.8 gives the lowest MSE. α=0.6 gives the lowest roughness and the lowest CSSE. However, using the lowest CSSE to select the optimal α is not a good idea, since low roughness values ​​can lead to over-smoothing of the predicted illumination spectrum. Over-smoothing of the predicted illumination spectrum can cause significant absorption bands in the spectrum to be lost. The actual average roughness of the test dataset is 0.0000585, and the closest predicted roughness value is α=0.8. It is proposed that the roughness value of α=1.0 is significantly higher, and the higher the roughness value, the higher the MSE. α can be further fine-tuned by training with values ​​of 0.6<α<1.0. However, in the remainder of this disclosure, the results for α=0.8 are presented.

[0086] [Table 2]

[0087] Figures 10-15 show the actual and predicted illumination spectra of images captured under various lighting conditions, indoors and outdoors. In most cases, the predicted illumination spectrum closely matches the actual illumination spectrum. In Figure 12, the shapes of the two spectra are very similar, but their magnitudes are different. This is likely due to illumination non-uniformity, where some areas of the scene are darker than others. The actual illumination spectrum is the illumination spectrum from a white target, while the predicted illumination spectrum may be the average illumination spectrum of the scene. The only way to measure this is to use multiple white targets in the scene.

[0088] Figure 15 shows the results of mixing different light sources. The images were captured in a room with a ceiling fluorescent light and a halogen light source pointed at the scene. The fluorescent light spectrum is spiky due to the use of phosphors in the bulbs to attenuate the UV light emitted by mercury vapor. The real illumination spectrum is a combination of the halogen and fluorescent spectra. The shape of the predicted spectrum is very similar to that of the real spectrum, but is different in magnitude and slightly coarser. This shows that the proposed illumination restoration method is capable of predicting reasonably accurate illumination spectra even under difficult lighting conditions.

[0089] conclusion Disclosed herein is a method for recovering the illumination spectrum of a hyperspectral image captured by any camera. A dataset called IllumNet is created. The dataset contains 1004 images captured under various illuminants both indoors and outdoors. The task of illumination recovery is formulated as a regression analysis problem and a deep learning network based on ResNet18 is disclosed. ResNet18 is modified to use a 3D kernel that better fits the 3D characteristics of the spectral data. A cubic smoothing spline error function is used as the loss function in the disclosed deep learning framework. This allows for the adaptation and control of the roughness of the predicted spectrum. Experimental results show that the disclosed deep learning method is capable of recovering the illumination spectrum of an image.

[0090] Dataset To capture the images, we use a Specim IQ (Specim Ltd., Oulu, Finland) hyperspectral camera. The Specim IQ is a handheld hyperspectral camera that performs hyperspectral data capture, illumination and reflectance reconstruction, and visualization of classification results in a single integrated unit. The sensor captures images using a push-broom mechanism, and each image cube consists of 204 bands with a spatial resolution of 512 × 512 pixels. The camera's wavelength range is 400-1000 nm.

[0091] The illumination dataset, IllumNet, consists of 1004 images and contains images captured for building attribute and material classification. Images were captured under different lighting conditions and light sources, namely sunlight, shade / cloudy, halogen, LED, fluorescent, and mixed light. Outdoor images were captured at different times of the day to take into account the changes in the solar spectrum. For indoor images, a variety of objects were used including leaves, fruits, rocks, paper, biscuits, metal, plastic, etc. to create complex and diverse scenes. The white reference panel was cropped from all images to prevent bias during the training process.

[0092] It should be noted that the training set is not exhaustive, e.g., it does not include images captured in other geographic locations or the use of lights from different lighting manufacturers. However, the proposed deep learning network is suitable for general lighting spectrum restoration, and the network can be retrained with other data to suit user needs and other application situations.

[0093] It will be understood by those skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments without departing from the broad general scope of the present disclosure, and the present embodiments are therefore considered in all respects to be illustrative and not restrictive. [Explanation of symbols]

[0094] 100 Scenes 102 Mountain 104 Sun (light source) 204 Light Source Spectrum 210 Reflectance Spectrum 212 Sensors 214 Radiance Spectrum 300 Data Structures 302 pixels 400 Computer Systems 402 Sensor 404 Computer 406 communication port 408 Memory 410 Processor 412 Monitor 601 Filter 602 Initial position 603 output pixels 604 x-dimension 605 y dimension 606 Spectral Dimension 700 output layers 701 Input 702 Output 703 First Output 704 First Strength 705 Lighting Spectrum

Claims

1. 1. A method for determining an illumination spectrum in a digital image, comprising: computing a 3D convolution in one or more convolutional layers of the neural network, the 3D convolution comprising a convolution along a spectral dimension; evaluating an output layer connected to the one or more convolutional layers in the neural network, the output layer having a plurality of output values ​​each providing an intensity value for a respective band of the illumination spectrum of the digital image; applying said neural network to said digital image by The method includes:

2. 2. The method of claim 1, further comprising training the neural network by applying a smoothing function to the output values ​​of the output layer to calculate a cost value to be minimized during training.

3. The method of claim 2 , wherein the smoothing function comprises a cubic spline approximation to the output values ​​of the output layer.

4. The method of claim 1 , further comprising downsampling bands of the digital image.

5. The method of claim 4 , wherein the one or more convolutional layers are configured to downsample the bands of the digital image.

6. The method of claim 1 , further comprising upsampling a result of the convolutional layer.

7. The method of claim 1 , further comprising training the neural network with a plurality of training images.

8. The method of claim 7 , wherein training comprises extracting from the plurality of training images an illumination spectrum observed from a white patch in the image.

9. 9. The method of claim 7 or 8, wherein training further comprises generating a plurality of sub-images from the plurality of training images and minimizing an error between the determined illumination spectrum and the observed illumination spectrum for the plurality of sub-images.

10. The method of claim 9 , wherein the error is based on a cubic smoothing spline function.

11. The method of claim 10 , wherein the error is represented by an error function comprising a first summand based on a mean squared error and a second summand representing a roughness penalty.

12. The method of claim 11 , wherein the roughness penalty is based on a forward difference of output values.

13. The method of claim 1 , wherein the neural network is based on ResNet.

14. The method of claim 1 , wherein the output layer is a fully connected layer.

15. The method of claim 1 , further comprising processing the digital image based on the illumination spectrum.

16. The method of claim 15 , wherein processing the digital image includes calculating a reflectance image by normalizing the digital image with respect to the illumination spectrum.

17. 17. The method of claim 1, wherein each of the one or more convolutional layers comprises a convolutional filter having a depth, a height, and a width, the width of the filter being greater than the height of the filter and greater than the width of the filter.

18. 20. The method of claim 17, wherein the neural network comprises a max pooling layer, the max pooling layer comprising a filter having a smaller size than the convolution filter.

19. 19. The method of any one of claims 1 to 18, wherein the digital image is a hyperspectral image.

20. Software which, when executed by a computer, causes the computer to carry out the method of any one of claims 1 to 19.

21. 1. A computer system for determining an illumination spectrum in a digital image, comprising: computing a 3D convolution in one or more convolutional layers of the neural network, the 3D convolution comprising a convolution along a spectral dimension; evaluating an output layer connected to the one or more convolutional layers in the neural network, the output layer having a plurality of output values ​​each providing an intensity value for a respective band of the illumination spectrum of the digital image; 4. A computer system comprising: a processor configured to apply the neural network to the digital image by:

22. 22. The computer system of claim 21, further comprising an image sensor that generates the digital image and a storage medium that stores the digital image and the illumination spectrum.

Citation Information

Cited By

  • Image processing system and image processing method

    WO2026120748A1